Singularity is a container technology that allows users and administrators to build complex software stacks inside a single file for execution. This has several benefits which include the ability to choose an operating system of their choice as well as specific versions of software dependencies. OpenFoam is one of those applications which has several dependencies and requires a specific compile environment to be successful.
The ICTS HPC team has built a singularity container for OpenFOAM. In order to execute Openfoam commands via the SLURM scheduler, the following job script should exist.
#!/bin/bash
#SBATCH --account=icts
#SBATCH --partition=ada
#SBATCH --nodes=1 --ntasks=4
#SBATCH --time=10:00:00
#SBATCH --job-name="OpenFoam"
#SBATCH --mail-user=user@domain.co.za
#SBATCH --mail-type=BEGIN,FAIL
# Set this environment if your software is being called from a container
SINGULARITY_PATH=/opt/exp_soft/singularity-containers
FOAM_BASHRC=/opt/OpenFOAM/OpenFOAM-v1912/etc/bashrc
# try to compile solver as part of a SLURM job
singularity exec $SINGULARITY_PATH/openfoam/openfoam-v1912.sif bash -c "source $FOAM_BASHRC && wclean"
singularity exec $SINGULARITY_PATH/openfoam/openfoam-v1912.sif bash -c "source $FOAM_BASHRC && decomposePar -region fluid"
Let’s breakdown the singularity exec section of the SLURM job script. The first command and its associated argument inform the script to execute singularity. The next section $SINGULARITY_PATH/openfoam/openfoam-v1912.sif tells singularity which container to use. The final argument is the command to execute inside the container bash -c "source $FOAM_BASHRC && wclean" which says, execute the bash shell and inside the bash shell, run the following commands. With OpenFOAM, it is important to source the OpenFOAM bashrc to set up the environment. After sourcing the bashrc, the OpenFoam commands are executed.
The syntax related to OpenFOAM commands specifically can be updated as you see fit. For example, the script above does not take into account the data location from which the decomposePar is used.
Singularity containers are read-only and therefore cannot be changed. The ICTS HPC team has added /scratch and /home into the container as bind mounts which makes it easy for users to work with their data from within a container.