Job Script Templates

Starting points for running your own work on Boole. Each template is a complete job script; copy it into your project folder and edit the lines marked CHANGE.

The templates are included in the example files (see Get the Example Files), in ~/hpc_examples/templates/. You can also download each one from its section below.

Your work looks like

Template

Background

One program, one core

serial.slurm

Your First Batch Job

One program using several cores (multiprocessing, joblib, OpenMP, R parallel)

multicore.slurm

4. Many Cores on One Node

The same program on many input files or parameters

array_files.slurm

5. Many Independent Jobs: Job Arrays

An MPI program across nodes

mpi.slurm

6. Many Nodes: MPI

Your own OpenFOAM case, in parallel

openfoam.slurm

Running OpenFOAM in Parallel on Boole

Training or inference on a GPU

gpu.slurm

Getting the Most from Boole’s GPUs

A program in your own Conda environment

conda.slurm

Conda / Miniforge

Not sure which? See 3. Pick the Right Pattern.

How to Use a Template

  1. Copy it next to your code, e.g. mkdir -p ~/myproject && cp ~/hpc_examples/templates/serial.slurm ~/myproject/run.slurm.

  2. Edit every line marked CHANGE (nano ~/myproject/run.slurm).

  3. Test small first. Run a short version of your work with a short --time, so mistakes cost minutes, not hours.

  4. When it finishes, check what it really used:

    sacct -j <jobid> --format=JobID,JobName,AllocCPUS,Elapsed,CPUTime,TotalCPU,MaxRSS,State
    

    Then set --mem, --time and the CPU count to what you saw plus a margin. See 9. Was My Job Efficient?.

  5. Scale up.

Output files are named <job name>-<job ID>.out (%x-%j.out), or <job name>-<array job ID>_<task>.out for the array template, and appear in the folder you ran sbatch from.

serial.slurm

Download serial.slurm

#!/bin/bash -l
# One program on one CPU core. Copy this file and edit the lines marked CHANGE.
#SBATCH --job-name=my-job            # CHANGE: a short name
#SBATCH --partition=cloud            # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G                     # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

module load python/3.11.9            # CHANGE: the software your program needs

python3 my_script.py                 # CHANGE: your command

multicore.slurm

Download multicore.slurm

#!/bin/bash -l
# One program using several cores on one node (multiprocessing, joblib, OpenMP, R parallel).
# Your program must use $SLURM_CPUS_PER_TASK workers or threads, not os.cpu_count().
#SBATCH --job-name=my-job            # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8            # CHANGE: cores (at most 8 on physical)
#SBATCH --hint=nomultithread         # whole cores, not hyperthreads
#SBATCH --mem=8G                     # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

module load python/3.11.9            # CHANGE: the software your program needs

export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK"   # for libraries that use OpenMP threads
python3 my_script.py --workers "$SLURM_CPUS_PER_TASK"   # CHANGE: your command

array_files.slurm

Download array_files.slurm

#!/bin/bash -l
# The same program on many input files: one array task per line of inputs.txt.
# Make the list, then submit one task per line (at most 10 running at once):
#
#   ls data/*.csv > inputs.txt
#   sbatch --array=1-$(wc -l < inputs.txt)%10 array_files.slurm
#
# (If you forget --array, the job fails and explains why in
# <job name>-<jobid>_4294967294.out.)
#
#SBATCH --job-name=my-array          # CHANGE: a short name
#SBATCH --partition=cloud            # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G                     # CHANGE: peak memory of ONE task plus a margin
#SBATCH --time=00:30:00              # CHANGE: run time of ONE task plus a margin
#SBATCH --output=%x-%A_%a.out        # %A = array job ID, %a = task number

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

: "${SLURM_ARRAY_TASK_ID:?Submit with sbatch --array=1-N, one task per line of inputs.txt}"
INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" inputs.txt)
if [[ -z "$INPUT" ]]; then
  echo "inputs.txt has no line ${SLURM_ARRAY_TASK_ID}" >&2
  exit 1
fi
echo "Task ${SLURM_ARRAY_TASK_ID}: $INPUT"

# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

module load python/3.11.9            # CHANGE: the software your program needs

python3 my_script.py "$INPUT"        # CHANGE: your command

mpi.slurm

Download mpi.slurm

#!/bin/bash -l
# An MPI program across several nodes: one rank per core.
#SBATCH --job-name=my-mpi            # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=2                    # CHANGE: nodes (physical has 4)
#SBATCH --ntasks-per-node=8          # CHANGE: ranks per node (at most 8 on physical)
#SBATCH --hint=nomultithread         # one rank per real core
#SBATCH --mem=8G                     # CHANGE: memory PER NODE, plus a margin
#SBATCH --time=01:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

module load openmpi/5.0.5 python/3.11.9 py-mpi4py/4.0.1 py-numpy/2.1.2   # CHANGE if needed

export PMIX_MCA_psec=native          # avoids harmless "psec: munge" error messages
srun --mpi=pmix_v5 python3 my_mpi_script.py   # CHANGE: your command (keep srun in front)

openfoam.slurm

Download openfoam.slurm

#!/bin/bash -l
# Your own OpenFOAM case, in parallel with MPI. Submit it from inside the case
# folder (the one holding 0/, constant/ and system/):
#
#   cd ~/myproject/mycase
#   sbatch ~/myproject/openfoam.slurm
#
# The case needs a system/decomposeParDict; copy one from a tutorial if it has none.
#SBATCH --job-name=my-openfoam       # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=1                    # CHANGE: nodes (physical has 4)
#SBATCH --ntasks-per-node=8          # CHANGE: MPI ranks per node (at most 8 on physical)
#SBATCH --hint=nomultithread         # one rank per real core
#SBATCH --mem=16G                    # CHANGE: memory PER NODE, plus a margin
#SBATCH --time=02:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

# OpenFOAM is listed under the MPI it was built with: load that MPI first.
module load openmpi/5.0.5-vzxfkrh openfoam

SOLVER=simpleFoam                    # CHANGE: your solver (icoFoam, pimpleFoam, interFoam, ...)

fail() { echo "$1 failed: see log.$1 in $PWD" >&2; exit 1; }

# blockMesh > log.blockMesh 2>&1 || fail blockMesh   # CHANGE: uncomment if the mesh isn't built yet

# One mesh piece per MPI rank (scotch splits a mesh into any number of pieces),
# then the solver on all ranks, then join the pieces back together.
foamDictionary -entry numberOfSubdomains -set "$SLURM_NTASKS" system/decomposeParDict > /dev/null 2>&1
foamDictionary -entry method -set scotch system/decomposeParDict > /dev/null 2>&1
decomposePar -force > log.decomposePar 2>&1 || fail decomposePar

export PMIX_MCA_psec=native          # avoids harmless "psec: munge" error messages
srun --mpi=pmix_v5 "$SOLVER" -parallel > "log.$SOLVER" 2>&1 || fail "$SOLVER"

reconstructPar -latestTime > log.reconstructPar 2>&1 || fail reconstructPar
echo "Done: results for the last time step are in $PWD"

gpu.slurm

Download gpu.slurm

#!/bin/bash -l
# A GPU job using the PyTorch environment from examples/gpu/setup_torch_env.sh.
# Each node has ONE GPU. Ask for the CPUs your data loading needs, not the whole
# node: the rest of the node can then run other people's jobs.
#SBATCH --job-name=my-gpu            # CHANGE: a short name
#SBATCH --partition=physical-gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=9            # CHANGE: data-loader workers + 1 (see docs/gpu_guide.md)
#SBATCH --mem=32G                    # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=02:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

# The GPU nodes have no `module` command: activate your environment directly.
source "$HOME/venvs/hpc-torch/bin/activate"   # CHANGE if you use another environment

# Record GPU use every 30 seconds; check it afterwards to see if the GPU was kept busy.
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 30 \
  > "gpu-usage-${SLURM_JOB_ID}.csv" &

# In your code, use SLURM_CPUS_PER_TASK - 1 data-loader workers (see docs/gpu_guide.md).
python3 train.py                     # CHANGE: your command

conda.slurm

Download conda.slurm

#!/bin/bash -l
# A program in your own Conda environment (see the Conda section of the platform guide).
# Don't load the cluster's python modules in the same job.
#SBATCH --job-name=my-job            # CHANGE: a short name
#SBATCH --partition=cloud            # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1            # CHANGE: more only if your program uses them
#SBATCH --mem=2G                     # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00              # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out           # %x = job name, %j = job ID

set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"

source "$HOME/miniforge3/bin/activate" my_env   # CHANGE: your environment name

python my_script.py                  # CHANGE: your command