Starting points for running your own work on Boole. Each template is a
complete job script; copy it into your project folder and edit the lines
marked CHANGE.
The templates are included in the example files (see Get the Example Files),
in ~/hpc_examples/templates/. You can also download each one from its
section below.
Your work looks like |
Template |
Background |
|---|---|---|
One program, one core |
||
One program using several cores (multiprocessing, joblib, OpenMP, R |
||
The same program on many input files or parameters |
||
An MPI program across nodes |
||
Your own OpenFOAM case, in parallel |
||
Training or inference on a GPU |
||
A program in your own Conda environment |
Not sure which? See 3. Pick the Right Pattern.
Copy it next to your code, e.g.
mkdir -p ~/myproject && cp ~/hpc_examples/templates/serial.slurm ~/myproject/run.slurm.
Edit every line marked CHANGE (nano ~/myproject/run.slurm).
Test small first. Run a short version of your work with a short
--time, so mistakes cost minutes, not hours.
When it finishes, check what it really used:
sacct -j <jobid> --format=JobID,JobName,AllocCPUS,Elapsed,CPUTime,TotalCPU,MaxRSS,State
Then set --mem, --time and the CPU count to what you saw plus a
margin. See 9. Was My Job Efficient?.
Scale up.
Output files are named <job name>-<job ID>.out (%x-%j.out), or
<job name>-<array job ID>_<task>.out for the array template, and appear in
the folder you ran sbatch from.
#!/bin/bash -l
# One program on one CPU core. Copy this file and edit the lines marked CHANGE.
#SBATCH --job-name=my-job # CHANGE: a short name
#SBATCH --partition=cloud # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
module load python/3.11.9 # CHANGE: the software your program needs
python3 my_script.py # CHANGE: your command
#!/bin/bash -l
# One program using several cores on one node (multiprocessing, joblib, OpenMP, R parallel).
# Your program must use $SLURM_CPUS_PER_TASK workers or threads, not os.cpu_count().
#SBATCH --job-name=my-job # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8 # CHANGE: cores (at most 8 on physical)
#SBATCH --hint=nomultithread # whole cores, not hyperthreads
#SBATCH --mem=8G # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
module load python/3.11.9 # CHANGE: the software your program needs
export OMP_NUM_THREADS="$SLURM_CPUS_PER_TASK" # for libraries that use OpenMP threads
python3 my_script.py --workers "$SLURM_CPUS_PER_TASK" # CHANGE: your command
#!/bin/bash -l
# The same program on many input files: one array task per line of inputs.txt.
# Make the list, then submit one task per line (at most 10 running at once):
#
# ls data/*.csv > inputs.txt
# sbatch --array=1-$(wc -l < inputs.txt)%10 array_files.slurm
#
# (If you forget --array, the job fails and explains why in
# <job name>-<jobid>_4294967294.out.)
#
#SBATCH --job-name=my-array # CHANGE: a short name
#SBATCH --partition=cloud # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1
#SBATCH --mem=2G # CHANGE: peak memory of ONE task plus a margin
#SBATCH --time=00:30:00 # CHANGE: run time of ONE task plus a margin
#SBATCH --output=%x-%A_%a.out # %A = array job ID, %a = task number
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
: "${SLURM_ARRAY_TASK_ID:?Submit with sbatch --array=1-N, one task per line of inputs.txt}"
INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" inputs.txt)
if [[ -z "$INPUT" ]]; then
echo "inputs.txt has no line ${SLURM_ARRAY_TASK_ID}" >&2
exit 1
fi
echo "Task ${SLURM_ARRAY_TASK_ID}: $INPUT"
# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
module load python/3.11.9 # CHANGE: the software your program needs
python3 my_script.py "$INPUT" # CHANGE: your command
#!/bin/bash -l
# An MPI program across several nodes: one rank per core.
#SBATCH --job-name=my-mpi # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=2 # CHANGE: nodes (physical has 4)
#SBATCH --ntasks-per-node=8 # CHANGE: ranks per node (at most 8 on physical)
#SBATCH --hint=nomultithread # one rank per real core
#SBATCH --mem=8G # CHANGE: memory PER NODE, plus a margin
#SBATCH --time=01:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
module load openmpi/5.0.5 python/3.11.9 py-mpi4py/4.0.1 py-numpy/2.1.2 # CHANGE if needed
export PMIX_MCA_psec=native # avoids harmless "psec: munge" error messages
srun --mpi=pmix_v5 python3 my_mpi_script.py # CHANGE: your command (keep srun in front)
#!/bin/bash -l
# Your own OpenFOAM case, in parallel with MPI. Submit it from inside the case
# folder (the one holding 0/, constant/ and system/):
#
# cd ~/myproject/mycase
# sbatch ~/myproject/openfoam.slurm
#
# The case needs a system/decomposeParDict; copy one from a tutorial if it has none.
#SBATCH --job-name=my-openfoam # CHANGE: a short name
#SBATCH --partition=physical
#SBATCH --nodes=1 # CHANGE: nodes (physical has 4)
#SBATCH --ntasks-per-node=8 # CHANGE: MPI ranks per node (at most 8 on physical)
#SBATCH --hint=nomultithread # one rank per real core
#SBATCH --mem=16G # CHANGE: memory PER NODE, plus a margin
#SBATCH --time=02:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
# --- Boole module setup: keep as it is ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
# OpenFOAM is listed under the MPI it was built with: load that MPI first.
module load openmpi/5.0.5-vzxfkrh openfoam
SOLVER=simpleFoam # CHANGE: your solver (icoFoam, pimpleFoam, interFoam, ...)
fail() { echo "$1 failed: see log.$1 in $PWD" >&2; exit 1; }
# blockMesh > log.blockMesh 2>&1 || fail blockMesh # CHANGE: uncomment if the mesh isn't built yet
# One mesh piece per MPI rank (scotch splits a mesh into any number of pieces),
# then the solver on all ranks, then join the pieces back together.
foamDictionary -entry numberOfSubdomains -set "$SLURM_NTASKS" system/decomposeParDict > /dev/null 2>&1
foamDictionary -entry method -set scotch system/decomposeParDict > /dev/null 2>&1
decomposePar -force > log.decomposePar 2>&1 || fail decomposePar
export PMIX_MCA_psec=native # avoids harmless "psec: munge" error messages
srun --mpi=pmix_v5 "$SOLVER" -parallel > "log.$SOLVER" 2>&1 || fail "$SOLVER"
reconstructPar -latestTime > log.reconstructPar 2>&1 || fail reconstructPar
echo "Done: results for the last time step are in $PWD"
#!/bin/bash -l
# A GPU job using the PyTorch environment from examples/gpu/setup_torch_env.sh.
# Each node has ONE GPU. Ask for the CPUs your data loading needs, not the whole
# node: the rest of the node can then run other people's jobs.
#SBATCH --job-name=my-gpu # CHANGE: a short name
#SBATCH --partition=physical-gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=9 # CHANGE: data-loader workers + 1 (see docs/gpu_guide.md)
#SBATCH --mem=32G # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=02:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
# The GPU nodes have no `module` command: activate your environment directly.
source "$HOME/venvs/hpc-torch/bin/activate" # CHANGE if you use another environment
# Record GPU use every 30 seconds; check it afterwards to see if the GPU was kept busy.
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 30 \
> "gpu-usage-${SLURM_JOB_ID}.csv" &
# In your code, use SLURM_CPUS_PER_TASK - 1 data-loader workers (see docs/gpu_guide.md).
python3 train.py # CHANGE: your command
#!/bin/bash -l
# A program in your own Conda environment (see the Conda section of the platform guide).
# Don't load the cluster's python modules in the same job.
#SBATCH --job-name=my-job # CHANGE: a short name
#SBATCH --partition=cloud # CHANGE if needed: cloud or physical
#SBATCH --cpus-per-task=1 # CHANGE: more only if your program uses them
#SBATCH --mem=2G # CHANGE: peak memory plus a margin (check MaxRSS with sacct)
#SBATCH --time=01:00:00 # CHANGE: expected run time plus a margin
#SBATCH --output=%x-%j.out # %x = job name, %j = job ID
set -euo pipefail
cd "${SLURM_SUBMIT_DIR:-$PWD}"
source "$HOME/miniforge3/bin/activate" my_env # CHANGE: your environment name
python my_script.py # CHANGE: your command