Slurm is the workload manager used to run jobs on Boole HPC. This page covers the login and compute nodes, partitions, interactive and batch jobs, and how to monitor them.
The commands assume you have downloaded the example files and are in ~/hpc_examples (see Get the Example Files).
When you open a terminal on Boole you are on the login node. Many people share it, and it is only for light work: editing files, moving data, and submitting jobs. Do not run heavy computations on the login node.
The real work runs on compute nodes. You get time on them by asking Slurm, the scheduler. Slurm queues your request until the CPUs, memory and time you asked for are free, then runs your work there. Your home directory is the same on every node, so files you create on the login node are visible from your jobs, and the other way round.
There are two ways to run work on compute nodes:
Interactive job (srun ... --pty bash): you get a shell on a compute node and type commands yourself. Good for testing and debugging.
Batch job (sbatch script.sh): you write the commands in a script, submit it, and Slurm runs it when resources are free, even after you log out. This is how most real work runs.
Jobs are submitted to partitions. A partition is a queue for one type of hardware:
Partition |
Nodes |
Each node |
Max job time |
Use |
|---|---|---|---|---|
|
2 |
8 cores, 22 GB RAM |
14 days |
Smaller or flexible jobs on virtual machines. |
|
4 |
8 cores (shown as 16 hyperthreads), 256 GB RAM |
10 days |
High-performance CPU jobs on dedicated hardware. See 1. Know Your Hardware. |
|
2 |
64 cores, 480 GB RAM, 1 NVIDIA A100 GPU |
7 days |
GPU-accelerated work. |
See the live state with sinfo, and each node’s details with scontrol show node.
If a workload needs longer than the maximum job time, split it into multiple jobs or use checkpointing where possible.
Important
Default Resource Allocation
If you don’t ask for more, a job gets 1 CPU and 1 GB of memory. Request what you need with --cpus-per-task, --mem and --time in your srun command or sbatch script. Ask for what you will use, not the maximum: a smaller request starts sooner and leaves room for others.
srun¶Request 1 CPU and 1 GB of memory for 15 minutes on the cloud partition:
srun --partition=cloud --cpus-per-task=1 --mem=1G --time=00:15:00 --pty bash
When the job starts, your prompt changes to show a compute node name, such as cpt2. Run your commands, then type exit to end the job and return to the login node. The job also ends when its time runs out.
A batch script is a shell script with #SBATCH lines at the top. Slurm reads those lines as options; bash treats them as comments. Comments after an #SBATCH option are fine, but never put one on the #! first line: the script then fails with bash: invalid option. This is examples/slurm/cpu_test.sh, with explanations:
#!/bin/bash -l
# The line above runs this script with bash, as a login shell (needed for modules).
#SBATCH --job-name=cpu_test # name shown in squeue
#SBATCH --output=cpu_test_%j.out # output file; %j becomes the job ID
#SBATCH --partition=cloud # which partition to run on
#SBATCH --cpus-per-task=2 # CPUs (on cloud, 1 CPU = 1 core)
#SBATCH --mem=2G # memory for the job
#SBATCH --time=00:10:00 # time limit (HH:MM:SS); the job is killed after this
set -euo pipefail # stop at the first error
SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}" # the directory you ran sbatch from
cd "$SUBMIT_DIR"
# --- Boole module setup: copy these lines as they are into your own scripts ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---
module load python/3.11.9 # the software this job needs
echo "Running on host: $(hostname)"
... # your commands
The module lines are explained in Loading Software with Modules.
From ~/hpc_examples, submit the script:
cd ~/hpc_examples
sbatch examples/slurm/cpu_test.sh
Slurm replies with the job ID:
Submitted batch job 19086
Use that number wherever these guides write <jobid>, without the < >. Check on the job:
squeue -u "$USER"
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
19086 cloud cpu_test you PD 0:00 1 (Priority)
The ST column is the state: PD means pending (waiting for resources; the reason is in the last column), and R means running. When the job finishes it disappears from squeue. This job takes about a second, so it is often gone by the time you look. Then read its output file, which appears in the directory you submitted from:
cat cpu_test_19086.out
Running on host: cpt2.boole.cloudcix.com
Job ID: 19086
Submit directory: /home/<you>/hpc_examples
Allocated CPUs: 2
...
Hello from Slurm on cpt2.boole.cloudcix.com.
This task can use 2 CPU core(s).
If you lose track of a job ID, ls -t *.out | head lists the newest output files, and sacct -u "$USER" -S today lists your jobs from today.
Options given to sbatch on the command line override the #SBATCH lines in the script, so you can try different requests without editing it:
sbatch --cpus-per-task=4 examples/slurm/cpu_test.sh
Command |
What it does |
|---|---|
|
Your pending and running jobs. |
|
Cancel a job. |
|
Full details of a pending or running job. |
|
Record of a finished job (state, run time, memory). |
sacct shows several lines per job: one for the whole job, then one per step. .batch is your script, .extern is Slurm’s own bookkeeping, and each srun inside the script adds a numbered step (.0, .1, …). Memory use (MaxRSS) appears on the step lines, not the first line.
To check whether a finished job used the CPUs and memory it reserved, see 9. Was My Job Efficient?.
Request GPUs with --gres=gpu:<N>. Each physical-gpu node has one GPU, so ask for --gres=gpu:1. Interactive session:
srun --partition=physical-gpu --gres=gpu:1 --cpus-per-task=4 --mem=16G --time=00:15:00 --pty bash
The batch example examples/gpu/gpu_test.sh prints nvidia-smi (the GPU and its driver) and nvcc --version (the CUDA compiler) from the GPU node:
#!/bin/bash
#SBATCH --job-name=gpu_test
#SBATCH --output=gpu_test_%j.out
#SBATCH --partition=physical-gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=00:10:00
set -euo pipefail
SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}"
cd "$SUBMIT_DIR"
echo "Running on host: $(hostname)"
echo "Job ID: ${SLURM_JOB_ID:-not-running-under-slurm}"
echo "Submit directory: ${SUBMIT_DIR}"
nvidia-smi
nvcc --version
python3 - <<'PY'
print("GPU allocation test complete. Add your CUDA or ML workload here.")
PY
sbatch examples/gpu/gpu_test.sh
cat gpu_test_<jobid>.out
| 0 NVIDIA A100-SXM4-80GB Off | 00000000:08:00.0 Off | 0 |
...
Cuda compilation tools, release 12.0, V12.0.140
To choose how many CPUs and how much memory to request with a GPU, and to check that the GPU is kept busy, see Getting the Most from Boole’s GPUs.
To use more than one core, or more than one node, see Parallel Computing on Boole: Cores, Nodes and MPI. It covers:
How cores, hyperthreads and nodes map to Slurm options on each partition.
Using many cores on one node, and running many jobs with job arrays.
MPI, scaling studies, hybrid jobs, and checking whether a job used what it asked for.
Slurm can email you when a job starts, ends, fails, or is cancelled. Add these options to your srun command or sbatch script:
Option |
Description |
|---|---|
|
Event or events that trigger an email: |
|
Email address to send notifications to. |
For an interactive session:
srun --partition=cloud --cpus-per-task=1 --mem=1G --time=00:15:00 \
--mail-type=ALL --mail-user=your.name@example.org --pty bash
examples/slurm/notify_test.sh waits a minute and emails you when it ends:
#!/bin/bash
#SBATCH --job-name=notify_test
#SBATCH --output=notify_test_%j.out
#SBATCH --partition=cloud
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=you@example.com # replace on the command line: sbatch --mail-user=...
set -euo pipefail
SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}"
cd "$SUBMIT_DIR"
echo "Job running on $(hostname)"
echo "Submit directory: ${SUBMIT_DIR}"
echo "You should get an email when this job ends."
sleep 60
echo "Job finished."
Give your own address on the command line:
sbatch --mail-user=your.name@example.org examples/slurm/notify_test.sh