Slurm Basics

Slurm is the workload manager used to run jobs on Boole HPC. This page covers the login and compute nodes, partitions, interactive and batch jobs, and how to monitor them.

The commands assume you have downloaded the example files and are in ~/hpc_examples (see Get the Example Files).

Login Node and Compute Nodes

When you open a terminal on Boole you are on the login node. Many people share it, and it is only for light work: editing files, moving data, and submitting jobs. Do not run heavy computations on the login node.

The real work runs on compute nodes. You get time on them by asking Slurm, the scheduler. Slurm queues your request until the CPUs, memory and time you asked for are free, then runs your work there. Your home directory is the same on every node, so files you create on the login node are visible from your jobs, and the other way round.

There are two ways to run work on compute nodes:

  • Interactive job (srun ... --pty bash): you get a shell on a compute node and type commands yourself. Good for testing and debugging.

  • Batch job (sbatch script.sh): you write the commands in a script, submit it, and Slurm runs it when resources are free, even after you log out. This is how most real work runs.

Partitions

Jobs are submitted to partitions. A partition is a queue for one type of hardware:

Partition

Nodes

Each node

Max job time

Use

cloud (default)

2

8 cores, 22 GB RAM

14 days

Smaller or flexible jobs on virtual machines.

physical

4

8 cores (shown as 16 hyperthreads), 256 GB RAM

10 days

High-performance CPU jobs on dedicated hardware. See 1. Know Your Hardware.

physical-gpu

2

64 cores, 480 GB RAM, 1 NVIDIA A100 GPU

7 days

GPU-accelerated work.

See the live state with sinfo, and each node’s details with scontrol show node.

If a workload needs longer than the maximum job time, split it into multiple jobs or use checkpointing where possible.

Important

Default Resource Allocation

If you don’t ask for more, a job gets 1 CPU and 1 GB of memory. Request what you need with --cpus-per-task, --mem and --time in your srun command or sbatch script. Ask for what you will use, not the maximum: a smaller request starts sooner and leaves room for others.

Fair-Share Scheduling

Boole uses fair-share scheduling to make sure the cluster stays available to everyone.

You can still use as much compute capacity as you need when resources are free. However, if multiple users are waiting for the same resources, Slurm gives preference to users who have used less of the cluster recently.

This means that someone who has been running jobs continuously will temporarily receive a lower queue priority than someone who has had little or no recent access.

Usage is not treated as a permanent penalty. Recent usage gradually loses importance over time, so your priority naturally recovers.

Fair-share does not stop long or demanding workloads from running, and it does not reserve fixed amounts of the cluster for individual users. Its purpose is simply to prevent one user from repeatedly dominating shared resources when other users are waiting.

Interactive Jobs with srun

Request 1 CPU and 1 GB of memory for 15 minutes on the cloud partition:

srun --partition=cloud --cpus-per-task=1 --mem=1G --time=00:15:00 --pty bash

When the job starts, your prompt changes to show a compute node name, such as cpt2. Run your commands, then type exit to end the job and return to the login node. The job also ends when its time runs out.

Your First Batch Job

A batch script is a shell script with #SBATCH lines at the top. Slurm reads those lines as options; bash treats them as comments. Comments after an #SBATCH option are fine, but never put one on the #! first line: the script then fails with bash: invalid option. This is examples/slurm/cpu_test.sh, with explanations:

#!/bin/bash -l
# The line above runs this script with bash, as a login shell (needed for modules).
#SBATCH --job-name=cpu_test           # name shown in squeue
#SBATCH --output=cpu_test_%j.out      # output file; %j becomes the job ID
#SBATCH --partition=cloud             # which partition to run on
#SBATCH --cpus-per-task=2             # CPUs (on cloud, 1 CPU = 1 core)
#SBATCH --mem=2G                      # memory for the job
#SBATCH --time=00:10:00               # time limit (HH:MM:SS); the job is killed after this

set -euo pipefail                     # stop at the first error

SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}"   # the directory you ran sbatch from
cd "$SUBMIT_DIR"

# --- Boole module setup: copy these lines as they are into your own scripts ---
unset -f module ml 2>/dev/null || true
if [[ -r /etc/profile.d/lmod.sh ]]; then
  source /etc/profile.d/lmod.sh
elif [[ -r /usr/share/lmod/lmod/init/bash ]]; then
  source /usr/share/lmod/lmod/init/bash
fi
module purge
module unuse /opt/spack/spack/share/spack/lmod/linux-ubuntu22.04-x86_64/Core
module use /opt/spack/spack/share/spack/lmod/linux-ubuntu24.04-x86_64/Core
# --- end of module setup ---

module load python/3.11.9             # the software this job needs

echo "Running on host: $(hostname)"
...                                   # your commands

Download cpu_test.sh

The module lines are explained in Loading Software with Modules.

From ~/hpc_examples, submit the script:

cd ~/hpc_examples
sbatch examples/slurm/cpu_test.sh

Slurm replies with the job ID:

Submitted batch job 19086

Use that number wherever these guides write <jobid>, without the < >. Check on the job:

squeue -u "$USER"
JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
19086     cloud cpu_test      you PD       0:00      1 (Priority)

The ST column is the state: PD means pending (waiting for resources; the reason is in the last column), and R means running. When the job finishes it disappears from squeue. This job takes about a second, so it is often gone by the time you look. Then read its output file, which appears in the directory you submitted from:

cat cpu_test_19086.out
Running on host: cpt2.boole.cloudcix.com
Job ID: 19086
Submit directory: /home/<you>/hpc_examples
Allocated CPUs: 2
...
Hello from Slurm on cpt2.boole.cloudcix.com.
This task can use 2 CPU core(s).

If you lose track of a job ID, ls -t *.out | head lists the newest output files, and sacct -u "$USER" -S today lists your jobs from today.

Options given to sbatch on the command line override the #SBATCH lines in the script, so you can try different requests without editing it:

sbatch --cpus-per-task=4 examples/slurm/cpu_test.sh

Monitoring and Cancelling Jobs

Command

What it does

squeue -u $USER

Your pending and running jobs.

scancel <jobid>

Cancel a job.

scontrol show job <jobid>

Full details of a pending or running job.

sacct -j <jobid>

Record of a finished job (state, run time, memory).

sacct shows several lines per job: one for the whole job, then one per step. .batch is your script, .extern is Slurm’s own bookkeeping, and each srun inside the script adds a numbered step (.0, .1, …). Memory use (MaxRSS) appears on the step lines, not the first line.

To check whether a finished job used the CPUs and memory it reserved, see 9. Was My Job Efficient?.

Batch GPU Job

Request GPUs with --gres=gpu:<N>. Each physical-gpu node has one GPU, so ask for --gres=gpu:1. Interactive session:

srun --partition=physical-gpu --gres=gpu:1 --cpus-per-task=4 --mem=16G --time=00:15:00 --pty bash

The batch example examples/gpu/gpu_test.sh prints nvidia-smi (the GPU and its driver) and nvcc --version (the CUDA compiler) from the GPU node:

#!/bin/bash
#SBATCH --job-name=gpu_test
#SBATCH --output=gpu_test_%j.out
#SBATCH --partition=physical-gpu
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=00:10:00

set -euo pipefail

SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}"
cd "$SUBMIT_DIR"

echo "Running on host: $(hostname)"
echo "Job ID: ${SLURM_JOB_ID:-not-running-under-slurm}"
echo "Submit directory: ${SUBMIT_DIR}"

nvidia-smi
nvcc --version

python3 - <<'PY'
print("GPU allocation test complete. Add your CUDA or ML workload here.")
PY

Download gpu_test.sh

sbatch examples/gpu/gpu_test.sh
cat gpu_test_<jobid>.out
|   0  NVIDIA A100-SXM4-80GB          Off |   00000000:08:00.0 Off |                    0 |
...
Cuda compilation tools, release 12.0, V12.0.140

To choose how many CPUs and how much memory to request with a GPU, and to check that the GPU is kept busy, see Getting the Most from Boole’s GPUs.

Parallel Jobs and MPI

To use more than one core, or more than one node, see Parallel Computing on Boole: Cores, Nodes and MPI. It covers:

  • How cores, hyperthreads and nodes map to Slurm options on each partition.

  • Using many cores on one node, and running many jobs with job arrays.

  • MPI, scaling studies, hybrid jobs, and checking whether a job used what it asked for.

Email Notifications

Slurm can email you when a job starts, ends, fails, or is cancelled. Add these options to your srun command or sbatch script:

Option

Description

--mail-type=TYPE

Event or events that trigger an email: BEGIN (job starts), END (job finishes), FAIL (job fails), CANCEL (job is cancelled), or ALL. Combine with commas, e.g. END,FAIL.

--mail-user=EMAIL

Email address to send notifications to.

For an interactive session:

srun --partition=cloud --cpus-per-task=1 --mem=1G --time=00:15:00 \
     --mail-type=ALL --mail-user=your.name@example.org --pty bash

examples/slurm/notify_test.sh waits a minute and emails you when it ends:

#!/bin/bash
#SBATCH --job-name=notify_test
#SBATCH --output=notify_test_%j.out
#SBATCH --partition=cloud
#SBATCH --cpus-per-task=1
#SBATCH --mem=1G
#SBATCH --time=00:05:00
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=you@example.com   # replace on the command line: sbatch --mail-user=...

set -euo pipefail

SUBMIT_DIR="${SLURM_SUBMIT_DIR:-$PWD}"
cd "$SUBMIT_DIR"

echo "Job running on $(hostname)"
echo "Submit directory: ${SUBMIT_DIR}"
echo "You should get an email when this job ends."
sleep 60
echo "Job finished."

Download notify_test.sh

Give your own address on the command line:

sbatch --mail-user=your.name@example.org examples/slurm/notify_test.sh