Boole has two GPU nodes, gpu0 and gpu1, in the physical-gpu
partition. Each has:
one NVIDIA A100 GPU (80 GB),
64 CPU cores,
480 GB of RAM.
This guide shows how to size a GPU job so the GPU stays busy, without holding resources it doesn’t need. All the numbers come from real runs on Boole.
The commands below are run from the examples folder, ~/hpc_examples. See
Get the Example Files to download it.
Each GPU node has one GPU, but also the cluster’s biggest CPUs and memory.
physical-gpu accepts CPU-only jobs too, so whatever a GPU job doesn’t
reserve can run someone else’s work.
In 30 days of Slurm records, jobs on the GPU nodes reserved 48,000 CPU-hours (one core for one hour each) but used only 6% of them. Most of that came from jobs asking for 30-60 cores and 200-400 GB alongside a single GPU. Those cores and memory sat idle, and nobody else could use them.
There is no PyTorch module on Boole, and the GPU nodes have no module
command. Instead, create a Python virtual environment (“venv”): a
folder in your home directory holding its own Python packages, which you
switch on with source <folder>/bin/activate. Run this once, on the login
node:
cd ~/hpc_examples
bash examples/gpu/setup_torch_env.sh
Creating /home/<you>/venvs/hpc-torch and installing PyTorch. This takes a few minutes.
Done in 162 s, using 5.4G in /home/<you>/venvs/hpc-torch
PyTorch 2.14.1+cu130 built for CUDA 13.0
GPU job scripts then use it with
source "$HOME/venvs/hpc-torch/bin/activate". See the GPU template in
Job Script Templates. PyTorch brings its own CUDA
libraries (cu130 = CUDA 13.0), so the separate nvcc 12.0 on the GPU
nodes doesn’t matter for it.
Training needs data. In PyTorch, a DataLoader prepares each batch of
images (reading, decoding, augmenting) on the CPU, using num_workers
data-loader workers: separate processes, one per CPU core. With
num_workers=0 the main Python process prepares every batch itself, between
training steps. If the CPU can’t keep up, the GPU waits.
examples/gpu/gpu_dataloader.py trains a small network on images that each
take a little CPU time to prepare, like real image data. It times the same
training with 0, 1, 2, 4, 8 and 15 workers, and samples how busy the GPU is:
sbatch examples/gpu/gpu_dataloader.slurm
cat gpu-dataloader-<jobid>.out
GPU: NVIDIA A100-SXM4-80GB
CPU cores this job may use: 16
(trimmed: each row is followed by a RESULT line)
workers images/s GPU busy
0 2376 15%
1 2428 15%
2 4994 30%
4 9976 56%
8 15814 81%
15 15191 82%
Download gpu_dataloader.slurm ·
Download gpu_dataloader.py
What this shows:
With 0 or 1 workers, the GPU was idle 85% of the time, waiting for the CPU. Most GPU jobs that “run slowly” look like this.
8 workers made training 6.7 times faster, and the GPU 81% busy.
15 workers were no faster than 8. Past this point the GPU is the limit, and extra CPUs only sit idle.
So for this job, ask for 9 CPUs (8 workers plus the main Python process),
not 16, and certainly not 64. With only 4 CPUs (--cpus-per-task=4, 3
workers) it managed 7,276 images/s, less than half the speed. Too few CPUs
wastes the GPU; too many wastes the CPUs.
Your own code will have a different sweet spot. Find it the same way: run a
short training with a few num_workers values and stop adding workers when
images/s stops rising. In PyTorch:
import os
from torch.utils.data import DataLoader
workers = len(os.sched_getaffinity(0)) - 1 # CPUs given to this job, minus the main process
loader = DataLoader(dataset, batch_size=256, num_workers=workers)
The 16-CPU run above used at most 2.4 GB of RAM (MaxRSS, the peak memory
sacct reports; see below), so
--mem=8G is plenty. The GPU’s own 80 GB of memory is separate and doesn’t
need requesting: you get all of it with --gres=gpu:1.
During a job, look at the GPU from the login node:
srun --jobid=<jobid> --overlap nvidia-smi
GPU-Util is the share of time the GPU was running work. Below about 50%
often means it is waiting for data: add data-loader workers (and CPUs), or
increase batch_size in your code.
For the whole job, have it log GPU use as it runs. gpu_dataloader.slurm
and the GPU template in Job Script Templates start this in the
background:
nvidia-smi --query-gpu=timestamp,utilization.gpu,memory.used --format=csv -l 30 \
> "gpu-usage-${SLURM_JOB_ID}.csv" &
Read it afterwards as a table with column -s, -t < gpu-usage-<jobid>.csv.
After a job, check CPU and memory as for any job:
sacct -j <jobid> --format=JobID,JobName,AllocCPUS,Elapsed,CPUTime,TotalCPU,MaxRSS,State,NodeList
JobID JobName AllocCPUS Elapsed CPUTime TotalCPU MaxRSS State NodeList
------------ ---------- ---------- ---------- ---------- ---------- ---------- ---------- ---------------
19162 gpu-datal+ 16 00:01:04 00:17:04 03:16.871 COMPLETED gpu0
19162.batch batch 16 00:01:04 00:17:04 03:16.866 2371960K COMPLETED gpu0
TotalCPU / CPUTime is only 19% here, because the sweep spends part of its
time on 0-4 workers, and 15 workers were more than needed.
Jupyter and CloudCIX AI Lab can start on a GPU in Open OnDemand (see Interactive Apps in Open OnDemand: Ask for What You Use). There are only two GPUs, so:
ask for 4-8 CPU cores and 32 GB, not the maximum,
delete the session when you stop working, since a closed browser tab keeps holding the GPU,
move long training runs into a batch job.
--gres=gpu:1. There is one GPU per node, so never more.
--cpus-per-task = data-loader workers + 1, found with a short sweep.
--mem = MaxRSS from a test run plus a margin.
num_workers in your code is the CPUs you were given, minus one.
Checked that the GPU is busy (nvidia-smi, or the usage log).
Interactive GPU sessions deleted when finished.