Work through these in order. Each exercise says what to read first, what to run, and what you should see. Click Answers under an exercise to check yourself.
Get the example files. Download them into your home directory as described in Get the Example Files. Every command below assumes you are in that folder:
cd ~/hpc_examples
If you get sbatch: error: Unable to open file, you are in the wrong
folder: run cd ~/hpc_examples again.
Job IDs. sbatch replies Submitted batch job 19086. That number is
the job ID. Wherever you see <jobid>, type your number without the
< >, e.g. cat cpu_test_19086.out.
Is my job finished? squeue -u "$USER" lists your jobs that are
waiting (PD) or running (R). A finished job disappears from the list,
and its output file is in ~/hpc_examples. ls -t *.out | head shows the
newest output files.
Don’t edit the scripts. Options given to sbatch on the command line
override the #SBATCH lines in a script, so the exercises change things
that way and every script stays as it was.
The cluster is small and shared. Keep the resource requests as shown,
and exit interactive sessions when you’re done with them.
Part |
Exercises |
Time, including reading |
|---|---|---|
|
1-4 |
45 min |
|
5-7 |
40 min |
|
8-12 |
65 min |
|
13-15 |
30 min |
|
16 |
15 min |
When many people submit at once, jobs may wait a few minutes in the queue. Read the next section while you wait.
Read first, in Slurm Basics: the sections on login and compute nodes,
Partitions and interactive jobs with srun.
sinfo
scontrol show node h17u17
Questions:
Which partitions can you see, and how many nodes does each have?
Which nodes are idle (free), mix (partly used) or alloc (full)?
Where would a small CPU test job fit best?
h17u17 reports CPUTot=16. Using Sockets (processor chips),
CoresPerSocket and ThreadsPerCore, how many physical cores does it
really have?
cloud (2 nodes, the default, marked *), physical (4 nodes) and
physical-gpu (2 nodes).
It changes all the time. Check the STATE column.
cloud: small, general-purpose, and it leaves the bare-metal nodes free
for big jobs.
Sockets=2 x CoresPerSocket=4 = 8 physical cores. ThreadsPerCore=2
means each core has 2 hyperthreads, so Slurm counts 16 CPUs.
srun --partition=cloud --cpus-per-task=1 --mem=1G --time=00:15:00 --pty bash
When the prompt changes (for example to <you>@cpt2), run:
hostname
echo "SLURM_JOB_ID=$SLURM_JOB_ID"
echo "SLURM_JOB_NODELIST=$SLURM_JOB_NODELIST"
ls ~/hpc_examples
Then leave the job, which frees its CPU for others:
exit
Questions:
Did hostname change from the login node?
Can you see the same files in ~/hpc_examples from the compute node?
What job ID did Slurm assign to the interactive session?
Yes: from login1... to a compute node such as cpt2.boole.cloudcix.com.
Yes. Home directories are shared by all nodes.
The number printed for SLURM_JOB_ID. Interactive sessions are jobs too,
and appear in squeue while they run.
Read first: Your First Batch Job, which explains every line of the script. Then:
sbatch examples/slurm/cpu_test.sh
squeue -u "$USER"
The job takes about a second, so it may already be gone from squeue. Read
its output:
cat cpu_test_<jobid>.out
You should see:
Running on host: cpt2.boole.cloudcix.com
Job ID: 19086
Submit directory: /home/<you>/hpc_examples
Allocated CPUs: 2
...
This task can use 2 CPU core(s).
Now ask for 4 CPUs instead, without editing the script:
sbatch --cpus-per-task=4 examples/slurm/cpu_test.sh
Questions:
Which lines of the new output changed?
Where did the output file appear, and why there?
Allocated CPUs: 4 and This task can use 4 CPU core(s). The job ID and
possibly the node also differ.
In ~/hpc_examples, the folder you ran sbatch from (SLURM_SUBMIT_DIR).
The script’s --output=cpu_test_%j.out line names it, with %j
replaced by the job ID.
Use the job ID from Exercise 3:
sacct -j <jobid> --format=JobID,JobName,State,Elapsed,ReqMem,MaxRSS
JobID JobName State Elapsed ReqMem MaxRSS
------------ ---------- ---------- ---------- ---------- ----------
19086 cpu_test COMPLETED 00:00:01 2G
19086.batch batch COMPLETED 00:00:01 424K
19086.extern extern COMPLETED 00:00:01 0
The first line is the whole job. .batch is your script, and .extern is
Slurm’s own bookkeeping. Memory use (MaxRSS) appears on the .batch line.
To list all your jobs from today: sacct -u "$USER" -S today.
Questions:
Did the job finish successfully, and how long did it run?
How much memory did it request (ReqMem) and use (MaxRSS)?
What would you change before running hundreds of jobs like this?
COMPLETED, in about one second.
It requested 2 GB. MaxRSS shows a few MB at most. For a job this short
that is an underestimate (Slurm samples memory every 30 seconds), but
either way it is far below 2 GB.
Request much less memory (e.g. --mem=500M), and 1 CPU instead of 2: the
Python in this script only ever uses one. Smaller requests start sooner
and leave room for others. Leave some margin above the real memory use,
because a job that exceeds its memory is killed.
Read first: sections 1-4 of Parallel Computing on Boole: Cores, Nodes and MPI.
pool_pi.slurm asks for 8 whole cores on a physical node and times the
same calculation with 1, 2, 4 and 8 worker processes:
sbatch examples/multicore/pool_pi.slurm
It runs for about 20 seconds. Then:
cat pool-pi-<jobid>.out
os.cpu_count() = 16: every CPU on this node, not just yours.
This job may use: 8 physical cores (16 hyperthreads).
SLURM_CPUS_PER_TASK = 8, so this program uses up to 8 worker process(es).
workers seconds speedup efficiency error
1 9.75 1.00 100% 6.3e-13
...
8 1.26 7.73 97% 2.2e-14
Now turn off whole-core allocation, and then also ask for 16 CPUs. Each of
these holds half or all of a physical node, so if you are working in pairs,
run one each and compare:
sbatch --hint=multithread examples/multicore/pool_pi.slurm
sbatch --hint=multithread --cpus-per-task=16 examples/multicore/pool_pi.slurm
Questions:
In the first run, how close to 8x was the speedup with 8 workers?
What would happen if the program started os.cpu_count() workers?
With --hint=multithread, how many physical cores did you get, and what
happened to the 8-worker time?
With 16 CPUs, was 16 workers faster than 8?
Usually between 6.5x and 7.7x (roughly 80-97% efficiency); it varies from run to run, partly because of other jobs on the same node. The workers never talk to each other, so nearly all the time is useful work.
It would start 16 workers on 8 cores. They would compete for the cores and run no faster than 8, and on a smaller allocation (say 2 cores) much slower.
Only 4 physical cores (8 hyperthreads). 8 workers took about 2.9 s instead of about 1.3 s, more than twice as slow, because pairs of workers shared a core.
No: 16 workers were no faster than 8 (both about 1.3-1.5 s). Hyperthreads don’t add speed for this kind of work. Count cores, not CPUs.
Read first: 5. Many Independent Jobs: Job Arrays. Submit 6 tasks, at most 2 running at a time, then watch them for a minute:
sbatch --array=1-6%2 examples/multicore/array_sweep.slurm
squeue -u "$USER"
While tasks are still waiting, cancel the last one:
scancel <jobid>_6
Run squeue -u "$USER" again every 15 seconds or so. When all the tasks are
gone, collect the last line of each task’s output:
tail -qn1 array-sweep-<jobid>_*.out
Questions:
How many tasks ran at the same time, and what reason did squeue give
for the others?
How many output files did you get, and why?
How does the error (last column) change each time n grows ten times?
Two, because of the %2 limit. The others showed PD with reason
JobArrayTaskLimit, as one line such as 19089_[3-6%2]. If the cluster
is busy you may see Resources or Priority first: then even the first
two are waiting for a free CPU.
Five: task 6 was cancelled before it started, so it never wrote a file.
It shrinks about a hundred times (8.3e-04, 8.3e-06, …), until it
reaches the limit of floating-point rounding around 1e-14.
Read first: 9. Was My Job Efficient?. Submit a job that holds 8 cores but only uses one:
sbatch examples/multicore/pool_pi.slurm --workers 1
(Arguments after the script name are passed to the program, here
pool_pi.py.) When it has finished, compare it with your first job from
Exercise 5. Give both job IDs, separated by a comma:
sacct -j <ex5_jobid>,<ex7_jobid> --format=JobID,JobName,AllocCPUS,Elapsed,CPUTime,TotalCPU,MaxRSS,State
If you’ve lost a job ID, sacct -u "$USER" -S today lists all your jobs from
today.
CPUTime is what the job reserved (Elapsed x AllocCPUS). TotalCPU is
what it actually used. Read both from the first line.
Questions:
What CPU efficiency (TotalCPU / CPUTime) did each job get?
Why is even the normal sweep from Exercise 5 well below 100%?
The script requests 4 GB. Was that close to MaxRSS?
About 12% for --workers 1 (about 10 s used out of 80 s reserved),
and about 25-30% for the normal sweep.
The sweep spends most of its time on the 1- and 2-worker rounds, when most of the 8 cores are idle. A job’s efficiency is dragged down by its serial parts.
No: MaxRSS was only a few MB. That is an underestimate for a job this
short, but a real job of this kind could still ask for far less memory.
Read first: 6. Many Nodes: MPI.
sbatch examples/mpi/mpi_demo.slurm
cat mpi-demo-<jobid>.out
At the end of the output, each rank prints its node and the CPUs it is pinned to:
Hello from rank 0 of 8 on h17u22.boole.cloudcix.com (cpus [1])
Hello from rank 1 of 8 on h17u22.boole.cloudcix.com (cpus [3])
...
The lines arrive in any order. Now try two other layouts:
sbatch --ntasks-per-node=8 examples/mpi/mpi_demo.slurm
sbatch --hint=multithread examples/mpi/mpi_demo.slurm
Questions:
In the first run, how many ranks ran on each node?
With --ntasks-per-node=8, how many ranks are there in total?
With --hint=multithread, look at the cpus of the ranks on one node.
On these nodes, CPU N and CPU N+8 are the same physical core. Do any
ranks share a core?
4 per node, on 2 nodes: --nodes=2 --ntasks-per-node=4 = 8 ranks.
16 (2 nodes x 8), each on its own core, CPUs 0-7.
Yes. The ranks land on CPUs such as 0, 8, 2, 10. 0 and 8 are one core,
and 2 and 10 are another, so 4 ranks get only 2 cores.
Read first: 7. Scaling: Does More Cores Mean Faster?.
Each scaling run uses all four physical nodes, the whole partition, so
read the sample output in 7. Scaling: Does More Cores Mean Faster? (the same tables these runs
produce) and only run them yourself when the cluster is quiet.
# Each of these uses the whole physical partition:
sbatch examples/mpi/mpi_scaling.slurm
sbatch examples/mpi/mpi_scaling.slurm mpi_heat.py
sbatch examples/mpi/mpi_scaling.slurm mpi_heat.py --n 200000
Questions:
Is the value column the same on every row? Why does that matter?
Does either program get close to the ideal 32x speedup on 32 ranks?
Some efficiencies for mpi_heat.py are above 100%. How can more ranks
be more than proportionally faster?
For the small heat problem (--n 200000), what is the best number of
ranks? What happens when the job spreads to a second node, and why?
Yes. It shows the parallel program still computes the same answer. A fast but wrong answer is worthless.
Both do. mpi_pi.py reaches about 31x because its ranks only talk twice.
The big mpi_heat.py problem even beats 32x (see question 3).
Spread over more nodes, each node’s share of the data shrinks until it fits in the processors’ fast cache, so each rank works faster than the single rank did.
About 4-8 ranks on one node. On 2 or more nodes it gets slower: each step has little work but still exchanges messages, and a message between nodes takes about 30 times longer than within a node.
Read first: 8. Hybrid MPI and Threads.
Each run uses 2 physical nodes, so if you are working with others,
split these between you and compare results:
sbatch examples/mpi/mpi_hybrid.slurm # 4 ranks x 4 threads
sbatch --ntasks-per-node=8 --cpus-per-task=1 examples/mpi/mpi_hybrid.slurm # 16 ranks x 1 thread
sbatch --ntasks-per-node=1 --cpus-per-task=8 examples/mpi/mpi_hybrid.slurm # 2 ranks x 8 threads
sbatch --ntasks-per-node=1 --cpus-per-task=8 examples/mpi/mpi_hybrid.slurm --chunk 262144
The seconds= value on the RESULT line is the run time.
Questions:
Check the cpus lists: did each rank get the cores you expected?
Which layout was fastest?
Why were 8 threads per rank so slow, and why did --chunk 262144 help
them?
Yes: 4 CPUs per rank, 1 per rank, or 8 per rank respectively.
16 ranks x 1 thread (about 0.5 s), close to the ideal of 8.0 s / 16.
Python’s GIL. Each thread needs it back after every numpy call. With small chunks that happens thousands of times a second, so the threads mostly wait. Bigger chunks mean fewer handovers, so the threads overlap more, but pure MPI was still faster.
Read first: Running OpenFOAM in Parallel on Boole.
Run the lid-driven cavity on 8 ranks of one node:
sbatch examples/openfoam/cavity.slurm
It takes about 10 seconds once it starts. Then:
cat openfoam-cavity-<jobid>.out
ls openfoam-cavity-<jobid>/ranks-8
Mesh 200 x 200 = 40000 cells, 100 time steps. Cases in /home/<you>/hpc_examples/openfoam-cavity-<jobid>
8 ranks on 1 node(s): 7.3 s (Courant Number mean: 0.100853 max: 0.49187)
Now run a mesh four times smaller:
sbatch --export=ALL,MESH=100 examples/openfoam/cavity.slurm
The scaling study takes all four physical nodes, the whole partition, for
about 10 minutes, so read its tables under How Well Does It Scale? in
Running OpenFOAM in Parallel on Boole and only run it yourself when the cluster is quiet.
# These use the whole physical partition for about 10 minutes:
sbatch --nodes=4 examples/openfoam/cavity.slurm 1 2 4 8 16 32
sbatch --nodes=4 --export=ALL,MESH=300 examples/openfoam/cavity.slurm 1 2 4 8 16 32
Questions:
What is in the ranks-8 folder, and why are there 8 processor folders?
How many cells did each rank get with MESH=100? Was it much faster
than with 200?
In the scaling tables, why did 16 ranks on 2 nodes run slower than 8 ranks on 1 node?
You have a 2-million-cell case. How would you choose how many ranks and nodes to use?
The case: 0 (start), 0.025 (the result after 100 time steps),
constant, system, a log for each step, and processor0 to
processor7. decomposePar split the mesh into one piece per MPI rank,
and each rank read and wrote only its own piece.
10,000 cells / 8 ranks = 1,250 each. It took about 2 seconds instead of about 7: four times fewer cells, but not four times less communication.
The pressure solver exchanges data hundreds of times per time step, and each message between nodes takes about 30 times longer than within a node. The extra waiting outweighs the extra cores.
Measure: run a few time steps on 8 ranks (one node) and on 16 (two nodes), and keep the fastest that is still efficient. At 2 million cells you’d have 250,000 cells per rank on one node, so a second node might help, but on Boole today, check before assuming.
Copy the pi program:
cp examples/mpi/mpi_pi.py examples/mpi/my_pi.py
Open it in an editor (nano examples/mpi/my_pi.py; save with Ctrl+O, Enter,
and quit with Ctrl+X). Replace the comm.reduce(...) line with point-to-point
messages. Every rank except 0 sends its local sum to rank 0 with
comm.send(local, dest=0). Rank 0 adds its own local to
comm.recv(source=r) for each other rank r. Test it on one node with 1, 2,
4 and 8 ranks:
sbatch --nodes=1 examples/mpi/mpi_scaling.slurm my_pi.py --n 100000000
Check that the value column is still pi on every row. Then think: why do
MPI libraries provide reduce anyway?
if rank == 0:
pi = local + sum(comm.recv(source=r) for r in range(1, size))
else:
comm.send(local, dest=0)
pi = None
reduce is shorter and harder to get wrong. It is also faster on many
ranks: rank 0 above receives size - 1 messages one after another, while
reduce combines partial sums in a tree, which takes about log2(size)
steps.
Read first: Getting the Most from Boole’s GPUs.
There are only 2 GPUs, so take turns. First check you can reach one:
sbatch examples/gpu/gpu_test.sh
cat gpu_test_<jobid>.out
You should see an NVIDIA A100-SXM4-80GB in the nvidia-smi table, and
Cuda compilation tools, release 12.0 from nvcc.
The data-loader sweep holds one of the two GPUs for about a minute, plus
queueing, so read its output under How Many CPUs Does a GPU Job Need? in
Getting the Most from Boole’s GPUs and only run it yourself when a GPU is free. It first needs
bash examples/gpu/setup_torch_env.sh (about 3 minutes and 5.5 GB in your
home directory).
# Each of these holds a GPU for about a minute:
sbatch examples/gpu/gpu_dataloader.slurm
sbatch --cpus-per-task=4 examples/gpu/gpu_dataloader.slurm
Questions:
With 0 data-loader workers, how busy was the GPU, and why?
How many CPUs should a job like this request, and why not 64?
What happened with only 4 CPUs?
About 15%. Every batch was prepared by the main process on one CPU core, and the GPU waited for it most of the time.
About 9: 8 workers plus the main process. Throughput stopped rising at 8 workers (15 were no faster), so more CPUs would sit idle. The GPU node’s other cores and memory can then run other people’s jobs.
With 3 workers it reached about 7,300 images/s, less than half the speed with 8. Too few CPUs leave the GPU waiting.
Read first: Running Docker Images with Apptainer.
sbatch examples/containers/apptainer_hello.slurm
cat apptainer-hello-<jobid>.out
ls -lh hello-world_latest.sif
The output includes Hello from Docker!. The job converted the Docker image
into hello-world_latest.sif in ~/hpc_examples, and the next run reuses that
file instead of downloading it again. The WARNING: passwd file doesn't
exist lines are harmless.
Read first: Interactive Apps in Open OnDemand: Ask for What You Use.
In Open OnDemand, launch Jupyter Notebook with Partition Cloud Compute, 1 CPU core, 2 GB RAM and 1 hour.
When it starts, connect, open a new notebook and run:
import os
print(os.cpu_count(), len(os.sched_getaffinity(0)), os.environ.get("SLURM_JOB_ID"))
Open a terminal (in Jupyter: File → New → Terminal) and run
squeue -u $USER.
Go back to Open OnDemand, open My Interactive Sessions, and Delete the session.
In any terminal on Boole:
sacct -u $USER -S today --name=sys/dashboard/sys/Jupyter \
--format=JobID,JobName%30,AllocCPUS,Elapsed,CPUTime,TotalCPU,State
Questions:
How many CPUs does the node have, and how many could your notebook use?
What did your session look like in squeue?
What was the session’s CPU efficiency (TotalCPU / CPUTime)? Why is it
low, and what does that tell you about how much to ask for?
os.cpu_count() shows the whole node (8 on a cloud node), but the
affinity count is the 1 CPU you asked for.
A running job (R) named sys/dash..., the start of
sys/dashboard/sys/Jupyter. The session is an ordinary Slurm job.
Very low, probably a few percent. Most of an interactive session is reading and typing, so its cores sit idle. Ask for the minimum, and move heavy work into batch jobs.
Read first: Job Script Templates.
Run one small program on many files using the job-array template. First make five sample input files and a list of them:
mkdir -p mydata
for i in 1 2 3 4 5; do seq $((i * 100000)) > mydata/numbers_$i.txt; done
ls mydata/*.txt > inputs.txt
cat inputs.txt
Copy the template and open it in an editor:
cp templates/array_files.slurm my_array.slurm
nano my_array.slurm
Leave the other CHANGE lines as they are for now. Go to the last line,
python3 my_script.py "$INPUT", delete it with Ctrl+K, and type this
instead, which counts the lines of each file:
wc -l "$INPUT"
Save (Ctrl+O, Enter) and quit (Ctrl+X). Then submit one task per line of
inputs.txt:
sbatch --array=1-$(wc -l < inputs.txt) my_array.slurm
When the tasks have finished:
cat my-array-<jobid>_*.out
Questions:
How many array tasks ran, and what did each one print?
How would you run your own analysis script on 200 data files, at most 10 at a time?
Five, one per line of inputs.txt. Each printed its task number and file,
then the line count, e.g. Task 3: mydata/numbers_3.txt and
300000 mydata/numbers_3.txt.
List the files (ls data/*.csv > inputs.txt), put your command on the
last line of the template (e.g. python3 analyse.py "$INPUT"), set the
memory and time for one file, and submit with
sbatch --array=1-$(wc -l < inputs.txt)%10 my_array.slurm.