HPC Grid Tutorial: Introduction to SlurmShare & Print
This knowledge article will give an overview of Slurm, new job submission requirements as well as commands and examples, and PBS to Slurm equivalents to convert jobscripts.
We have just completed a very major upgrade to our scheduling software from PBS to Slurm. Although this will prove to benefit us greatly in the future, we must first adapt to it in the present.
Slurm is a modern job-scheduler with capabilities that are compatible with the WSU Grid's heterogeneous hardware. It is an open source, fault-tolerant, and highly scalable cluster management, and job scheduling system for large and small Linux clusters. It allocates exclusive and/or non-exclusive access to resources (compute nodes) to users for some duration of time so they can perform work. It provides a framework for starting, executing, and monitoring work on the set of allocated nodes. It arbitrates contention for resources by managing a queue of pending work.
Slurm differs from PBS in its commands to submit and monitor jobs, syntax to request resources and how environment variables behave. In Slurm, sets of compute nodes are called partitions rather than queues (PBS). Resources are classified into QoS’s. A QOS is a classification that determines what kind of resources your job can use. Users can specify certain features of nodes utilizing the constraint directive. A job is given an allocation of resources to run. Jobs spawn steps, which are allocated resources from within the job's allocation.
Here are some basic commands that will get you up and running with Slurm:
Provides freenode, partition, and QoS information
Still works and provides information from squeue now
Submit a job to the batch queue system, e.g., sbatch myjob.sh, where myjob.sh is a SLURM job script.
Submit an interactive job to the batch queue system, e.g., srun --pty bash, will begin an interactive shell.
Cancel a job, e.g., scancel 123, where 123 is a job ID.
Check current jobs in the batch queue system.
Key options to set when submitting your jobs
When submitting a job, the two key options required are the QoS and a maximum time limit for your job.
A QoS is a classification that determines what kind of resources your job can use. Our QoS are as follows:
This contains our fastest publicly available nodes. It has a core limit of 512 cores
This contains our old equipment. Slow nodes that may still be useful for high throughput jobs, many core jobs, or high latency mpi jobs.
Gpu may be found here.
If you own equipment, you can submit your jobs here and kick everyone else off them.
If you have filled your allocations and want some space to work on stuff, then you can use the resources here.
Unlimited use. Your jobs might get killed and requeued though. Single node only.
A maximum time limit for the job is required under all conditions. Jobs submitted without providing a time limit will be rejected by the scheduler. This can be specified with the following directive:
Acceptable time formats include "minutes", "minutes:seconds", "hours:minutes:seconds", "days-hours", "days-hours:minutes" and "days-hours:minutes:seconds".
Sample Simple Job Script
Submit with the following command "sbatch -q primary simple.sh"
#SBATCH --job-name Simple
#SBATCH -q primary
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -o output_%j.out
#SBATCH -e errors_%j.err
#SBATCH -t 1-:0:0
Sample MPI Job Script
Submit with the following command "sbatch -q secondary MPI.sh
#SBATCH --job-name MPI
#SBATCH -q secondary
#SBATCH -N 12
#SBATCH -n 12
#SBATCH -o output_%j.out
#SBATCH -e errors_%j.err
#SBATCH -t 7-0:0:0
Interactive Job: srun -q debug -t 10:0 --pty bash
PBS to Slurm Equivalents
Job Submission - Interactive
srun --pty bash
Job Status (by job)
Job Status (by user)
qstat -u [user_name]
squeue -u [user_name]
qstat -f [job_id]
scontrol show job [job_id]
scontrol hold [job_id]
scontrol release [job_id]
squeue -p [partition]
sinfo -N OR scontrol show nodes
Like PBS job scripts, Slurm has batch scripts that are submitted using the command: sbatch [script_file]. In your script rather than #PBS, you would use #SBATCH. The following table is job specifications to help convert your job scripts to Slurm.
-l ppn=[count] OR -l
Wall Clock Limit
-t [min] OR -t [days-hh:mm:ss]
Standard Output File
Standard Error File
-j oe (both to stdout) OR -j eo
(use -o without -e)
--export=[ALL | NONE | variables]
--requeue OR --no-requeue
--mem=[mem][M|G|T] OR --mem-per-cpu=[mem][M|G|T]
Account to charge
Tasks Per Node
-l mppnppn [PEs_per_node]
CPUs Per Task
Job host preference
--nodelist=[nodes] AND/OR --exclude=[nodes]
Quality Of Service
-A "YYYY-MM-DD HH:MM:
Slurm has its own environment variables just like PBS. Here are PBS environment variables and their Slurm equivalents.
Job Array Index
If you need any further assistance in converting your job scripts to Slurm, please email us at firstname.lastname@example.org.