SLURM Commands

For a general introduction to using SLURM, watch the video tutorial.

Job Submission

Use the Script Generator to check for syntax. Each #SBATCH line contains a parameter that you can use on the command-line (e.g. --time=1:00:00).

sbatch is used to submit jobs, just like qsub in Torque/PBS. The output is sent by default to a file in your local directory: slurm-$SLURM_JOB_ID.out. A simple sbatch invocation example is: sbatch somefilename

If you are still using qsub, qsub --show plus the normal parameters and submission script you use can be used to show equivalent sbatch syntax. Note that a temporary filename is appended to the end that looks like "/tmp/qsub_wrapper_ia78HgaO". Omit that parameter.

Environment variables:

  • SLURM_JOB_ID - job ID
  • SLURM_SUBMIT_DIR - the directory you were in when sbatch was called
  • SLURM_CPUS_ON_NODE - how many CPU cores were allocated on this node

If you need a PBS-style nodefile, add this line to your submission script:
export PBS_NODEFILE=`/fslapps/fslutils/generate_pbs_nodefile`

If your job fails to submit, remember that you must specify a memory request for your jobs (e.g. --mem-per-cpu=1G). See the Script Generator for examples.

Job/Queue Management

squeue is used to show the queue. Useful options:

  • -l ("l" for "long"): gives more verbose information
  • -u someusername: limit output to jobs by someusername
  • -A someprofessor: limit output to jobs by someprofessor's research group
  • --state=pending: limit output to pending (i.e. queued) jobs
  • --state=running: limit output to running jobs

scancel is used to cancel (i.e. kill) a job. "scancel 1234" cancels job 1234. "scancel -u myusername9" cancels all jobs by myusername9. "scancel -u myusername9 --state=pending" cancels all pending jobs by myusername9. "scancel -u myusername9 --state=running" cancels all running jobs by myusername9.

scontrol show job is used to display job information for pending and running jobs. "scontrol show job 1234" shows information for job 1234. Add "-d" to see extra output. "-dd" shows extra output and the job script that was submitted.

scontrol hold holds a job. Pass it a job ID (e.g. "scontrol hold 1234").

scontrol release releases a held job. Pass it a job ID (e.g. "scontrol release 1234").

sacct shows current and historical job information. Important options:

  • -S from_date: Show jobs that started on or after from_date. There are several valid formats, but the easiest is probably "MMDD". See "man sacct" for more options.
  • -l ("l" for "long"): gives more verbose information
  • -u someusername: limit output to jobs by someusername
  • -A someprofessor: limit output to jobs by someprofessor's research group
  • -j jobid: specify a particular job to examine
  • -o formatoptions: see "man sacct" for more fields to examine; there are a lot

Launching tasks within a job

OpenMPI has SLURM support built in, thus no special effort is required to launch under SLURM. If the program you use requires a PBS-style nodes file (a line with the hostname of each allocated node, with the number of hostname entries per host equal to the number of processes allocated on that node), add the following line to your submission script:
export PBS_NODEFILE=`/fslapps/fslutils/generate_pbs_nodefile`

srun is a command that is similar to pbsdsh. It launches tasks under SLURM on allocated resources in what are called "job steps". Think of them as sub-allocations. sbatch reserved an allocation of a certain amount of processors, nodes, memory, etc. srun is able to launch tasks within that allocation. Think of sbatch as the lessee of a house(s) who sublets a specified number of rooms to several tenants (srun instances).

By default srun launches the same command in parallel on all nodes then blocks until completion. For example, "srun hostname" will be launched once for each processor you are assigned. If you requested multiple processors, "srun -n2 hostname" will only show two lines because it only ran on two processors. srun will not overschedule your allocation of resources, so if you launch three instances of "srun -n2" that are still running but only requested six processors for your job, a fourth instance of "srun -n2" will block until one of the other instances completes. By backgrounding instances of srun, it is possible to have a simple queueing system within your job allocation. Just be sure to call "wait" or similar after backgrounding everything or your job will exit prematurely.