Feature-based Scheduling

Feature-based scheduling is a new scheduling approach at the Fulton Supercomputing Lab, which will allow greater utilization, and improved access to resources. It will allow the users to specify exactly what resources they need, allowing the scheduler software flexibility in determining the best resources to run on.

In the past, for example, users without any special resource requests were still required to choose a particular queue, which meant that the job was committed to a particular type of hardware, even when the user did not really care about the specific hardware. The queue structure creates hard boundaries between resource groups, no crossing of boundaries was allowed. However, many times, the users simply didn't care what resources they ran on, as long as they ran. As a result, the feature-based scheduling approach allows the user to specify exactly which resources or features they need, and not specify anything they don't care about.

Similarly, users have only needed a few processors, but a great deal of memory. Since all scheduling was done by processors and time, the best we could recommend was to request more processors, to get the associated memory. This is no longer the case. You can now request the amount of memory separate from the number of processors.

General Guidelines

As a general rule, users should request the hardware or features that they need, and not request anything else. The more resources that are requested, the more restricted the scheduler has to be, and the longer it will take to schedule the job. Just for clarity, we'll say this again:

Request the fewest possible resources for your job, in order to minimize the waiting time for the job to start.

What else does this change?

Feature-based scheduling is much more versatile than the old queue based system, which it replaced. We recommend that users learn the feature scheduling approach as soon as possible. If you have any old scripts that used the queue based system (eg. "cuda" or "anynode"), these have been deactivated, and jobs requesting them will not run.

What features are available for scheduling, and how do I use it?

Features that can be used for scheduling fit generically into two major categories: Node-specific features, and Job-specific features, and each of these has it's own syntax. Note that not all combinations of features are available at this time - for example requesting AMD processors with Infiniband - since the lab does not currently have the specific hardware to fulfill that request.

Node-specific features

Resources and features that are node-specific are listed as a part of the resource request, separated by a colon (":"). Most users are already familiar with this for the use of the ppn or processors-per-node request. You may add as many of the following features to your request as you wish.

Note that there is an alternative "feature=" syntax also available. This is more flexible, but most users will not need that level of flexibility. This syntax alternative is documented in the next section.

Alternative Syntax for Node-specific features

Instead of the colon-sparated syntax as shown above, it is also possible to use a "feature=" syntax, which allows more flexibility in choosing and excluding necessary features. Examples are shown here, which include some more advanced examples. Note that because some symbols (eg. ! and |) have special meaning in the shell environment, they must be quoted as shown here to have the desired effect.

Feature name Description Syntax example
nodes Requests a specific number of nodes, or processing hosts #PBS -l nodes=3:ppn=4
ppn Requests a specific number of processors per node #PBS -l nodes=1:ppn=4
amd Requests that the nodes have AMD processors without caring which specific processor model #PBS -l nodes=1:ppn=1:amd
ib Requests that the nodes have access to an Infiniband interconnect #PBS -l nodes=1:ppn=1:ib
Description Syntax example
Requests that the nodes have Intel processors without caring which specific processor model #PBS -l nodes=1:ppn=1,feature='intel'
Requests nodes that have intel processors, and infiniband (ib) #PBS -l nodes=2:ppn=1,feature='intel:ib'
Requests nodes that have intel processors, and do NOT have infiniband (ib) #PBS -l nodes=2:ppn=1,feature='intel:!ib'
Requests that nodes have either Nehalem or Westmere processors (doesn't matter which) #PBS -l nodes=1:ppn=1,feature='nehalem|westmere'
Requests that nodes have any kind of Intel processor, except not Harpertown #PBS -l nodes=1:ppn=1,feature='intel:!harpertown'

Job-specific features

Job-specific features apply to the job as a whole, and are listed in the resource request, separated by a comma (","):

Feature name Description Syntax example Note
walltime Requests a total amount of running time for the job #PBS -l nodes=1:ppn=4,pmem=2gb,walltime=1:00:00 Format is walltime=hh:mm:ss; if not specified, the system will assume 1 hour
mem Requests a total amount of memory requested by all processes in the job #PBS -l nodes=1:ppn=4,mem=16gb This translates into an even division of memory per process (4gb per process in the example)
pmem Requests a specific amount of memory per process #PBS -l nodes=1:ppn=4,pmem=4gb This is equivalent to the "mem=16gb" listed above
qos Requests a specific QOS, or quality of service. This is used primarily for preemption on BYU's supercomputers #PBS -l nodes=1:ppn=4,pmem=2gb,qos=preemptee The example specifies that this job is preemptable

Additional Examples

The following examples combine several of the above requirements, and include explanation of the specifics.

Syntax Example Explanation
#PBS -l nodes=1:ppn=4,pmem=2gb:ib:intel,walltime=4:30:00 Requesting 4 processors on 1 node, with Infiniband, and Intel processors, for 4.5 hours
#PBS -l nodes=16:ppn=8,pmem=2gb:nehalem:ib+2:ppn=4:gpu,walltime=168:00:00,qos=preemptee Requesting 16 nodes with 8 Intel Nehalem processors each, with Infiniband, plus 2 additional nodes with 4 processors each (any kind) with GPGPU's attached, for 168 hours (7 days), and the preemptee QOS

What resource requests are required?

The purpose of the feature scheduling approach is to allow users to request only the resources or features they care about, and not specify the features they don't care about. However, we recommend as a minimum the use of nodes, ppn, and walltime as shown below. If these are not specified, the system has some defaults it will apply, which may not be what you intended. At the time of this writing, the defaults are 1 node, 1 processor per node, and 1 hour of walltime.

#PBS -l nodes=10:ppn=8,pmem=2gb,walltime=24:00:00