HPC Grid Tutorial: How to Utilize Resources

We offer a variety of resources for Grid users. For a list of the nodes and their respective resources visit the Available Nodes page. 

These resources can be requested in the select statement. In the PBS User Guide, page 54, this is referred to as requesting resources in chunks. A chunk refers to the value of each resource in a set of resources which are to be allocated as a unit to a job. Only host-level resources can be requested. You can request chunk resources using any of the following:

  • The qsub -l select=[N:][(chunk spefication)][+[N:](chunk specification)] option
  • A #PBS -l select=[N:][(chunk specification)][+[N:](chunk specification)] directive

A single chunk can be requested. For example, one chunk requested with 3 CPUs and 6GB of memory:

-l select=ncpus=3:mem=6gb

Multiples of a chunk can be requested by prefixing it by the number of chunks desired. For example, eight of the previous chunk:

-l select=8:ncpus=3:mem=6gb

Different chunks can be requested by concatenating the chunks with the plus sign '+'. There are no spaces allowed between chunks and you must specify all your chunks in a single select statement. For example, two sets of chunks with one set of 8 chunks having 3 CPUs, and one set of 4 chunks has 8 CPUs, and both have 6GB of memory: 

-l select=8:ncpus=3:mem=6gb+4:npcus=8:mem=6gb

 

There are several resources that can be requested in the select statement:

  • cpu_model: The model of the processors (case sensitive)
    • 4332HE
    • 4332HE
    • 6126
    • 6142
    • 6148
    • 6226
    • 6240
    • 6282SE
    • E5-2620v3
    • E5-2643
    • E5-2665
    • E5-2680v2
    • E5-2690
    • E5-2690v4
    • E5-2695
    • E5-2697v3
    • E5-4627v4
    • X5550
    • X5650
    • X5670
  • cpu_speed: The speed of the processor in GHz
    • 2.1
    • 2.4
    • 2.6
    • 2.7
    • 2.8
    • 2.9
    • 3.0
    • 3.3
  • cpu_type: The type of processor (case sensitive)
    • AMD
    • Intel
  • mem: The amount of memory (RAM) needed in GB (Default is 1GB) 
    • 12
    • 16
    • 61
    • 64
    • 93
    • 96
    • 127
    • 128
    • 192
    • 256
    • 259
    • 291
    • 292
    • 512
    • 768
    • 1536
    • 1579
  • ncpus: The number of cores needed (Default is 1 core)
  • net: The speed of network needed, 1Gb or 10Gb Ethernet, or 56Gb or 100Gb Infinibad (case sensitive)
  • ngpus: The number of GPU needed
  • vmem: The amount of virtual memory (real memory and swap) needed
  • mpiprocs: The number of MPI processes per node

 

Requesting Large Amount of Memory

Go to the Available Nodes page and sort by Memory. 

There are two sets of nodes available in the wsuq qith 128GB of RAM, request 122GB leaving 6GB for the system.

#PBS -q wsuq

#PBS -l select=1:mem=122GB

There are three sets of nodes available in the mwsuq with 256GB of RAM, request 250GB leaving 6GB for the system.

#PBS -q mwsuq

#PBS -l select=1:mem=250GB

 

Requesting the Fastest Processors

Go to the Available Nodes page and sort by Speed. 

There are 160 nodes in the wsuq with 3.0GHz processors, request 3.0. 

#PBS -q wsuq

#PBS -l select=1:cpu_speed=3.0

There are two sets of nodes available in the mwsuq with 3.3 GHz processors, request 3.3. 

#PBS -q mwsuq

#PBS -l select=1:cpu_speed=3.3

 

Requesting Many Cores

Go to the Available Nodes page and sort by Cores. 

There are 42 nodes available in the wsuq with 64 cores, request 64 ncpus and 60 mpiprocs leaving 4 for the system. 

#PBS -q wsuq

#PBS -l select=1:ncpus=64:mpiprocs=60

 

Requesting a Specific Network

Go the Available Nodes page and sort by Network.

There are many nodes with 10Gb Ethernet, request 10e. 

#PBS -l select=1:net=10e

 

Requesting Specific Nodes

Go to the Available Nodes page and determine what nodes you want. 

For example, requesting 4 nodes each of 64 cores in the wsuq connected over 10Gb Ethernet.

#PBS -q wsuq

#PBS -l select=4:ncpus=64:mpiprocs=60:net=10e

Or 3.0 GHz nodes in the mwsuq connected over 1Gb Ethernet.

#PBS -q mwsuq

#PBS -l select=20:cpu_speed=3.0:net=1e

Or only nodes with AMD 2354 processors.

#PBS -l select=20:cpu_model=2354

Or wsu100 and wsu101 with 12 cores on each.

#PBS -l select=2:npcus=12:mpiprocs=24:vnode=wsu100+vnode=wsu101

 

Requesting a Specific Node

Go to the Available Nodes page and determine what node you want. 

For example, requesting a particular node like zfh14. The memory won't be specified so 1GB of memory will be allocated. 

#PBS -l select=1:vnode=zfh14