The HPC now supports GPU jobs.  The HPC includes several processing nodes with NVIDIA GeForce GTX 1080 Ti GPU cards.  Each GPU node contains four GPUs.

GPU nodes are available on the backfill , backfill2, and genacc_q partitions. In addition, if your department, lab, or group has purchased GPU resources, they will be available in your owner-based partition.

If you expect your job to run for less than four hours, we recommend submitting it to the backfill2 partition, since this partition has access to all GPU nodes in the cluster, and your job will probably start sooner.

Submitting GPU Jobs

If you wish to submit your job to node(s) that support GPU processing, simply add the following line to your Slurm submit script.

#SBATCH --gres=gpu:[1-4]    # <-- Choose between 1 and 4 GPU cards to use.

Each node contains four GPU cards. Specify the number of GPU cards per node you wish to use after the gpu: directive.  For example, if your job uses all four GPU cards, simply specify 4:

#SBATCH --gres=gpu:4    # This job will reserve all four GPU cards.

Example Job

The following HPC job will run on the GPU node and simply print information about the available GPU cards:


#SBATCH --job-name="gpu_test"
#SBATCH -n 1
#SBATCH --mail-type="ALL"
#SBATCH -t 1:00

# Here is the magic line to ensure we're running on a node with GPUs
#SBATCH --gres=gpu:1

# If your owner-based partition has access to GPU nodes, you can use that. 
# For general access users, GPU jobs will run only on backfill.
#SBATCH -p backfill

# Not strictly necessary for this example, but most
# folks will want to load the CUDA module for GPU jobs
module load cuda

# Print out GPU information
/usr/bin/nvidia-smi -L

Your job output should look something like this:

GPU 0: GeForce GTX 1080 Ti (UUID: GPU-dc0def06-a6a8-e652-a626-967ca59ea0cd)
GPU 1: GeForce GTX 1080 Ti (UUID: GPU-cdb555e0-dce2-52c6-1029-361375ed79ce)
GPU 2: GeForce GTX 1080 Ti (UUID: GPU-a8606fee-ac2b-4763-fb0d-2f5c9f75244a)
GPU 3: GeForce GTX 1080 Ti (UUID: GPU-433b5155-9a11-055a-5970-770545ae6264)

For more examples and information about GPU programming, refer to our CUDA software documentation.