On our cluster we have some nodes with GPUs. The HPC Status page will indicate what nodes are GPU capable.
Normally to run a job on the cluster you login to the login node and run
qsub my_job_script.sh which contains PBS commands specifying the CPU, memory and time resources that you need. PBS then allocates your job the most appropriate node for your requirements. You can't login to nodes other than the login node. But in the case of GPUs to even compile your CUDA job you need to be able to login to a GPU node. You do this by submitting a interactive PBS job on the login node that logs you into a GPU node for a specified period of time. From there you can compile and test. You can also submit a PBS job of your progam.
Create a script called
gpu_access.sh. See the Example Interactive PBS Script at the end of this page. In this script set a walltime sufficient for you to do your task, compiling, launching a qsub job or whatever. For instance if you set walltime=01:00:00 you will have 1 hour. You can launch your job on either the gpu node or the login node that you first logged into.
Now submit this script as a PBS interactive job (note the -I to invoke interactive mode).
login node $ qsub -I gpu_access.sh <-- starting on login node qsub: waiting for job 109267.hpcnode1 to start qsub: job 109267.hpcnode1 ready GPU node $ <--- Note node has changed.
You will now have a shell on a GPU node. (Note: the prompt won't actually show "GPU node". It might show "ermdc21" or "ermdc22", whatever is the node that you asked for an interactive session on.)
Now you can submit your real job just as you would have done on a the login node. Note that in your real job submission script you need to specify that it needs to run on a GPU node by including a line like
#PBS -l host=ermdc21
$ qsub fft_job.sh <-- Example, submitting an FFT job. 109275.hpcnode1 <-- It will show your job normally. $
You can exit the GPU node. Just type exit. You will drop back to the login node. Your job will be running on the node with the GPU.
Example Interactive PBS Script
Here is an example gpu_access.sh script.
#!/bin/bash # This is a simple PBS script that uses the "interactive" mode of PBS. # See "Interactive-batch Jobs" in the "PBS User Guide". # # For interactive use you must submit this job with -I # qsub -I this_script.sh #PBS -l ncpus=1 #PBS -l mem=5GB #PBS -q smallq # Set your interactive wall time limit, hh:mm:ss #PBS -l walltime=01:00:00 # Set the appropriate node to start an interactive session on. #PBS -l host=ermdc22 # Note: don't have any other commands below here!
Note: instead of writing a specific script to submit interactively to PBS you can also just start an interactive session like this:
$ qsub -I -l select=1:ncpus=1:mem=5gb:host=ermdc21 -l walltime=01:00:00