Using R on HPC Clusters

1. Introduction

This page covers the gnarly system-specific overhead required to run R in parallel on shared computing resources such as clusters and supercomputers. It does not cover the conceptual or practical details of specific parallel libraries available to R programmers; this information can be found in my page on Parallel Options for R. Rather, it covers what you need to know about installing libraries and running R on someone else's supercomputer rather than your personal laptop or desktop.

2. Installing Libraries

2.1. Most Libraries

Users are typically not allowed to install R libraries globally on most clusters, but R makes it very easy for users to install libraries in their home directories. To do this, fire up R and when presented with the > prompt, use the install.packages() method to install things:

> install.packages('doSNOW')
Installing package(s) into '/opt/R/local/lib'
(as 'lib' is unspecified)
Warning in install.packages("doSNOW") :
  'lib = "/opt/R/local/lib"' is not writable
Would you like to use a personal library instead?  (y/n)

This error comes up because you can't install libraries system-wide as a non-root user. Say y and accept the default which should be something similar to ~/R/x86_64-unknown-linux-gnu-library/3.0.
Pick a mirror and let her rip. If you want to install multiple packages at once, you can just do something like

> install.packages(c('multicore','doMC'))

Some libraries are more complicated than others, and you may eventually get an error like this:

>  install.packages('lmtest');
Installing package(s) into '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15'
(as 'lib' is unspecified)
trying URL 'http://cran.stat.ucla.edu/src/contrib/lmtest_0.9-30.tar.gz'
Content type 'application/x-tar' length 176106 bytes (171 Kb)
opened URL
==================================================
downloaded 171 Kb
  
* installing *source* package 'lmtest' ...
** package 'lmtest' successfully unpacked and MD5 sums checked
** libs
gfortran   -fpic  -g -O2  -c pan.f -o pan.o
gcc -std=gnu99 -shared -L/usr/local/lib64 -o lmtest.so pan.o -lgfortran -lm -lquadmath
/usr/bin/ld: cannot find -lquadmath
make: *** [lmtest.so] Error 2
ERROR: compilation failed for package 'lmtest'
* removing '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15/lmtest'

The downloaded source packages are in
    '/tmp/RtmpDTJxhk/downloaded_packages'
Warning message:
In install.packages("lmtest") :
  installation of package 'lmtest' had non-zero exit status

The relevant part is the failure to compile the portion of the module written in Fortran due to /usr/bin/ld: cannot find -lquadmath. This missing quadmath library first appeared in GNU GCC 4.6, so if this error comes up, it means your default gcc is not the same one used to compile R. On SDSC's systems, this is remedied by doing module load gnubase.

Alternatively, you might see an error like this:

Error: in routine alloca() there is a
stack overflow: thread 0, max 137437318994KB, used 0KB, request 48B

Again, this is caused by gnu/4.8.2 not being loaded. The following commands will remedy the issue on Trestles:

$ module unload pgi
$ module load gnu/4.8.2 openmpi_ib R

2.2. Rmpi

In order to use distributed memory parallelism (i.e., multi-node jobs) within R, you will need to use Rmpi in some form or another. Under the hood, the ClusterApply and related functions provided in the parallel library use Rmpi as the most efficient way to utilize the high-performance interconnect available on supercomputers.

It's a lot harder to install Rmpi using install.packages() because you have to feed the library installation process some system-specific library locations before it will work correctly. So, exit R and return to the Linux prompt. Download the Rmpi source distribution:

$ wget http://cran.r-project.org/src/contrib/Rmpi_0.6-3.tar.gz

Rmpi 0.6-3 was the most recent version at the time I wrote this, but you can get the most recent version's download URL from the Rmpi page at CRAN. Once you have downloaded the file, make sure you have the proper compiler and MPI library modules loaded (gnu and openmpi_ib on SDSC's systems), then issue the install command with the paths to your MPI library:

$ mkdir -p ~/R/x86_64-unknown-linux-gnu-library/3.0
$ R CMD INSTALL \
    --configure-vars="CPPFLAGS=-I$MPIHOME/include LDFLAGS='-L$MPIHOME/lib'" \
    --configure-args="--with-Rmpi-include=$MPIHOME/include \
                      --with-Rmpi-libpath=$MPIHOME/lib \
                      --with-Rmpi-type=OPENMPI" \
    -l ~/R/x86_64-unknown-linux-gnu-library/3.0 Rmpi_0.6-3.tar.gz

Be sure to double-check the text in green! In particular, your R library directory's version (3.0 in the example above) is the same as the version of R you're using, and be sure to fill in $MPIHOME with the path to your MPI library (which mpicc might give you a hint). If all goes well, you should see a lot of garbage that ends with

gcc -std=gnu99 -shared -L/usr/local/lib64 ...
installing to /home/glock/R/x86_64-unknown-linux-gnu-library/3.0/Rmpi/libs
** R
** demo
** inst
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
--------------------------------------------------------------------------
The OpenFabrics (openib) BTL failed to initialize while trying to
allocate some locked memory.  This typically can indicate that the
memlock limits are set too low.  For most HPC installations, the
memlock limits should be set to "unlimited".  The failure occured
here:
  
  Local host:    trestles-login2.sdsc.edu
  OMPI source:   btl_openib_component.c:1216
  Function:      ompi_free_list_init_ex_new()
  Device:        mlx4_0
  Memlock limit: 65536
  
You may need to consult with your system administrator to get this
problem fixed.  This FAQ entry on the Open MPI web site may also be
helpful:
  
    http://www.open-mpi.org/faq/?category=openfabrics#ib-locked-pages
--------------------------------------------------------------------------
--------------------------------------------------------------------------
WARNING: There was an error initializing an OpenFabrics device.

  Local host:   trestles-login2.sdsc.edu
  Local device: mlx4_0
--------------------------------------------------------------------------
* DONE (Rmpi)

That error about the OpenFabrics device is nothing to worry about; it happens because the test is running on one of the cluster's login nodes (where you are doing all of this) and cannot access the MPI execution environment that real jobs on compute nodes use.

At this point you should have Rmpi installed, and this allows the snow package to use MPI for distributed computing. If you run into an error that looks like this:

...
** preparing package for lazy loading
** help
*** installing help indices
** building package indices
** testing if installed package can be loaded
Error: in routine alloca() there is a
stack overflow: thread 0, max 10228KB, used 0KB, request 48B
ERROR: loading failed
* removing '/home/glock/R/x86_64-unknown-linux-gnu-library/2.15/Rmpi'

You probably forgot to load the prerequisite modules correctly. Purge all of your currently loaded modules, then re-load the ones necessary to build R libraries

$ module purge
$ module load gnu/4.6.1 openmpi R
$ R CMD INSTALL --configure-vars=...

$ mpirun_rsh -np 1 -hostfile $PBS_NODEFILE $(which R) CMD BATCH ./snowapp.R
[gcn-14-17.sdsc.edu:mpispawn_0][readline] Unexpected End-Of-File on file descriptor 5. MPI process died?
[gcn-14-17.sdsc.edu:mpispawn_0][mtpmi_processops] Error while reading PMI socket. MPI process died?
/opt/R/lib64/R/bin/BATCH: line 60: 130758 Segmentation fault      ${R_HOME}/bin/R -f ${in} ${opts} ${R_BATCH_OPTIONS} > ${out} 2>&1
[gcn-14-17.sdsc.edu:mpispawn_0][child_handler] MPI process (rank: 0, pid: 130753) exited with status 139

*** caught segfault ***
address 0x5ddcbc7, cause 'memory not mapped'
  
Traceback:
 1: .Call("mpi_comm_spawn", as.character(slave), as.character(slavearg),     as.integer(nslaves), as.integer(info), as.integer(root),     as.integer(intercomm), as.integer(quiet), PACKAGE = "Rmpi")
 2: mpi.comm.spawn(slave = mpitask, slavearg = args, nslaves = count,     intercomm = intercomm)
 3: snow::makeMPIcluster(spec, ...)
 4: makeCluster(10, type = "MPI")
aborting ...

Error in mpi.universe.size() : 
  This function is not supported under MPI 1.2
Calls: mpi.spawn.Rslaves -> mpi.comm.spawn -> mpi.universe.size
Execution halted

3. Submitting R Jobs to a Cluster

On personal workstations, R is often used by running the R shell in an interactive fashion and either typing in commands or doing something like source('script.R'). Supercomputers generally operate through batch schedulers though, so you will want to get your R script running non-interactively. There are a few ways of doing this:

1. Add #!/usr/bin/env Rscript to the very top of your R script and make it executable (chmod +x script.R), then just run the script as you would a bash script or any program (./script.R)
2. Call Rscript with the script's name as a command line parameter: Rscript script.R. I've seen this break an otherwise working R script though, and I never got to the bottom of it.
3. Call R CMD BATCH script.R

I am going to use #3 in the following examples because it is the most proper way. The sample job submit scripts that follow are for Torque, which is the batch manager we run on SDSC Gordon and Trestles. These scripts can be trivially adapted to SGE/Grid Engine, Slurm, LSF, Load Leveler, or whatever other batch system your system may have.

3.1. Running Serial R Jobs

Running an R script in serial is quite straightforward. On XSEDE's Gordon resource at SDSC, your queue script should look something like this:

#!/bin/sh
#PBS -q normal
#PBS -l nodes=1:ppn=16:native
#PBS -l walltime=0:05:00
  
module load R
  
cd $PBS_O_WORKDIR
  
export OMP_NUM_THREADS=1 
R CMD BATCH test.serial.R

The peculiar bit to note here is our use of export OMP_NUM_THREADS=1. If you don't specify this, R will use as many threads as it can grab if your script uses a library that supports multithreading. This isn't bad per se, but explicitly specifying OMP_NUM_THREADS is good practice--that way you always know exactly how many cores your script will use.

3.2. Running Shared-Memory R Jobs

Running shared-memory parallel R on a single node is also quite simple. Here is a sample queue script that uses all 16 cores on a dedicated (non-shared) node.

#!/bin/sh
#PBS -q normal
#PBS -l nodes=1:ppn=16:native
#PBS -l walltime=0:05:00
  
module load R
  
cd $PBS_O_WORKDIR
  
export OMP_NUM_THREADS=1
R CMD BATCH test.doMC.R

It is actually the same script as the serial version. Bear in mind that the OMP_NUM_THREADS only controls the number of cores used by libraries which support OpenMP. By comparison, the multicore (and parallel) libraries do not use OpenMP; they let you control the number of cores from within R.

3.3. Running Parallel Jobs with snow/doSNOW

Snow (and its derived libraries) does its own process managements, so you must run it as if it were a one-way MPI job. For example,

#!/bin/sh
#PBS -q normal
#PBS -l nodes=2:ppn=16:native
#PBS -l walltime=0:05:00
  
module swap mvapich2_ib openmpi_ib
module load R
  
cd $PBS_O_WORKDIR
  
export OMP_NUM_THREADS=1
mpirun -np 1 -hostfile $PBS_NODEFILE R CMD BATCH test.doSNOW.R

If you forget to request only one core, your job will fail and you will get a lot of horrific errors:

CMA: unable to open RDMA device
CMA: unable to open RDMA device
[[59223,26],16][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device
[[59223,27],23][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device
[[59223,31],5][btl_openib_component.c:1493:init_one_device] error obtaining device context for mlx4_0 errno says No such device

This is because the R script winds up running in duplicate, and each duplicate tries to spawn a full complement of its own MPI ranks. Thus, instead of getting 2×16 ranks, you get (2×16)×(2×16).

4. Trivial sample code

I've created some trivial Hello World samples that can be found on my GitHub account. They illustrate the very minimum needed to use the parallel backends for the foreach package and are a good way to verify that your parallel libraries and R environment are set up correctly. The idea here is that you can replace the hello.world() function with something useful and be off to a good start.

However, these samples do not illustrate how data, libraries, and subfunctions may have to be transferred to other nodes when using MPI. For more details on how to approach those more realistic problems, see the next part in this series: Parallel Options for R