Introduction

SuperLU is a linear algebra package designed to solve large, sparse and nonsymmetric systems of linear equations. SuperLU is designed to solve these types of systems directly using LU Factorization with partial pivoting. The program can run in parallel either using shared-memory (OpenMP) or distributed memory (MPI) and can also run on GPU systems that support CUDA as well.

Using SuperLU on RCC Resources

The distributed version of SuperLU is installed on the listed RCC resources and the library itself does not require the use of a module. Since this library uses Intel Open MPI or Intel MVAPICH2, those modules are required. The Open MPI version of the library is located in the /opt/hpc/intel/openmpi/lib64/ directory and requires the Open MPI module intel-openmpi. The MVAPICH2 version is located in the /opt/hpc/intel/mvapich2/lib64/ directory and requires the MVAPICH2 module intel-mvapich2.

The header files for OpenMPI are located in the /opt/hpc/intel/openmpi/include directory, and the header files for MVAPICH2 are located in the /opt/hpc/intel/mvapich2/include directory.

Here we compile and run an example described in the documentation pdf. Further described on page 59, we compile the PDDRIVE program that solves a linear system. First we create a new directory for SuperLU, then copy the relevant files needed into that folder.

mkdir ~/SuperLU
cp /gpfs/research/software/userfiles/SuperLU_examples/pddrive.c ~/SuperLU/
cp /gpfs/research/software/userfiles/SuperLU_examples/dcreate_matrix.c ~/SuperLU/
cd ~/SuperLU

While in the SuperLU directory, compile the three source files, taking care to include either the Open MPI or the MVAPICH2 directory. In this example, we use the Open MPI version. Make sure that you load intel-openmpi or intel-mvapich2 module first.

mpicc -std=c99 -O3 -Wall -w2 -mkl -openmp -DDEBUGlevel=0 -DPRNTlevel=1 -DPROFlevel=0 -DNoChange -DUSE_VENDOR_BLAS -I../SRC -c pddrive.c dcreate_matrix.c

Now link the library files to create the pddrive executable.

mpicc -openmp  pddrive.o dcreate_matrix.o  /opt/hpc/intel/mvapich2/lib64/libsuperlu_dist_4.0.a \
-L/opt/hpc/intel/mvapich2/lib64 -lparmetis -L/opt/hpc/intel/lib64 -lmetis -L/gpfs/research/software/intel/mkl/lib/intel64 \
-lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread -lpthread  -lm -o pddrive

Successful execution of this program requires the use of some command-line arguments, described in the comments in the pddrive.c source file.

Additional source files for examples are available in the /gpfs/research/software/userfiles/SuperLU_examples/ directory, and would be compiled in a similar manner. Any additional files would need to be downloaded from the SuperLU home page. Download the the 4.0 version (this is the distributed version) using the command

wget http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_4.0.tar.gz

After extracting the files, some potentially necessary/helpful source files will be located in the SRC directory. The library files will not be available, however. The library files would need to be compiled, but we can instead use the library files already compiled and available on the HPC, using a method similar to that used in the linking stage described above.