OpenPower Server

CHPC has a machine donated by IBM which runs OpenPower Linux (CentOS 7 LE) and features two 8-core Power8 CPUs. 256 GB RAM and two NVidia K-80 GPUs. It is of the older, "Firestone", platform (IBM Power S822LC), which does not feature direct communication between the GPUs, but, nevertheless, it is a good resource for testing codes on the OpenPower platform, and to run GPU programs to utilize the GPUs.

Access

The OpenPower machine is named p8.chpc.utah.edu, and can be accessed with ssh. Due to the fact that we only have one machine of this kind, we are limiting the user access. Please, contact us in order to be added to the approved user list and specify what you would like to use the OpenPower machine for (e.g. to test the OpenPower system, or to run on the GPUs).

Usage

p8 is running the LE (little endian) variant of the OpenPower Linux, so, binary data files are compatible with x86 Linux. The machine is mounting user home directories and group spaces, so, there is no need to move the user data to the local machine. However, user programs need to be recompiled.

We are running a limited amount of environment modules on p8, which can be listed by running module avail.

Compiling programs

There are several versions of GNU compiler available, we recommend version 5.3.1 which has some OpenPower optimizations built in.  However, this version does not work with CUDA 7.5, for that use GNU 4.9.4. In the CUDA, the nvcc compiler is included which allows to build CUDA programs. There is also PGI compiler, which supports OpenACC for the GPU use. When testing the machine, we have also installed the evaluation copy of the IBM XL compilers; they seem to still work, though we will not be able to obtain newer versions. In the order of preference, we recommend the PGI compilers, followed by the GNU. 

Program building is essentially the same as on the x86-64 systems, using the same Makefiles and most compiler flags. For those who want to learn more about program optimizations for the OpenPower systems, see this good IBM RedBook.

If your program build requires high performance libraries such as BLAS, contact us so we can point you to the right direction or build the latest version (the OpenPower software stack is still under development so it's worth it to use the latest versions of programs).

As for MPI distributions, there are three available. The IBM PE is IBM optimized MPI implementation. While good, its usage is somewhat different than the familiar other distros, plus, we only have a relatively dated evaluation copy which license may or may not still work. We have built MPICH ourselves, and we also include OpenMPI that comes with the PGI compiler distribution. All are loadable by their respective modules. Since we don't have module hierarchy set up much on p8, make sure to only have loaded one compiler and one MPI at the given time.

We also have CUDA 7.5 installed which can be loaded by module load cuda.

Running programs

Running programs is identical to the x86-64 systems. For serial programs, simply type the executable name, for MPI programs, use mpirun from the preferred MPI distribution.

From the applications, we only have built VASP 5.4.1, available via module load pgi mpich vasp.

If you need to build some program, we will most likely work with you to build the program yourself as we have assigned limited resources to p8 support.