Compiling Wombat

How to compile and run Wombat on laptops and large computers, including Cray machines. We also provide examples of how to analyze Wombat data in Julialang.

We are testing Wombat with Cray, Intel and GNU compilers, with performance in that order. We recommend you use the newest version you can get your hands on.

We also encourage you to compile & install libraries yourself with the compiler of your choice, especially if your system does not come with the newest version. 

We are currently running Wombat on a number of Unix and Linux distributions as well as MacOS 10.12.

Requirements

Wombat requires cmake, get it from your Linux distribution. You can download a binary here.

Wombat requires git, get it from your Linux distribution. You can download it here.

Wombat requires an MPI_THREAD_MULTIPLE capable MPI library and zlib. If your system MPI library does not support this feature, ask your admin to get OpenMPI, or recompile it yourself. E.g.

export MYPREFIX=/home/jdonnert/LIB/

export PATH=$MYPREFIX/bin:$PATH
export LD_LIBRARY_PATH=$MYPREFIX/lib:$LD_LIBRARY_PATH

export CC=/opt/intel/bin/icc
export FC=/opt/intel/bin/ifort

wget https://download.open-mpi.org/release/open-mpi/v3.1/openmpi-3.1.1.tar.gz
tar xfvz openmpi-3.1.1.tar.gz
cd openmpi-3.1.1
./configure --prefix=$MYPREFIX
make -j 2
make install

We are testing Wombat with CrayMPI, OpenMPI, MVAPICH, MPICH and IntelMPI, with performance roughly in that order. As the code uses a number of advanced features of the MPI standard, we recommend to use the newest MPI library available to you. Wombat is part of the OpenMPI regression suite, so it should run well with that library. By July 2018 CrayMPI, OpenMPI and MVAPICH can use the THREAD_HOT_MPI features of Wombat.

On Cray systems, use Cray's compiler and MPI library exclusively.

Building Wombat

Wombat's build system uses cmake to (hopefully) simplify the build process. Compile time options are set by commenting in the CMakeLists.txt in the top Wombat directory.

We currently provide three targets:

  • release - with all optimizations build in. Build time can be in the minute range

  • debug - with variable backtrace and no optimization. Use this for development.

  • genparfile - a binary to generate only a valid parameter file and exit. See Running Wombat

Steps

  • Clone the git repo into ./wombat and change into it

   git clone https://user@bitbucket.org/pmendygral/wombat.git
   cd wombat

  • Edit CMakeLists.txt to set the target (debug,release, genparfile) and the compile time options, by editing the comments.

  • If your system uses modules, you might want to load them, e.g.

   module load cmake intel/2016

  • Create a build sub-directory, so cmake's files do not clutter your main directory

   mkdir build

  • Run cmake in the build directory to generate the Makefiles. This step is necessary only once, unless you are altering the CMakeLists.txt file. 

    cd build
    cmake ..

  • Run parallel make. Avoid using all threads (-j without a number) to not overload your login node. 4 threads usually gives a good speedup.

    make -j 4

The executable "wombat" or "gen-parfile" will be built in the ''build'' directory.

To re-build the executable, running ''make -j 4'' is usually sufficient.

Note that for a release build the interprocess optimization at the linking stage (end) takes the most time. We detect the number of threads on the build system and parallelize this step as well as possible.

Remove the build with

    make clean

Remove the cmake build by removing all files in the build directory with

    rm -r build/*

If it doesn't work

If you are on a Cray machine, please contact us on HipChat or via email immediately. It should work flawlessly out of the box.

Similar to the GNU autotools, cmake tries to guess the system configuration. We found that this generally works well, if:

  • cmake is the newest version available, best 3.4 or higher.

  • The environment variables CC and FC are set to your compilers and they are in the $PATH. E.g. for Intel this could be: 

    export CC=icc
    export FC=ifort

  • The MPI wrappers (often: mpicc & mpifort) are in your $PATH variable, i.e these return a path:

    which mpifort
    which mpicc

  • The environment variables for including and linking are pointing to the header and library files, respectively. On standard systems that means at least that 

    export LD_LIBRARY_PATH="/lib:/lib64:/usr/lib:/usr/lib64:/usr/local/lib:/usr/local/lib64"
    export INCLUDE="/include:/usr/include:/usr/local/include"

  • The MPI library was build with the same compiler version

  • The MPI library was build with threading, e.g. MPICH: 

    ./configure --prefix=$MYPREFIX --enable-fc --enable-threads=multiple

  • You can run wombat's make like this, 

    make VERBOSE=1

to see the full compilation invoked by the Makefile. This is often helpful to track problems with linking or flags.

cmake can't find my MPI library

Sometimes cmake has problems finding the correct mpi library using FindMPI. As described here, you have two options: 1. Telling FindMPI explicitly, which mpicc & mpifort to use (option 1). Or setting the library variables directly (option 2). To do this in Wombat, the CMakeLists.txt already contains the necessary statements for both options in the MPI section. Change the paths according to your system configuration and FindMPI should really find your library.