Parameter Files

Wombat has a build target to compile an executable that outputs a valid parameter file for the chosen compile options. To build it, set the target to "GenParfile" in "CMakeLists.txt" and set the compile time options as you intend to run. Then build as usual. This will generate an executable gen-parfile. Produce a valid parameter file "myrun.par" in the parent directory with standard values like this:

 .../build$ ./gen-parfile ../myrun.par

Edit the parameter file to your needs. Then clean the build directory: 

 .../build/$ rm -r *

Then build the Debug or Release target as described here

Parameter Values

It is your job to figure out the details of the run: required memory, rank decomposition, patch size, number of threads, box size. 

Usually the lower bound of the number of ranks is set by the available memory. Estimate what your run will require from 

 # of zones * # of state variables * 8 bytes  +  # of particles * # of fields (7) * 4 bytes

Wombat scales very well, leave some headroom for memory imbalance + 1 GB per rank for thread local scratch variables. Loss of efficiency is moderate if you increase the number of MPI ranks. Plans for a more sophisticated memory management system are on its way. You will get estimate in the beginning of the run. Currently, only WENO returns an estimate of its thread private buffers.

We recommend :

  • A patch size of ~32^3 zones per patch (TVD, WENO 18^3) (Mendygral et al. 2017)

  • 2 MPI ranks per NUMA domain. 

  • One OpenMP thread per compute unit (physical core) of your system. Hyperthreads usually slow the code down. 

Note that:

  • The net resolution of the simulation along one direction is (wombat.f90) n_xpatches * patch_nx * patch_size(1) .

  • Boxsizes in y- and z-direction are set relative to the x-direction from the y and z number of zones.

Units

All values in the parameter file are in code units : length_cgs, mass_cgs, velocity_cgs. The standard values are cosmological units, kpc, 10^10 Msol, km/s. time_cgs is then roughly 1Gyr. These units have proven useful in the galaxy formation context. For test problems, we often set all units = 1.

Internally the unit values are used to convert the hard-coded physical constants in CGSM (i.e. G=6.6e-8) into code units (G=47003). CGS units should not appear anywhere in the code or in the snapshots. See simunits.f90.

For comoving/cosmological simulations the conversion from code to physical units changes with time/redshift, because the expansion of the Universe is scaled out of the equations.

Running Wombat

To run Wombat:

mpirun -np [# ranks]  ./wombat [parameter file]

Advanced Parameter Interface

If you know what you are doing, you might want to enable the ADVANCED_PARAMETER_INTERFACE in CMakeLists.txt. It exposes more parameters in the file. For most runs, the standard values will do and you can leave the advanced parameter interface disabled.