From Debian Clusters

This is the second part of a two-page tutorial on Gromacs. The full tutorial includes

• Gromacs: Molecular Dynamics
• Gromacs: Installation

Installing Gromacs

Gromacs should be installed from source, because compilation on the hardware it will be running on is faster than a general compilation (and guaranteed to have all the necessary binaries). Further, while there is a Debian package for gromacs, it isn't always up-to-date.

Supporting Packages

Gromacs relies upon FFTW, the Fastest Fourier Transform in the West, to do optimized Fourier transforms, and should be installed prior to compiling gromacs. To get it, run

apt-get install fftw3 fftw3-dev sfftw-dev sfftw2

fftw3 (although not the dev packages) needs to be installed on all of the worker nodes. You can do this one node at a time, or see the Cluster Time-saving Tricks page to learn some scripting techniques.

If you don't install fftw3 on all of the worker nodes, sooner or later down the line, you'll get error messages like this:

grompp: error while loading shared libraries: libfftw3f.so.3: 
   cannot open shared object file: No such file or directory

Getting and Building Gromacs

The gromacs source code is available online at the Gromacs website. Right click the most recent .tar.gz file and then, from wherever you keep source code, use wget to download it. The compiled source will need to be available to all of the worker nodes, so you may want to keep the source code on an NFS mount as well. (For instance, my NFS mount is on /shared, and I'm keeping worker node source at /shared/usr/local/src.) From the correct directory, the wget command should look similar to this:

wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-3.3.3.tar.gz

Once it has finished downloading, untar the file with

tar xvzf gromacs*.tar.gz

and then cd into the new directory. Gromacs follows the typical source install paradigm, so no surprises there. Just make sure you're compiling it on one of your worker nodes! Run ./configure to see all the available options for compiling. Important ones include

• --prefix=<location> should be used to specify where to install, otherwise files will be placed in /usr/local/gromacs/
• --enable-mpi to enable MPI functionality - assuming you have MPICH installed
• --with-fft=fftw3 to tell it to use the FFTW package just installed

After deciding upon any other options, run ./configure will all of them. For instance, the line I ran was

./configure --prefix=/shared --enable-mpi --with-fft=fftw3

When compiling with MPI support, it should finish with a nice little message like this:

* Seems you are compiling with MPI support. You can install the MPI-
  enabled programs with suffixed names to have both MPI and non-MPI
  versions. This is useful e.g. on supercomputers where you usually
  cannot run MPI-linked programs on the login node.
  Set a suffix with e.g. --program-suffix=_mpi (or _mpi_d for double).
  You only need MPI for mdrun, so if you already have non-MPI stuff
  installed you can issue make mdrun; make install-mdrun.

After this, run make. If it finishes without errors, run make install. At the end of the make install, you should get a message like this:

GROMACS is installed under /shared.
Make sure to update your PATH and MANPATH to find the
programs and unix manual pages, and possibly LD_LIBRARY_PATH
or /etc/ld.so.conf if you are using dynamic libraries.

Please run "make tests" now to verify your installation.

If you want links to the executables in /usr/local/bin,
you can issue "make links" now.

Testing the Gromacs Install: Gromacs Test Files

At this point, the installation files should be copied out to <prefix>/bin and also <prefix>/share. installation verification sounds like a good idea. However, the test source code isn't include as part of the original tar file. Check the gromacs site for the most recent version of the test files, and then (still inside the gromacs source directory), wget them. For instance,

wget http://www.gromacs.org/content/view/169/210/

then untar the file,

tar xvzf gmxtest*

Then go ahead and do make tests.

The first time I ran it, I received these errors:

sh: line 1: 26201 Aborted                 mdrun >mdrun.out 2>&1
FAILED. Check files in rb1
1 out of 16 simple tests FAILED
FAILED. Check files in dec+water
1 out of 14 complex tests FAILED
All 63 kernel tests PASSED
N      Reference   This test
  10    -33.9883    -29.6936
  11    -33.9883    -29.6936
There were 2 differences in final energy with the reference file
All 45 pdb2gmx tests PASSED

This means there were differences in the output between the reference file and the values seen. The differences (in my case), could be viewed in complex/dec+water/checkpot.out, but I didn't find a file to view for rb1. It's up to you how much difference you're willing to accept.

Check the Gromacs Install: Gmxbench Suite

Another great way to test the Gromacs installation is to install the Gromacs benchmarks suite, gmxbench. These are available at the gromacs site. Visit this site and right-click the .tar.gz file to copy the location of the newest set of benchmarks. Then, from where you keep your source code that needs to be accessible by all the worker nodes, run the following (replacing the location if necessary):

wget ftp://ftp.gromacs.org/pub/benchmarks/gmxbench-3.0.tar.gz

Make a new directory called gmxbench (mkdir gmxbench) and copy the tar file into it (mv gmxbench-* gmxbench). Then, cd into the new directory and untar the file:

tar xvzf gmxbench*

Four new directories will be created; each one of these is a sample of Gromacs input to be run as a benchmark. I'm going to run the villin benchmark. This simulates the folding of a 36-amino acid protein that fits on the top of the villin molecule. Cd into d.villin.

Before running gromacs, the Gromacs pre-processor (grompp) is used to prepare the code to run in parallel. (Make sure you either have an mpd ring started, or that you're doing this from within a qsub script if you're using the Torque-enhanced MPICH - see below for a sample qsub script.) From within the d.villin directory, run

grompp -np 4 -shuffle -sort -v

• -np indicates the number of processors. I'm using four because I have four processors per machine.
• -shuffle distributes the loads equally amongst the processors.
• -sort indicates that atoms should be sorted by their coordinates, further helping parallelization.
• -v stands for verbose, meaning to output to the screen as it is running.

If it only finishes with a warning or two, you're good to go. Each time grompp or gromacs runs, it also gives a quote at the bottom of the output. Just FYI, not all of these are appropriate for a classroom setting!

Next, still from within the Villin directory, it's time to run gromacs using mpi. Yours should look along the lines of this:

mpiexec -np 4 /shared/bin/mdrun -v -c output.gro

• -v specifies to be verbose
• -c specifies the output file

It should finish with some nice benchmarking statics about the system it was run on.

Gromacs in Qsub

If you're using the MPICH with Torque functionality, not the one installed by default by the MPICH2 installation, then you won't be able to run mpirun or mpiexec outside of a qsub script. (Unless you kept the old binaries hanging around with a different name, like mpirun_old and mpiexec_old.) Not a problem, since you'll want to test what your users will be seeing and using anyway.

Below is a sample qsub script for testing the villin benchmark. Make sure you change GPATH to the location on your NFS mount where the villin codes are located, and WORKDIR to where you'd like the final output spat out. In the example below, 8 nodes are used; this can be changed as well.

#PBS -N villin
#PBS -l nodes=8
cd $PBS_O_WORKDIR
GPATH=/shared/usr/local/src/gmxbench/d.villin
WORKDIR=~/gromacs
grompp -np 8 -shuffle -sort -f $GPATH/grompp.mdp -p $GPATH/topol.top -c $GPATH/conf.gro
mpiexec /shared/bin/mdrun -v -c output.gro
mv output.gro $WORKDIR
rm output.gro
exit 0