Build it and you will learn: Building an RPi2 cluster

Open MPI

I had hoped to build and install Open MPI [9] as I did MPICH; however, I had problems building it on the RPi2. You can read the Open MPI mailing list [10] about the issues. To have a second MPI library, I decided to install the default Open MPI libraries that come with Raspbian.

The four Open MPI packages (version 1.4.5-1 dating back to February 2012) that came with Raspbian were installed using apt-get:

  • openmpi-bin
  • openmpi-dev
  • openmpi-doc
  • openmpi-common

By default, Open MPI tools are installed in the standard $PATH (e.g., /usr/bin, /usr/lib/, /usr/include). When environment modules are created, you need to take care with these paths. I talk more about this in the next section.

Lmod

Another indispensable tool for HPC systems is environment modules. Environment modules allow you, the user, to specify which compilers you want, which libraries you want (perhaps built with the same compilers), or anything else you can do to manipulate the user environment.

Environment modules are important because you could have one application built with a specific compiler and another application built with a different compiler. Without environment modules, you would have to reconfigure your user environment before running either application.

There are two environment modules you can use. The one used in this project is called Lmod [11]. I've recently written an article about the latest version [12]. Lmod is easy to build (in the /work/pi/src/lmod-<version> directory):

$ ./configure --prefix=/work
$ make
$ sudo make install

Because all of the nodes need Lmod all the time, it is installed in /work rather than a private directory.

Before using Lmod, you have a little post-configuration work. Most importantly, the Lmod init scripts need to be executable by users. Symlinks from the init scripts are created (as root),

$ sudo /bin/bash
# ln -s /work/lmod/lmod/init/profile \
        /etc/profile.d/modules.sh
# ln -s /work/lmod/lmod/init/cshrc \
        /etc/profile.d/modules.csh

from the Lmod directory to the /etc/profile.d directory. A line also needs to be added to the global bashrc file (/etc/bash.bashrc) so that it is always initialized when a shell is started). Add the following lines to the end of this file:

# Lmod
. /etc/profile.d/modules.sh

Be sure you either log out of your RPi2 or start another shell for these changes to take place. Once you have done this, you need to create your module hierarchy.

Figure 3 shows the directory layout. Anything marked with f indicates a file. The Lmod modules are in a shared NFS directory, /work/modulefiles, so that all nodes have access to them.

Figure 3: Module hierarchy.

By default, Lmod starts by looking in the Core directory, where the compilers and any applications that are not dependent on a compiler or an MPI library reside. After a compiler or application module is loaded, the compiler branch of the tree becomes active. Modules in this branch are dependent on the compiler. The mpi branch contains applications or tools that are dependent on the compiler/MPI combination.

In the HPC world, parallel applications are commonly based on MPI libraries, so the module files for these are in the compiler branch. For this project, I'm using two: one each for MPICH and Open MPI. The first module file is for version 4.6.3 of the GNU compiler. Recall that compilers are installed in the standard $PATH; therefore, you have to be careful about adding anything to $PATH so that if the module is unloaded, the standard $PATH won't be compromised. Listing 6 is the compiler module file I used for this project.

Listing 6

Compiler Module File

01 -- -*- lua -*-
02 ------------------------------------------------------------------------
03 -- GNU 4.6.3 compilers - gcc, g++, and gfortran. (Version 4.6.3)
04 ------------------------------------------------------------------------
05
06 help(
07 [[
08 This module loads the gcc-4.6 compilers (4.6.3). The
09 following additional environment variables are defined:
10
11 CC   (path to gcc compiler wrapper      )
12 CXX  (path to g++ compiler wrapper      )
13 F77  (path to gfortran compiler wrapper )
14 F90  (path to gfortran compiler wrapper )
15
16 See the man pages for gcc, g++, gfortran (f77, f90). For
17 more detailed information on available compiler options and
18 command-line syntax.
19 ]])
20
21 -- Local variables
22 local version = "4.6"
23 local base = "/usr/bin/"
24
25 -- Whatis description
26 whatis("Description: GNU 4.6 compilers (4.6.3)")
27 whatis("URL: www.gnu.org")
28
29 -- Normally - Take care of $PATH, $LD_LIBRARY_PATH, $MANPATH
30 -- Don't add "/usr/bin" to path since it will get removed if module
31 -- is removed
32
33 -- Environment Variables
34 pushenv("CC", pathJoin(base,"gcc"))
35 pushenv("FC", pathJoin(base,"gfortran"))
36 pushenv("CXX", pathJoin(base,"g++"))
37 pushenv("F90", pathJoin(base,"gfortran"))
38 pushenv("cc", pathJoin(base,"gcc"))
39 pushenv("fc", pathJoin(base,"gfortran"))
40 pushenv("cxx", pathJoin(base,"g++"))
41 pushenv("f90", pathJoin(base,"gfortran"))
42
43 -- Setup Modulepath for packages built by this compiler
44 local mroot = os.getenv("MODULEPATH_ROOT")
45 local mdir = pathJoin(mroot,"compiler/gnu", version)
46 prepend_path("MODULEPATH", mdir)
47
48 -- Set family for this module (mpi)
49 family("compiler")

MPI implementations are based on specific compiler versions. The first MPI module file listing is for MPICH version 3.1.4. The other module is for Open MPI version 1.4.5, which comes standard with Raspbian, so you have to be very careful about manipulating the standard environment variables. The code I used for the mpich/3.1 and the openmpi/1.4 modules is available at the Raspberry Pi Geek website [13].

Once the modules are in the /work/modulesfiles directory, you can test Lmod with the command module avail (short for "module available") (Listing 7). You can now "load" the various modules. To begin, load the compiler (Listing 8). Notice that Lmod only presents the modules that match the compiler that is loaded. In this case, that is mpich-3.1 and openmpi-1.4. When the mpich-3.1 module is loaded (Listing 9), notice that the mpicc, mpif77, mpif90, and mpirun point to the correct binaries.

Listing 7

Testing Lmod

pi@raspberrypi ~ $ module avail
------------------- /work/modulefiles/Core -------------------
   gnu/4.6    lmod/6.0.1 (D)    settarg/6.0.1 (D)
-------------- /work/lmod/lmod/modulefiles/Core --------------
   lmod/6.0.1    settarg/6.0.1
  Where:
   (D):  Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".

Listing 8

Loading the Compiler

pi@raspberrypi ~ $ module load gnu/4.6
pi@raspberrypi ~ $ module list
Currently Loaded Modules:
  1) gnu/4.6
pi@raspberrypi ~ $ module avail
------------- /work/modulefiles/compiler/gnu/4.6 -------------
   mpich/3.1    openmpi/1.4
------------------- /work/modulefiles/Core -------------------
   gnu/4.6    lmod/6.0.1 (D)    settarg/6.0.1 (D)
-------------- /work/lmod/lmod/modulefiles/Core --------------
   lmod/6.0.1    settarg/6.0.1
  Where:
   (D):  Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".

Listing 9

Load the mpich-3.1 Module

pi@raspberrypi ~ $ module load mpich/3.1
pi@raspberrypi ~ $ module list
Currently Loaded Modules:
  1) gnu/4.6   2) mpich/3.1
pi@raspberrypi ~ $ module avail
------------ /work/modulefiles/compiler/gnu/4.6 ------------
   mpich/3.1    openmpi/1.4
------------------ /work/modulefiles/Core ------------------
   gnu/4.6    lmod/6.0.1 (D)    settarg/6.0.1 (D)
------------- /work/lmod/lmod/modulefiles/Core -------------
   lmod/6.0.1    settarg/6.0.1
  Where:
   (D):  Default Module
Use "module spider" to find all possible modules.
Use "module keyword key1 key2 ..." to search for all possible
modules matching any of the "keys".
pi@raspberrypi ~ $ which mpicc
/work/apps/gnu_4.6/mpich/3.1.4/bin/mpicc
pi@raspberrypi ~ $ which mpif77
/work/apps/gnu_4.6/mpich/3.1.4/bin/mpif77
pi@raspberrypi ~ $ which mpif90
/work/apps/gnu_4.6/mpich/3.1.4/bin/mpif90
pi@raspberrypi ~ $ which mpirun
/work/apps/gnu_4.6/mpich/3.1.4/bin/mpirun

Buy this article as PDF

Express-Checkout as PDF

Pages: 2

Price $2.95
(incl. VAT)

Buy Raspberry Pi Geek

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content