Jump to: navigation, search

This is a page that describes how make VisIt run in parallel. For information about building parallel on Mac look at Building in parallel on a Mac.

Note that you will almost certainly not be able to use a download from the VisIt website, since your cluster likely has a different version of MPI, networking library, etc. So you will need to be VisIt from scratch in all likelihood.

The game plan is five steps:

  1. Figuring out how to run a parallel program on your cluster
  2. Compiling VisIt to be parallel enabled
  3. Manually making the engine be launched in parallel
  4. Automating the launch of a parallel engine
  5. Confirming that you actually have a parallel engine running


[edit] Figuring out how to run a parallel program on your cluster

This step has nothing to do with VisIt. Most people write a hello world program, like:

#include <iostream>
#include <mpi.h>
int main(int argc, char *argv[])
      MPI_Init(&argc, &argv);
      int rank = -1;
      MPI_Comm_rank(MPI_COMM_WORLD, &rank);
      std::cerr << "My rank is " << rank << std::endl;

Then compile it with something like:

  g++ -I/path/to/mpi/include -L/path/to/mpi/lib -lmpi \
      -o helloworld helloworld.C

And invoke it with something like:

 mpirun -np 4 -o helloworld


 qsub -n 4 myscript

with a script that has an mpirun, ibrun, or something else inside it.

And get output like:

  My rank is 3
  My rank is 1
  My rank is 0
  My rank is 2

Of course, the details of the compile line and invocation command will likely be very different on your machine. Again, the purpose of this section is to ensure that you understand what the compile line is and what the invocation command is.

[edit] Compiling VisIt to run in parallel

To compile VisIt to run in parallel, you need to tell the build system where the include files and MPI libraries are located or at least tell the build system where to find your mpic++ compiler if you have one. This information is typically provided in a "config-site" file. If you have used "build_visit", a config-site was created for you. This file is located in <VisIt>/src/config-site/<hostname>.cmake and it contains the variables needed to tell VisIt's build system where it can find libraries needed to build VisIt.

[edit] Find MPI with mpic++

You can tell VisIt's CMake build system to determine your MPI compilation parameters using the mpic++ compiler, if you wish. This is a fairly automatic way for VisIt to discover how to compile MPI programs on your system but it does not have some of the fine controls that setting the parameters explicitly has.

You'll want to add this to your cmake config-site file:

## (configured w/ mpi compiler wrapper)

[edit] Setting MPI parameters explicitly

In order to provide the build system with information about your MPI, you can set 3 special variables in your <VisIt>/src/config-site/<hostname>.cmake file:

  • Tell the build system where to find the include paths for your MPI header files:
  SET(VISIT_MPI_CXX_FLAGS -I/usr/lib/mpi/include)
  • Tell the build system where to find your MPI libraries:
  SET(VISIT_MPI_LD_FLAGS -L/usr/lib/mpi/mpi_gnu/lib)
  • Or, if your system gets MPI libraries from multiple locations:
  SET(VISIT_MPI_LD_FLAGS "-L/usr/lib/mpi/mpi_gnu/lib -L/some/other/path/mpi")
  • Finally, tell the build system which libraries use:
  • Or if you require multiple libraries:
  SET(VISIT_MPI_LIBS mpi_cxx mpi open-rte open-pal)

After you are done, you need to tell VisIt when it compiles to also build in parallel:


Now you should have both the serial and parallel builds of VisIt.

[edit] OpenMPI

OpenMPI likes to build an MPI library that contains C++ language bindings. This can be a problem if you are using OpenMPI built using one compiler and VisIt built using another compiler because of C++ name mangling mismatches. You can force VisIt's build to only use the C language bindings for OpenMPI (all VisIt needs anyway) by adding the following definitions to your src/config-site/hosts.cmake configuration file.


[edit] Manually making the engine be launched in parallel

If you are running VisIt on your Linux or Mac desktop machine and you used an MPI that provides an mpirun command then you can easily run in parallel by adding -np # to your VisIt command line.

 visit -np 4

If you instead are trying to run on a machine that requires more setup to run a parallel job (such as when you have a batch submission system) then there are two parts to launching the parallel engine:

  1. Figuring out what the right command line is to launch the engine.
  2. Creating a host profile that will create this command line.

[edit] Figuring out the right command

This section is about figuring out the right command line to launch the parallel engine.

Invoke VisIt as:

 visit -cli -norun engine_par

This will launch the Python interpreter and give you a Python prompt:

 Running: cli2.10.0 -norun engine_par
 engine_par viewer engine_par_ser
 Running: viewer2.10.0 -norun engine_par -noint -host -port 5600

You will then want to specify a command to open a compute engine on a batch node. In this example, srun is used to launch a parallel job from within a batch system. The command would look something like this:

 OpenComputeEngine("localhost", ("-l", "srun", "-np", "16"))

VisIt will then print out something like:

 setenv LD_LIBRARY32_PATH /usr/gapps/visit/2.10.0/linux-x86_64/lib
 setenv LD_LIBRARY64_PATH /usr/gapps/visit/2.10.0/linux-x86_64/lib
 setenv LD_LIBRARYN32_PATH /usr/gapps/visit/2.10.0/linux-x86_64/lib
 setenv LD_LIBRARY_PATH /usr/gapps/visit/2.10.0/linux-x86_64/lib/osmesa:/usr/gapps/visit/2.10.0/linux-x86_64/lib:/usr/gapps/visit/bin/../current/linux-x86_64/lib:/usr/gapps/visit/2.10.0/linux-x86_64/lib:/usr/gapps/visit/bin/../current/linux-x86_64/lib
 setenv LIBPATH /usr/gapps/visit/2.10.0/linux-x86_64/lib
 setenv MESA_GLX_FX disable
 setenv TRAP_FPE 
 setenv VISITARCHHOME /usr/gapps/visit/2.10.0/linux-x86_64
 setenv VISITHOME /usr/gapps/visit/2.10.0
 setenv VISITPLUGINDIR /g/g17/brugger/.visit/2.10.0/linux-x86_64/plugins:/usr/gapps/visit/2.10.0/linux-x86_64/plugins
 setenv VISITULTRAHOME /usr/gapps/visit/2.10.0/linux-x86_64/ultrawrapper
 setenv VISIT_MESA_LIB /usr/gapps/visit/2.10.0/linux-x86_64/lib/osmesa/
 srun -n 16 /usr/gapps/visit/2.10.0/linux-x86_64/bin/engine_par -host surface86 -port 5600 -key 54ae602f16c01f2c6062

You would then try to get the engine to connect back to the viewer by:

  • Going to an interactive shell that is running on the system where you will launch the parallel compute engine.
  • Setting up the environment variables as directed
  • Invoking the command it says

At this point the engine should connect back to the viewer and you will get another prompt. You can then create a plot using the sample data to fully test that things are working properly.

 >>> OpenDatabase("/usr/gapps/visit/data/curv2.silo")
 >>> AddPlot("Pseudocolor", "d")
 >>> DrawPlots()
 >>> quit()

[edit] Automating the launch of a parallel engine

Once you have successfully launched a parallel engine (manually), the last step is to set up a host profile so that VisIt can launch the engine without manual intervention.

Host profiles are documented on page 304 of the current users manual (V1.5) on the Host Profiles section of the "Remote Visualization" chapter.

Also, Brock Palen put together this excellent video tutorial on how to set up a host profile on YouTube [1].

[edit] Making sure you actually have a parallel engine

Now that you can start a parallel engine, it is time to make sure that VisIt is really running a parallel job (as opposed to many serial jobs due to an MPI mishap). VisIt has an option to color by the processor ID of which one of VisIt's processors processed the data. So the blocks that MPI task 0 processed will be colored blue, the blocks from MPI task 1 will be cyan, and so on.

Here is a script that sets up this plot:

# you will need to change this command for your machine
OpenComputeEngine("localhost", ("-l", "qsub/mpirun", "-np", "8", "-nn", "1", "-machinefile", "$PBS_NODEFILE", "-mit-loki")) 
# you will need to change the data location for your machine
DefineScalarExpression("procid", "procid(mesh1)")
AddPlot("Pseudocolor", "procid")

This script will produce an image like this:

This image should be produced when you run the script above. The colors may vary depending on how many processors you use. If you don't run with multi_ucd3d.silo, make sure you do use a multi-block file.

[edit] Making your work accessible to all

If you:

  1. Set up host profiles for a parallel machine that you maintain VisIt on
  2. Have other users of this VisIt installation, especially users that connect client/server

Then: You should submit those host profiles back to the VisIt team to be incorporated in the VisIt install routine. Meaning: right now when you run visit-install, it asks which site you want to install for (LLNL, LBNL, ORNL, ANL, etc). So, if you send the VisIt team your host profiles, your site will show up on this list.

Q: Why is this good? A: When users of your supercomputer install the VisIt client, they can choose the configuration for your site, which includes the host profiles. When they fire up VisIt, it will then know about your supercomputer and they can immediate connect to it, without having to manually set up a host profile.

Similarly, you should contribute back the <machine-name>.conf file. That way building VisIt will be "cmake . ; make", meaning you don't have to run build_visit.

Personal tools