Libsim Batch

Libsim was originally developed to facilitate interactive connections from VisIt to running simulations so the simulation could be used as something of a live database that could be explored interactively. This permitted debugging and some simulation steering. As in situ has become more important as a means of avoiding I/O bottlenecks on large clusters, there has been an increasing need for batch-style in situ processing from Libsim. This capability was added in 2014. Batch simulations need to start up Libsim, specify some plots and continue to save images or export data. The integration with the simulation can often be simpler than interactive-style Libsim connections because the additions to the simulation main loop can be simpler by comparison. The data access callback functions that comprise the Libsim adaptor are the same whether interactive or batch integration is done.

Useful Links:


Building/Linking

Libsim is part of VisIt, which is typically built with the build_visit script. To build VisIt, get the build_visit script for the version of VisIt that you want to use and then invoke it to build and install VisIt. Note that --xdb flag tell the build to include FieldView XDB support.
Example:

sudo mkdir /usr/local/visit
./build_visit --prefix /usr/local/visit --xdb --server-components-only

When VisIt has been installed to /usr/local/visit, you can now begin to link your simulation with Libsim. Libsim has a front-end library and a runtime library. The front-end library is called libsimV2 and it dynamically loads the Libsim runtime library when in situ functions are needed at runtime. You'll later want to call VisItSetDirectory() to tell Libsim where VisIt is installed so the runtime loading of the library can be completed.

  • Include files are located in /usr/local/visit/current/linux-x86_64/libsim/V2/include
  • Library files are located in /usr/local/visit/current/linux-x86_64/libsim/V2/lib
    • You may link C/C++ simulations with -lsimV2
    • Fortran simulations link with -lsimV2 -lsimV2f
    • libsimV2dyn.so is a shared library version of libsimV2.a in case a shared library version is required by the host application

Initialization

The normal set of low-level Libsim functions need to be called whether the simulation is going to be interactive or batch. This means, the functions that do the following:

  • Tell Libsim to open a trace file
  • Tell Libsim where to locate VisIt
  • Tell Libsim options to use for VisIt
  • Set up the MPI communicator
  • Tell Libsim whether it is being run in parallel
  • Get the environment

Trace File

Running with trace file support is completely optional but it can be very helpful during instrumentation since all Libsim functions are journaled to the trace file. Any errors in locating runtime libraries, etc will be indicated in the trace file. Note that for MPI programs, each MPI rank should call VisItOpenTraceFile() with a different filename.

C/C++

int rank;
char filename[100];
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
sprintf(filename, "trace.%d.txt", rank);
VisItOpenTraceFile(filename);

Fortran

character (len=80) s
integer err, slen, par_rank
call MPI_COMM_RANK(MPI_COMM_WORLD, par_rank, err)
write(s,'(''trace.'',i1,''.txt'')') par_rank
slen = len(trim(str))
err = visitopentracefile(s, slen)

Locating VisIt

Libsim needs to know the path where VisIt was installed so the Libsim runtime library can be loaded. This applies when VisIt has been built using dynamic libraries.

C/C++

// "/path/to/visitdir" would be "/usr/local/visit" if that is where VisIt was installed.
VisItSetDirectory("/path/to/visitdir");

Fortran

integer err
character (len=50) dir
data dir/'/path/to/visitdir'/
err = visitsetdirectory(dir, len(trim(dir)))

Note that Libsim will attempt to dynamically load the Libsim runtime library and various VisIt and VTK dependency libraries. On some Linux systems, the dlopen() library call to load the Libsim runtime is not permitted to use the environment that Libsim itself sets up, causing a runtime failure to load the Libsim runtime library. If this happens to you, be sure to set the LD_LIBRARY_PATH environment variable before running your simulation.

export LD_LIBRARY_PATH=/usr/local/visit/current/linux-x86_64/lib:$LD_LIBRARY_PATH

Options

The VisItSetOptions() function enables the simulation to pass command line options to VisIt once its components are loaded via the Libsim runtime library. Calling VisItSetOptions() is optional but it can be useful.

If you want to make VisIt create debugging logs, do this:

VisItSetOptions("-debug 5 -clobber_vlogs");

If you want to limit the plugins that Libsim will use in order to speed up library loading:

VisItSetOptions("-plotplugins Contour,Mesh,Pseudocolor -operatorplugins Slice,Isosurface,Threshold -noconfig");

MPI communicator

By default, the Libsim runtime library will create a duplicate of the MPI_COMM_WORLD communicator unless you specify an alternate communicator. You might want to specify an alternate communicator for various reasons. For instance, perhaps only a subset of the MPI ranks in the simulation will have a role during in situ processing. Calling the VisItSetMPICommunicator() function is optional.

MPI_Comm par_comm;
MPI_Comm_dup(MPI_COMM_WORLD, &par_comm);
VisItSetMPICommunicator((void *)&par_comm);

Libsim parallel

Libsim can be run in serial or parallel. The front end Libsim library (simV2) actually does not contain any MPI and relies on 2 broadcast callback functions and some flags that indicate whether the library is being used from a parallel application. By not including MPI in the Libsim front end library, it can be used in both serial and parallel applications. When Libsim is used from a parallel application and programming in C/C++, the following functions need to be called:
C/C++

int SimBroadcastInt(int *value, int sender, void *cbdata)
{
    MPI_Comm *comm = (MPI_Comm *)cbdata;
    return MPI_Bcast(value, 1, MPI_INT, sender, *comm);
}

int SimBroadcastString(char *str, int len, int sender, void *cbdata)
{
    MPI_Comm *comm = (MPI_Comm *)cbdata;
    return MPI_Bcast(str, len, MPI_CHAR, sender, comm);
}

int main(int argc, char *argv[])
{
    /*...*/

    /* Install broadcast callback functions for Libsim. */
    VisItSetBroadcastIntFunction2(SimBroadcastInt, (void*)&par_comm);
    VisItSetBroadcastStringFunction2(SimBroadcastString, (void*)&par_comm);

    /* par_size and par_rank are integers that indicate the size and rank from MPI. */
    VisItSetParallel(par_size > 1);
    VisItSetParallelRank(par_rank);

    /*...*/
}

When programming in Fortran, the broadcast functions are still expected to be provided by the application but they must have specific names.
Fortran

c---------------------------------------------------------------------------
c visitbroadcastintfunction
c---------------------------------------------------------------------------
      integer function visitbroadcastintfunction(value, sender)
      implicit none
      include "mpif.h"
      integer value, sender, IERR
      call MPI_BCAST(value,1,MPI_INTEGER,sender,MPI_COMM_WORLD,ierr)
      visitbroadcastintfunction = 0
      end

c---------------------------------------------------------------------------
c visitbroadcaststringfunction
c---------------------------------------------------------------------------
      integer function visitbroadcaststringfunction(str, lstr, sender)
      implicit none
      include "mpif.h"
      character*8 str
      integer     lstr, sender, IERR
      call MPI_BCAST(str,lstr,MPI_CHARACTER,sender,MPI_COMM_WORLD,ierr)
      visitbroadcaststringfunction = 0
      end

      program main
c ...
      include "visitfortransimV2interface.inc"
      integer err, par_size, par_rank
      call MPI_COMM_RANK(MPI_COMM_WORLD, par_rank, err)
      call MPI_COMM_SIZE(MPI_COMM_WORLD, par_size, err)
      if(par_size.gt.1) then
          err = visitsetparallel(1)
      endif
      err = visitsetparallelrank(par_rank)
c ...
      stop
      main

Environment

Since VisIt and Libsim are usually built using dynamic shared libraries, Libsim most often consists of a front end library and a runtime library. By linking just the front end library, interactive applications benefit from having Libsim not use resources until VisIt connects to the simulation. For batch simulations, we know we will load the library explicitly. To successfully load the Libsim runtime library, it is often necessary to query the VisIt installation for the environment that will be needed. This can be expensive at scale so it is best to query the environment on one MPI rank and then broadcast the results to the other MPI ranks. If you want to eliminate the few seconds of initialization, you can instead make your own calls to putenv() to set up the VisIt environment based on the output from visit -env -engine.

C/C++

char *env = NULL;
if(par_rank == 0)
    env = VisItGetEnvironment();

/* Pass the environment to all other processors collectively. */
VisItSetupEnvironment2(env);
if(env != NULL)
    free(env);

Fortran

integer err
err = visitsetupenv()

Load the runtime library

This is where interactive and batch instrumentation methods begin to differ. In an interactive simulation, we'd start injecting calls to VisItDetectInput() into the main event loop. Then there would be code to try and complete a connection from the VisIt viewer, which would load the runtime library. In a batch simulation, we need only load the Libsim runtime library and register the adaptor callback functions. To see how to write adaptor callback functions, see the Getting Data Into VisIt manual.

C/C++

/* Explicitly load Libsim runtime library */
VisItInitializeRuntime();

/* Install data adaptor callback functions. The adaptor functions are the 
 * same whether the simulation is batch or interactive and they expose 
 * the simulation data to Libsim.
 */
VisItSetSlaveProcessCallback2(SimSlaveProcessCallback, (void*)sim);
VisItSetGetMetaData(SimGetMetaData, (void*)sim);
VisItSetGetMesh(SimGetMesh, (void*)sim);
VisItSetGetVariable(SimGetVariable, (void*)sim);
VisItSetGetDomainList(SimGetDomainList, (void*)sim);

When writing Fortran code, the data adaptor callback functions are given specific names in the libsimV2f.a library so you must provide certain function names in Fortran (e.g. visitgetmetadata, visitgetmesh, visitgetvariable,...).

Fortran

integer err
err = visitinitializeruntime()

Batch operations

The operations that the simulation does now that the runtime library is loaded depend on the desired data products.

Saving XDB files

Saving XDB files can be relatively straightforward using the extract helper functions contained in extract.c and extract.h. The provided extract functions enable slice planes and isosurfaces to be generated, though additional functions could be created to export any type of geometry that VisIt can produce. The data extraction is based on VisIt's notion of plots and operators, which are used to define an analysis pipeline, which is then called using export to save that extracted geometry (plus additional variables) to XDB. Other formats can be saved as well.

Extract helper functions:

#ifndef EXTRACT_H
#define EXTRACT_H

/*
 * Functions for programmatically extracting data from VisIt/Libsim.
 */

int extract_set_options(const char *format, int writeUsingGroups, int groupSize);

const char *extract_err(int err);

int extract_slice_3v(const char *filebase, 
                     const double *v0, const double *v1, const double *v2,
                     const char **extractvars);

int extract_slice_origin_normal(const char *filebase,
                                const double *origin,
                                const double *normal,
                                const char **extractvars);

int extract_slice(const char *filebase, int axis, double intercept,
                  const char **extractvars);

int extract_iso(const char *filebase, const char *isovar,
                const double *isovalues, int nisovalues,
                const char **extractvars);

#endif

Code to export extracts must be preceeded with a call to VisItTimeStepChanged() which will call the simulation adaptor's metadata callback function to ensure that the Libsim runtime library has an up to date inventory of the simulation data.

Extract a slice plane and some isosurfaces:

int sim_cycle, par_rank;

int err;
char filebase[100];
const char *extractvars[] = {"q", "xc", "radius", "dom", NULL};
double origin[] = {5., 5., 5.}, normal[] = {0., 0.707, 0.707};
double isos[] = {5., 11., 18.};

/* Tell VisIt that some metadata changed.*/
VisItTimeStepChanged();

/* Save an XDB of a slice plane. */
sprintf(filebase, "slice_%04d", sim_cycle);
err = extract_slice_origin_normal(filebase, origin, normal, extractvars);
if(par_rank == 0)
{
    printf("slice export returned %s\n", extract_err(err));
}

/* Save an XDB of isosurfaces of "radius". */
sprintf(filebase, "iso_%04d", sim_cycle);
err = extract_iso(filebase, "radius", isos, 3, extractvars);
if(par_rank == 0)
{
    printf("iso export returned %s\n", extract_err(err));
}

More on exporting to XDB format and using the extract functions can be located in the batch example simulation.


Performance

Performance over 10,000 cores is best achieved through the use of "write groups". Write groups will be available in VisIt 2.10 and they permit the set of MPI ranks to be divided into smaller groups that each aggregate and write their own partial data file. This results in multiple files per extract but this is needed to maintain performance at large core counts. Write groups have been tested with AVF-LESLIE up to 62,208 cores on NERSC's Edison computer.

/* Make the export routines save to XDB format using write groups of size 96.
 * MPI ranks that have extract data will be grouped into groups of 96 ranks so
 * each group of 96 will write one XDB file.
 */
extract_set_options("FieldViewXDB_1.0", 1, 96);

AVF-LESLIE-writegroups.png

  • The above graph shows the performance of writing a single XDB file using the original algorithm vs improved code that writes multiple XDB files using write groups.

Saving Images

Batch in situ lets you save out images of the computed data in addition to producing extracts. There are numerous Libsim functions for adding plots and operators and setting their attributes. Those functions can be used to build up visualizations.

C/C++

VisItAddPlot("Mesh", "mesh2d");
VisItAddPlot("Contour", "zonal");
VisItAddPlot("Pseudocolor", "zonal");
VisItDrawPlots();

Fortran

integer err
err = visitaddplot("Mesh", 4, "mesh2d", 6)
err = visitaddplot("Contour", 6, "zonal", 5)
err = visitaddplot("Pseudocolor", 11, "zonal", 5)
err = visitdrawplots()

The most comprehensive way to set up a visualization is to use session files. A session file is an XML description of the visualization that contains all of the settings that were in use when VisIt saved the session file, enabling the session file to return the user to where he left off previously. Sessions can be used by Libsim to set up the desired visualizations. Libsim can update the plots as the simulation advances and save images.

C/C++

VisItRestoreSession("visit.session");

Fortran

integer err
err = visitrestoresession("visit.session", 13)

Once plots have been set up manually or using a session file, saving images can occur.

C/C++

int saveCounter = 0;
char filename[100];

/* Tell Libsim that the simulation changed data. Tell it to update
 * the plots that have been set up.
 */
VisItTimeStepChanged();
VisItUpdatePlots();

/* Save an 800x800 pixel JPEG image. */
sprintf(filename, "output%04d.jpg", saveCounter);
if(VisItSaveWindow(filename, 800, 800, VISIT_IMAGEFORMAT_JPEG) == VISIT_OKAY)
{
    if(par_rank == 0)
        printf("Saved %s\n", filename);
    saveCounter++;
}

Fortran

integer err, flen, saveCounter
character (len=80) filename
saveCounter = 0

err = visittimestepchanged()
err = visitupdateplots()

write(filename,'(''output.'',i1,''.jpg'')') saveCounter
flen = len(trim(filename))
err = visitsavewindow(filename, flen, 800, 800, VISIT_IMAGEFORMAT_JPEG)
if(err.eq.VISIT_OKAY) then
    saveCounter = saveCounter + 1
endif

Finalization

To shut down the Libsim library, the simulation code can call VisItDisconnect().

C/C++

VisItDisconnect();

Fortran

integer err
err = visitdisconnect();