Building statically on Cray

I compiled version 2.6.3 of VisIt on our Cray XE6 with the Cray environment as of July 2013. With server-side components only, and a static flag. Specifically, I used the gnu compiler and the following configure steps. N.B I did not use the --parallel option:

• ./build_visit2_6_3 --console --no-visit --silo --szip --hdf5 --netcdf --h5part --system-qt --no-pyside --system-cmake --vtk --mesa --server-components-only --static
• I edited my host.cmake file to do this:
  • #VISIT_OPTION_DEFAULT(VISIT_C_FLAGS " -m64 -fPIC -fvisibility=hidden" TYPE STRING)
  • #VISIT_OPTION_DEFAULT(VISIT_CXX_FLAGS " -m64 -fPIC -fvisibility=hidden" TYPE STRING)
  • VISIT_OPTION_DEFAULT(VISIT_C_FLAGS "" TYPE STRING)
  • VISIT_OPTION_DEFAULT(VISIT_CXX_FLAGS "" TYPE STRING)
  • VISIT_OPTION_DEFAULT(VISIT_SERVER_COMPONENTS_ONLY ON TYPE BOOL)
  • VISIT_OPTION_DEFAULT(VISIT_PARALLEL ON TYPE BOOL)
  • VISIT_OPTION_DEFAULT(VISIT_PARALLEL_CXXFLAGS "-DPARALLEL -DMPICH_IGNORE_CXX_SEEK -I/opt/cray/mpt/5.6.3/gni/mpich2-gnu/47/include" TYPE STRING)
  • VISIT_OPTION_DEFAULT(VISIT_MPI_LIBS "mpich;pmi;xmlrpc-epi;alps;alpsutil;alpslli" TYPE STRING)
  • VISIT_OPTION_DEFAULT(VISIT_PARALLEL_LINKER_FLAGS "-L/opt/cray/mpt/5.6.3/gni/mpich2-gnu/47/lib -L/opt/cray/pmi/4.0.1-1.0000.9421.73.3.gem/lib64 -L/opt/cray/xe-sysroot/4.0.46.securitypatch.20120828/usr/lib/alps -L/opt/cray/xe-sysroot/4.0.46.securitypatch.20120828/usr/lib64" TYPE STRING)
  • VISIT_OPTION_DEFAULT(QT_QTUITOOLS_INCLUDE_DIR /apps/rosa/qt/4.7.4/gnu_434/include/QtUiTools)
  • VISIT_OPTION_DEFAULT(VISIT_QT_BIN /apps/rosa/qt/4.7.4/gnu_434/bin)

• I untarred the source code file, then
  • cp host.cmake visit2.6.3/src/config-site
  • mkdir visit2.6.3/build
  • cd visit2.6.3/build
  • ccmake -DCMAKE_INSTALL_PREFIX=/apps/rosa/VisIt/2.6-static ../src
  • Configured, generated the makefile (within cmake), make and make install.

• all the magic about the CXXFLAGS and LINKER_FLAGS can be tested with a hello world program and verbose mode, using the cray compiler wrappers. Since we're using straight gcc, I then copied the minimum set of the flags used by the wrapper.

• Given the compiled code above, I can do the two things I needed:
  • Connect to the Cray from a VisIt client running inside our firewall.
  • Run batch-mode stuff using SLURM.

• I modified the definition of class JobSubmitter_sbatch(JobSubmitter) in the internallauncher by adding one line to read

        if self.parallel.nn != None:
            ppn = self.PPN()
            parcmd = parcmd + ["--ntasks-per-node=" + str(ppn)]

• My test SLURM test is the following:

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks=24
#SBATCH --ntasks-per-node=12
#SBATCH --time=00:05:00
#SBATCH --output=/scratch/rosa/jfavre/out.txt
#SBATCH --error=/scratch/rosa/jfavre/err.txt
#SBATCH --account=usup
##SBATCH --reservation=main

/apps/rosa/VisIt/2.6-static/bin/visit -nowin -cli -s /apps/rosa/VisIt/parallel_run.py -np $SLURM_NTASKS -nn $SLURM_NNODES -l aprun

with the test python code here:

import os
basename = os.getenv('SCRATCH')

OpenDatabase("localhost:/apps/rosa/VisIt/2.6-static/data/multi_ucd3d.silo", 0)

DefineScalarExpression("procid", "procid(mesh1)")
AddPlot("Pseudocolor", "procid", 1, 0)
DrawPlots()
# Begin spontaneous state
View3DAtts = View3DAttributes()
View3DAtts.viewNormal = (-0.264402, 0.544304, 0.796131)
View3DAtts.focus = (0, 2.5, 10)
View3DAtts.viewUp = (0.256419, 0.835472, -0.486042)
View3DAtts.viewAngle = 30
View3DAtts.parallelScale = 11.4564
View3DAtts.nearPlane = -22.9129
View3DAtts.farPlane = 22.9129
View3DAtts.imagePan = (0, 0)
View3DAtts.imageZoom = 1
View3DAtts.perspective = 1
View3DAtts.eyeAngle = 2
View3DAtts.centerOfRotationSet = 0
View3DAtts.centerOfRotation = (0, 2.5, 10)
View3DAtts.axis3DScaleFlag = 0
View3DAtts.axis3DScales = (1, 1, 1)
View3DAtts.shear = (0, 0, 1)
SetView3D(View3DAtts)

AnnotationAtts = AnnotationAttributes()
AnnotationAtts.userInfoFlag = 0
SetAnnotationAttributes(AnnotationAtts)
# End spontaneous state

save_win_atts = SaveWindowAttributes()
save_win_atts.fileName = basename + "/multi_ucd3d."
save_win_atts.resConstraint = save_win_atts.NoConstraint
save_win_atts.SetWidth(1024)
save_win_atts.SetHeight(1024)
SetSaveWindowAttributes(save_win_atts)

SaveWindow()