Python Module Support Revamp

Overview

Goals

• Better support for hybrid py + compiled ext modules (Including Cmake scheme for compiling the exts)
• Support disutils setup of pure python modules
• More pythonic handling of module imports

Targeted Modules

• visit (frontend)
• visitmodule (main visit module)
• pyside_pysideviewer
• pyside_visithoook
• pyqt_viewer
• mpicom
• pyavt

Many of these these packages will be consolidated as sub modules in the main "visit" module:

visit/
  visitmodule
  pysideviewer
  pysidehook
  pyqtviewer

The new module includes new frontend magic (frontend.py) to allow the base module to dynamically load the proper visitmodule.

Potential Gotchas

• Static builds
• make install/package

Status

Item	Status: Linux	Status: OSX	Status: Windows
PYTHON_ADD_DISTUTILS_SETUP	Ok	Ok
module creation
visit & visitmodule	Ok	Ok
pyavt	Ok	Ok
mpicom	Ok	Ok
pyside support	Ok	Ok
pyqt support
build w/ shared libs
out of source	Ok	Ok
make install	Ok	Ok
make package	Ok	Ok
make bundle	N/A	Ok	N/A
build w/ static libs
out of source	Ok	Ok
make install	Ok	Ok
make package	Ok	Ok
make bundle	N/A	Ok	N/A
import tests
import visit (from build)	Ok	Ok
import visit (from install)	Ok	Ok - Requires setting DYLD_LIBRARY_PATH
from visit import * (from build)	Ok	Ok
from visit import * (from install)	Ok	Ok - Requires setting DYLD_LIBRARY_PATH
import visit (shared - import 2.4 module from 2.5 frontend)	Ok	Ok - Requires setting DYLD_LIBRARY_PATH
import visit (shared - import 2.5 module from 2.4 frontend)

CMake Support for Hybrid Py + Complied Ext Modules

Currently we use (a slightly modified version of) CMake's PYTHON_ADD_MODULE function.

This is works well for creating pure extension modules, for example:

PYTHON_ADD_MODULE(mymodule srcs...)

creates a python module named 'mymodule' (mymodule.so or mymodule.pyd)

The goal here is to support a more flexible combination of python source files and compiled python C libraries.

So instead of simply creating:

mymodule.so

We can easily support:

mymodule/
  __init__.py
  mypython_1.py
  mypython_2.py
  mymodule.so

We rely on CMake to build the extension modules b/c it encapsulates all of our build logic (compiler flags, etc) and existing targets for linking.

However copying *.py files and creating the proper module directory structure is a bit tedious in CMake's scripting language.

So instead of using CMake for this, the pure python portions of our modules will use a standard python distutils setup.py file.

To accomplish this we added a new CMake helper function PYTHON_ADD_HYBRID_MODULE:

PYTHON_ADD_HYBRID_MODULE(module_name setup_file py_src_list c_srcs...)

Example Usage (src/visitpy/mpicom/CMakeLists.txt):

  # identify our python sources
  SET(MPICOM_PY_SOURCES py_src/__init__.py
                        py_src/mpistub.py)
  # identify our c sources
  SET(MPICOM_C_SOURCES  src/mpicom.C
                        src/Buffer.C)
  # Create the mpicom
  PYTHON_ADD_HYBRID_MODULE(mpicom
                           site-packages/
                           setup.py
                           "${MPICOM_PY_SOURCES}"
                           ${MPICOM_C_SOURCES})

We will also support pure python modules, via a new new CMake helper function PYTHON_ADD_DISTUTILS_SETUP

PYTHON_ADD_DISTUTILS_SETUP(module_name dest_dir setup_file)

Example Usage (src/visitpy/pyavt/CMakeLists.txt):

# deps for pyavt
SET(pyavt_sources py_src/__init__.py
                  py_src/filters.py
                  py_src/templates/__init__.py
                  py_src/templates/advanced_expression.py
                  py_src/templates/advanced_query.py
                  py_src/templates/simple_expression.py
                  py_src/templates/simple_query.py)

PYTHON_ADD_DISTUTILS_SETUP(pyavt_py_setup
                           site-packages/
                           setup.py
                           ${pyavt_sources})

The new visit python module layout

visit/
  __init__.py
  frontend.py
  visitmodule.{so,pyd}  # not built for a static build
  pysideviewer.{so,pyd} # optional
  pyqtviewer.{so,pyd}   # optional
  pysidehook.{so,pyd}   # optional

Using the visit module from a standalone python

Importing from an installed version of VisIt

If VisIt is in your path:

import sys
sys.path.insert(0,"path/to/visit/<ver>/<arch>/lib/site-packages/")
import visit
visit.Launch()

If VisIt is is not in your path:

import sys
sys.path.insert(0,"path/to/visit/<ver>/<arch>/lib/site-packages/")
import visit
visit.Launch(vdir="path/to/visit/bin")

Note about use on OSX: visit.Launch() handles the DYLD_LIBRARY_PATH setup for the other VisIt components, however it is not possible to change the DYLD_LIBRARY_PATH used for the dynamic load of the visitmodule by the frontend. So on OSX you need to set DYLD_LIBRARY_PATH before you launch python

>export DYLD_LIBRARY_PATH=path/to/visit/<ver>/<arch>/lib/
>python

Importing from a development version of VisIt

If 'build/bin' is in your path:

import sys
sys.path.insert(0,"path/to/visit/lib/site-packages/")
import visit
visit.Launch()

If 'build/bin' is not in your path:

import sys
sys.path.insert(0,"path/to/visit/build/lib/site-packages/")
import visit
visit.Launch(vdir="path/to/visit/build/bin")

Using the VisIt PySideViewer from a standalone python

todo ...

virtualenv

To add better support for package management, we are considering using a virtualenv as the foundation for VisIt's python libs.