Proposed Database Plugin Changes

We all have some ideas about what we'd like to see changed in the database plugins. We can use this page to write down what those changes are and who might do them.

Fewer dependencies for mdserver

The mdserver has historically relied on using only a subset of the VTK libraries in order to decrease its load time. Unfortunately, throughout VisIt's development numerous interdependencies have arisen over the years. These dependencies must be removed instead of working around them using "stub" libraries and the like. Platforms such as Windows require shared library symbols to be resolved at link time so it is not possible to use a stub library and have a viable library; 2 versions of the library would need to be created.

Take avtGenericDatabase for example. The avtGenericDatabase class is instantiated in each of the plugin's Info classes with a file format interface (FFI). Over time, the avtGenericDatabase class has grown to include things like MIR and a transform manager, which provide good functionality but introduce other dependencies. The transform manager transforms various data types into data that VisIt can handle natively such as vtkCSGMesh into vtkUnstructuredGrid. The vtkCSGMesh case is an example of an unwanted dependency in the mdserver case. Although fine for the engine, the presence of the transform manager in the avtGenericDatabase requires vtkCSGMesh in the mdserver, which in turn requires the rest of libvisit_vtk. There are other examples like this where additional libraries must be added into the libdatabase link to work on Windows and on Mac.

I propose that we:

  1. Move the transform manager into the engine/main directory
  2. Replace avtGenericDatabase's usage of the transform manager with calls to virtual functions that do nothing
  3. Remove transform manager from avtGenericDatabase
  4. Create a subclass of avtGenericDatabase that contains a transform manager and implements methods from #2.
  5. Change all plugin info classes to return the FFI instead of an instance of avtGenericDatabase so we
  6. can choose based on whether we're in the engine or mdserver to instantiate avtGenericDatabaseWithExtras or avtGenericDatabase, respectively.
  7. This also means that the database factory would need to be changed also. I'd probably add a factory method that returns an avtGenericDatabase given an FFI. I'd probably also let the user install a callback that would be used preferentially so I could write a function in the engine that returns avtGenericDatabaseWithExtras
  8. Do the same thing for MIR as we do for the transform manager.

I think that if we follow this procedure, we will be able to create a lightweight version of libdatabase that does not have additional dependencies that lengthen load time.

Smaller libM

Maybe something can be done with database plugins to reduce the size of libM plugins. I often notice that most plugins don't have mdserver-specific compilation in them, although they should. This means that libM plugins contain code for GetMesh, GetVar even though they will NEVER be called. I propose we do something like this:

class FooFileFormatM : public avtSTSDFileFormat
   // methods to get state space (cycles, times)

   // method to get the metadata
   virtual void PopulateDatabaseMetaData(md);
   // any data members we need to understand the file format

class FooFileFormat : public FooFileFormatM
    // Hey, this one has GetMesh, GetVar, and any other real or exotic metadata.
    virtual vtkDataSet *GetMesh();
    virtual vtkDataArray *GetVar();

The mdserver libM plugin would include the M version of the file format sources and the engine libE plugin would include both. I also like this division because I think it helps delineate what should and should not be done in the mdserver.

State space mesh

Return a multidimensional table of state space that may include cycles, times, dates, whatever user labels we want to use. We'd have to develop some sort of cursor class that could iterate over it. This stuff would replace the GetCycle,GetTime methods.

fill in more...

Automatic file format detection


Ditch File Format Interface

I hate the file format interface constructs we use sometimes. For simple file formats it works well but there are a number of formats such as PDB, NETCDF, Silo where it is wholly inadequate. The problem with those file formats is that you don't know before opening the file which file format interface you should be creating. For example, NETCDF is sometimes "ST" (single time step) and it is often "MT" (multiple times step) and you can't know which before you read the file. What this means is you have to create a very strange reader that checks the file and creates the appriate FFI depending on what it finds in the file. I typically create a "reader" class that can read the file format and then I create 2 file formats (one ST and one MT) that both use the "reader" class. I would seem that it would be better to remove the FFI layer and provide an interface that always lets a plugin provide multiple time and multiple domain data and then handle it appropriately at a higher level.

Make MT file grouping work

File grouping doesn't work for most MT file formats. This is a flaw in the database factory.