Sources

VisIt can read from a great many file formats. However, there are other sources of data that VisIt might use. For example, VisIt could use procedurally generated geometry such as spheres or boxes as inputs to plots and operators. In addition, it would be useful to cache the output of plots or operators and package that as a source that could be used in subsequent plots. These and other source types will be covered on this page.

Sources

A source provides data that VisIt can plot. A source has a name and provides metadata indicating the meshes and variables that are available for plotting.

VisIt currently provides database sources, which are sources where the data comes from files via VisIt's database plugin infrastructure. A database source is identified by the name of its database and given that database name, VisIt can return avtDatabaseMetaData for the source's plottable contents by querying the metadata server.

There are other types of sources that VisIt could provide:

  • Geometry sources
    • Sphere : Unstructured mesh where the user provides origin, radius, and discretization parameters.
    • Box: Rectlinear mesh where the user provides NX, NY, NZ and box extents
    • Cone mesh: Unstructured mesh where the user provides cone origin, normal, angle, height (along normal), and discretization
    • y=f(x): Curve produced by evaluating user-provided f(x) in range [x0,xn] with specific sampling.
  • Pipeline sources
    • These sources take as input another source as a starting point and then apply various VisIt operators to transform the data. The output of the pipeline is the data produced by the source, which is then fed into other operations. These sources can be built in a database-forward manner in that no plot need be specified to begin the pipeline.
    • (Jeremy notes the it might be useful to leave unspecified exact variables. For example, NewSource = slice(OldSource) could be specified, allowing you to plot a slice of any of OldSource's variables simply by plotting those varaibles from NewSource.)
  • CMFE sources
    • CMFE is a mechanism that lets VisIt map the fields from one mesh onto another mesh. This is currently done using expressions and the user must create a CMFE expression for each variable that will be plotted.
    • Time derivative: The source would provide the same metadata as its input sources but its variables would be evaluated so that they are the time derivative of the desired fields
    • Difference: difference 2 sources, producing variables sourceA - sourceB for the intersection of the variables in the 2 sources' metadata.
    • Mapping: Map sourceA onto sourceB's mesh for the intersection of the variables in the 2 sources' metadata.
    • The CMFE sources since the viewer would be aware of them could actually use database correlations, as they probably should
  • Union sources
    • Produce a new source that includes all of the variables from sourceA,sourceB,...,sourceN. This type of source would take care of managing several open files and it could be used to make it look like data from many sources came from a single source, which would simplify operations such as writing CMFE expressions.

Since the output of a source is always a dataset of some sort that will be cached in the engine, probably in NetworkManager, I am convinced they should be handled in much the same way at all levels of VisIt. That's why I'm "unifying" what might seem like different types of sources.

Properties of sources

A source has:

  • name
  • metadata as an output
  • a source type (db, geometry, pipeline, CMFE, union)
  • some sources take other sources as inputs.
  • filters in a source can publish variables into the source's metadata
  • As with database sources, a source gets its own time slider in the viewer. This time slider is assembled from the time values

In a way, sources as I am thinking about them are much like the Sources, Filters, Sinks in AVT. Some sources don't use other sources to produce their metadata (database sources, geometry sources). Some sources take in other sources and transform them, producing new metadata from the input sources' metadata (pipeline sources, CMFE sources, union sources). The metadata from source inputs flows through the source and may be modified or restricted by AVT filters that make up the source. The output metadata is then the metadata that is associated with the source's name and is used to populate the plot menus when the source is activated.

Since sources may be built from the database forward, it means that the entire complement of input metadata can be seen as flowing through the source instead of a single variable, as with plots. This means that the sources are free to be more macro-like than our current plot-based pipelines.

Designing sources (Option #1)

(Brad)

I envision a separate Sources window that lets users manage and design sources. We could make it a wizard or similar type window that leads the user through setting up or editing a source. I am open to how this window looks but it probably is somewhat similar to the Annotation window's Object's tab, which has widgets for managing a list of items and an area that controls specific types of objects. Geometric sources would have pages for their various sphere, box options. Pipeline sources would provide an area that looks like the plot list when it is expanded: database->op1->op2->... CMFE sources would provide a page that lets you pick the type of CMFE you want and then pick the sources that are used in the CMFE.

With the ability to plug sources into other sources, this kind of gives us a crude network manager. We're able to box up pipelines into a source and give it a name. We could certainly allow for nested sources.

Once a source has been created, its name appears in the window's Sources list and a new item appears in the Main window's File panel where it can be opened or activated like any currently open any other database source.

It should also be possible to highlight a plot in the Plot list and promote it to a pipeline source. This would enable you to set up the visualization for 1 variable, promote to source, and then visualize any of the source's other variables in the same way -- the same way you'd do if you changed variables.

Implementation issues

  • The gui currently passes the source name to the mdserver to get metadata for the source. With more generic sources, we should instead query the viewer since it will be aware of the source definitions and their metadata. This could enable us to cut the gui->mdserver link that we've had for a while. The CLI and Java clients already are able to obtain metadata from the viewer. Of course, we could just change the gui to get metadata from the viewer in the case of non-database sources, if we wanted to.
  • The viewer will need to maintain the source definitions so those definitions, which in some cases are much like a ViewerPlot, can be saved to sessions for future use. These definitions will also be useful for sending the source definitions down to the engine in case it has restarted.
  • The viewer will need to send down pipeline descriptions for the sources as it needs to define them. Then, the engine will need to evaluate the source pipeline and cache it in the NetworkManager with the given source name. The network manager already caches databases based on their names; this is really the same thing but we're caching an avtDataObject (result from evalulating the source) instead of an avtDatabase.

Designing sources (Option #2)

(Jeremy)

The main significant difference between this option and Option #1 is that in this instance, the creation of new pipeline sources requires no new window -- at least not to create the pipelines. Instead, one uses the existing plot list, except that the plot list is no longer constrained to have each entry (pipeline) end in a plot.

Advantages (relative to option #1):

  • First, creating a plot-less pipeline in the "plot list" itself adds new capability to the plot list.
    • Take the Poincare example: one could start with MySource, add the Streamline operator of "vel", and add the Poincare operator.
      • At this point, in this Option (#2), one could then add a Pseudocolor plot of "WindingNumber" (a variable created by the poincare operator).
      • In Option #1, this plotless pipeline creation is done in a new window and one must now create a new source from to create any plot using the created "WindingNumber" variable. (Of course, in this Option #2, one can create a new source instead of adding the plot, and then create the plot from this new source at a later time.)
      • So Option #2 appears at first glance to be a superset of Option #1's capabilities.
    • As another example, one could create a plot-less pipeline, then export it. Exporting should ignore the plot anyway, so adding it seems not just superfluous, but potentially confusing.
    • And as another example, one could run a query on this pipeline instead of drawing a plot. Why must one create some plot before querying the number of zones in some database? (Or in some mesh, more specifically?)
  • Second, one need not create a new window with functionality essentially duplicating the existing plot list.

Downsides:

  • The new capabilities mentioned above require some thought and possibly nontrivial work. For example, how does one create a query without creating and drawing a plot? If there's not a plot at the end, can we execute the pipeline? What does the Draw button mean -- should it be renamed? Is "plot list" the right word?

Up/Downsides:

  • In the Poincare example given in the advantages above, the first action after starting with MySource was to add a Streamline operator with a variable. In this case, it made sense for the Streamline operator itself to have a choice of vector variables (similar to a vector plot). In other cases, e.g. the Slice operator, it takes no variable. And in third cases, like the Scatter plot, one needs at a minimum TWO variables to create a plot; it's silly to have one of these designated "primary".

Creation of CMFE or Geometry based sources would still probably best be served with a new window or two. There's little similarity between the CMFE source creation, the Geometry source creation, and the Pipeline source creation, so there doesn't seem to be a big advantage to having them all be within one window. (Not that there's no advantage, I'm merely suggesting the disadvantages may outweigh the advantages.)

Designing sources (Option #3)

(Jeremy)

This option is the same as Option #2, except that the existing plot list *is* still constrained to end in a plot. In that case, one simply takes the current (e.g. 1.12) behavior and adds a new e.g. menu option to take that pipeline (up to the plot) and create a new source from that. The advantage relative to Option #2 is that it's a very small change from current behavior -- certainly for the GUI, but even internally, we would not need to add the ability to create these pipelines from database-forward. Of course, we lose the advantages gained in Option#2 due to this ability.

(And again, geometry and CMFE sources are unsolved in this option and still probably need one or two new windows.)

Designing sources (Option #4)

(Jeremy)

Everything's a source.

Add an operator? New source. Export? Export a source. Query? Query a source. Plot? Plot a source.

This is totally un-fleshed out, and it's a huge change. But I'm being idealistic here.

Other misc notes about creating new sources

  • Simplify Export?
    • If we can create new sources from a pipeline, then we could get all the existing Export functionality while changing it to use a Source as input instead of a Plot. Whether this is a simplification or improvement at all may be debatable, though not having to create a plot to export a database certainly makes conversion easier.