Representing ghost data

WARNING: The content for this page was written several years ago. Although the information on it is believed to be "mostly true", there may be inaccuracies on the page.

Ghost Data

There are two types of ghost data – “ghost zones” and “ghost nodes”. In VisIt, these quantities are stored as data arrays named “avtGhostZones” and “avtGhostNodes”. These quantities are then used to handle situations where data is not used for a variety of reasons.

These tables include all of the possible types for ghost zones and ghost nodes. The types will be discussed in more detail throughout this document.

A ghost zone can belong to multiple ghost zone types. To account for this, ghost data is stored as an unsigned character and each type corresponds to one bit of the character.

Ghost Zone Types

Type Bit
Duplicated zone, internal to problem 0
Enhanced connectivity zone 1
Reduced connectivity zone 2
Refined zone in AMR grid 3
Zone exterior to the entire problem 4
Zone not applicable to problem 5

Ghost Node Types

Type Bit
Duplicated node 0
Node not applicable to problem 1

Using Ghost Data

A zone that has value ‘0’ in its “avtGhostZones” array is not a ghost zone. A zone that has value ‘9’ (= 2^3 + 2^0) is both a Duplicated zone, internal to problem, and a Refined zone in AMR grid.

Using Ghost Data to Handle Domain Decomposition

Types covered:

  1. Duplicated zone, internal to problem (ghost zone)
  2. Duplicated node (ghost node)

A dataset is distributed among many processors by decomposing that dataset into domains. Using this domain decomposition can lead to problems along each domain’s exterior layer of zones that would not occur if the dataset were considered as one large domain.

Consider the following example:

Ghost diagram.jpg

One common operation is to look at the external faces of a dataset. This can be done by finding the external faces of each of the dataset’s domain. But faces that are external to a domain can be internal to the entire dataset. In the example above, the domain’s external faces correspond to the orange, magenta, and bright blue lines. The magenta line is external to each of the domains, but internal to the entire dataset. These extra faces can have multiple negative impacts. One impact is that the number of triangles being drawn can go up by an order of magnitude, precluding the use of a single graphics card and forcing rendering to be done in a scalable, software based manner. When the faces are drawn opaquely, performance is the only impact, as the images resulting from rendering the extra internal faces will be identical to those drawn without the faces. If the faces are drawn transparently, however, then the resulting picture will also be incorrect.

Now consider the case where interpolation is needed to perform a visualization operation. For example, consider the case where a contour is to be calculated on a dataset that has a zone-centered scalar quantity defined on it. Since contouring is done with an algorithm that requires node-centered data, the first step of this process is to interpolate the data to be a node-centered quantity from a zone-centered quantity. Now consider zone “B” from domain 0 (in figure above). It has two nodes incident to the pink line corresponding to the exterior of domain 0. To correctly assign a value to those nodes, it will need to know the values of zones “D”, “E”, and “F” from domain 1. Without those values, the interpolated values for the shared nodes between domain 0 and domain 1 (which are all marked with ‘X’s), will not be the same, leading to a cracked contour surface.

Both of the above problems require ghost data. For the first case, it is sufficient to mark the nodes on the exterior faces of a domain that are interior to the problem as ghost nodes of type Duplicated node. Then faces that are made up entirely of ghost nodes are internal faces and can be discarded. The second case requires a redundant layer of ghost zones around the exterior of each domain of type Duplicated zone, internal to problem. This allows interpolation to be done correctly. After the operation that requires ghost zones is completed, all ghost zones or primitives resulting from ghost zones are discarded.

Enhanced and Reduced Connectivity Zones

Types covered:

  1. Enhanced connectivity zone (ghost zone)
  2. Reduced connectivity zone (ghost zone)

Enhanced and Reduced Connectivity Zones are used for the same purpose as ghost zones of type Duplicated zone, internal to problem. However, due to the unusual formations of domains that some simulations decompose curvilinear meshes into, degenerate cases can occur that require special handling. Consider the following diagram:

A mesh with enhanced and reduced connectivity zones

Consider the zone “A” in the upper right corner of domain 0. That zone should have a “right” neighbor, an “upper” neighbor, and an “upper-right” neighbor. Its upper neighbor is zone “B” and its right neighbor is zone “C”. But the manner in which domains 1 and 2 intersect make the “upper-right” neighbor be one of either zones “B” or “C”. A zone of this type is called a Reduced connectivity zone. VisIt places reasonable values is this zone (typically the value of the “upper” or “right” neighbor), but zones of this type typically still lead to interpolation problems.

Now consider the zone “D” in the lower left corner of domain 5. That zone should have a "left" neighbor, a "lower" neighbor and a “lower-left” neighbor. Its left neighbor is “E” and its lower neighbor is “F”. But the “lower-left” neighbor can be either “G” or “H”. A zone of this type is called an Enhanced connectivity zone.

Using Ghost Data to Remove Unwanted Zones

Types covered:

  1. Zone not applicable to this problem (ghost zone)

Some simulations will create a curvilinear mesh where not all of the zones are used, typically because they value the structured connectivity, but are unable to place some logical zones. These zones are marked as ghost zones of type Zone not applicable to this problem. The values in these zones should be ignored in all cases (when calculating extents, rendering, taking statistical measures, performing interpolations, etc).

Using Ghost Zones to Create Boundary Conditions

Types covered:

  1. Zones exterior to the entire problem (ghost zone)

Some simulations place zones on the exterior layer of their dataset to create a boundary condition. These zones are designated in VisIt as type Zone exterior to the entire problem. Although these zones should not be visualized, they are sometimes needed for interpolation purposes.

Ghost zones of this type are handled much differently than ghost zones of type Duplicated zone, interior to problem, especially when calculating external face lists. Since these zones are exterior to the entire problem, they are incident to external faces of the problem. Hence the zones must be removed before calculating the external face list so that the true external faces can be visualized.

Use of Ghost Zones with Adaptive Mesh Refinement (AMR) Data Sets

Types covered:

  1. Refined zone in AMR grid (ghost zone)

Adaptive Mesh Refinement grids will often contain a zone at one level of refinement that is replaced by some finer resolution zones at another level of refinement. Because the coarser mesh is often rectilinear or curvilinear, it is difficult to discard the refined zone without affecting the structure of the mesh. For zones of this type, the designation Refined zone in AMR grid is used. The zone is still processed, but any contribution it can make to the final picture is removed. In addition, zones of this type still can be used for interpolations.

Removed Nodes

Types covered: Node not applicable to problem (ghost node)

Sometimes nodes in a mesh are not needed, but must be present to preserve structured indexing. These nodes can be marked as type Node not applicable to problem. The impact of this ghost node type in VisIt is different from Duplicated node. A face that is incident to one or more nodes of type Node not applicable to problem should be discarded. When dealing with ghost nodes of type Duplicated node, all nodes incident to a face must be of this type to discard it.

Effect of Ghost Zones

The type of ghost zone affects how that zone is used, as shown in the table below. The properties of interest are:

  1. Are these zones used for interpolation?
  2. Do we have to remove zones of this type to find the true external face list?
  3. Are these zones exactly duplicated elsewhere by a real zone?
  4. Are these zones effectively duplicated elsewhere by some combination of zones?
  5. Are these zones always on the exterior of a domain?
  6. Can the values in these zones be used for statistical purposes?*
Type / Property #1 #2 #3 #4 #5 #6
Duplicated zone, internal to problem Yes No Yes Yes Yes Yes
Enhanced connectivity zone No No Yes Yes Yes Yes
Reduced connectivity zone No No Yes Yes Yes Yes
Refined zone in AMR grid Yes No No Yes No Yes
Zone exterior to the entire problem Yes Yes No No Yes No
Zone not applicable to problem No Yes No No No No

Note that property #6 has some subtleties. Clearly, zones of type “not applicable to problem” should not be used for calculating any statistical information. However, when taking averages, ghost zones of type “Duplicated zone, internal to problem” should also be excluded, or else it will be double counted, skewing the results. Zones of that type can be legitimately used for determining minimums and maximums, however.

Effect of Ghost Node Types

The type of ghost node affects how that node is used, as shown in the table below. The properties of interest are:

  1. If a face contains all nodes of this type, should it be excluded?
  2. If a face contains one node of this type, should it be excluded?
  3. Are these nodes used for interpolation?
Type / Property #1 #2 #3
Duplicated node Yes No Yes
Node not applicable to problem Yes Yes No

Requesting Ghost Data

VisIt decides what sort of ghost data, if any, is required for an operation using the following mechanism. During the “update” phase of a pipeline execution, each filter in the pipeline has an opportunity to modify the “data specification” which is a contract between the pipeline and the database which specifies what data should be read in and properties that data must have. Part of the data specification pertains to ghost data handling. In particular, a filter can specifically request that ghost nodes are turned on or that ghost zones are turned on.

When the database creates the specified data, there are many scenarios to consider. In terms of ghost data creation, there are three options: (1) ghost data was already written out into the database, (2) there is information about how to create ghost data in the database, or (3) no information exists about how to reconstruct ghost data.

In terms of the data specification, the options are that (1) no ghost data was requested, (2) only ghost nodes are requested, or (3) ghost zones were requested. (Note that ghost nodes are not relevant for case #3. Ghost zones provide more information than ghost nodes. For this reason, if a dataset already contains ghost zones, it is sufficient to return that dataset even if just ghost nodes are requested.) To request ghost data, define (or modify) your plot or filter's EnhanceSpecification method by setting the desired ghost type:

avtYourPlot::EnhanceSpecification(avtContract_p spec)
  string ov = atts.GetOpacityVariable();
  // tell pipeline that we need ghost data.

The following table shows what type of ghost data is returned based on what is requested in the data specification (Requested) and what information exists in the database (DB contains):

DB contains / Requested None Ghost Nodes Ghost Zones
No ghost data or info None None None
Info to create ghost data None Ghost Nodes Ghost Zones
Ghost Zones Ghost Zones Ghost Zones Ghost Zones

It is important to note that it is possible for ghost zones to be returned by the database even if they weren’t necessary. It is also possible for no ghost zones to be returned even if they were requested. Finally, VisIt’s material interface reconstruction algorithm depends on ghost zones. If this algorithm is required, it may alter the data specification to request ghost zones.

For some operations, the presence of ghost data is a complication. For example, consider creating a histogram of some scalar quantity. If ghost zones of type Duplicated zone, internal to the dataset are present, those zones will be double-counted, skewing the results. This case can become even more complicated if other operations are involved that require ghost zones. Should ghost zones be created when taking a histogram of a contour surface (which requires ghost zones)? For VisIt, the answer is yes. This is because it is easy to remove the ghost zones immediately before doing the histogram. It is not possible to create the correct contour surface if the ghost zones are not present.

Other complications can arise due to creating ghost zones. Some filters, like the expression that calculates material volumes fractions, request problem-sized objects from the database, such as material information. It then wants to index the material information based on the dataset that it has received as input, which has ghost zones. Concretely, a dataset may have 100 zones. After creating ghost zones, it has 110. So the filter received the dataset with 110 zones as input. But the material object it gets from the database has only 100.

Filters such as the material volume fraction expression can only operate on datasets which have their original connectivity. By creating ghost zones, we would be modifying that connectivity. So if the “maintainOriginalConnectivity” flag of the data specification is set to be true, we get the following behavior:

DB contains / Requested None/Ghost Nodes / Ghost Zones
No ghost data or info None
Info to create ghost data None
Ghost Zones Ghost Zones

Creating Ghost Data

Ghost data may be created by VisIt in three ways: for creating duplicated zones, internal to the dataset, duplicated nodes, or a refined zone in an AMR grid.

If a dataset is decomposed into many domains and there is information about how to create ghost data, then VisIt can create either ghost zones or ghost nodes based on the table provided in the “Requesting Ghost Data” section. The information comes from a derived type of “avtDomainBoundaries” which is a data structure that can be provided by a database plugin or simulation. The database module then uses this information to create the ghost data. Look at Domain Boundaries to see how domain boundary data are constructed.

For AMR grids, ghost zones are created when a coarse zone is replaced by some refined zones at a different level of refinement. The database module again performs this operation, using the “avtDomainNesting” data structure which is provided by the database plugin.