From VisItusers.org

There are various external tools and methods that can be used to measure memory consumption. In some environments though, your only means to measure memory consumption is to query it yourself in the code. BlueGene/P systems fall into this category since your program runs on dedicated compute nodes that only run your program and do not allow other programs to run.

[edit] Code instrumentation

Here are 2 methods of instrumenting your code to determine the amount of memory in use.

[edit] Using GetMemorySize

VisIt provides a function in Utility.h that provides the memory used and the resident set size, which is the portion of a process's memory that is held in RAM.

#include <Utility.h>
#include <DebugStream.h>
 
int size = -1, rss = -1;
GetMemorySize(size, rss);
 
debug5 << "Amount of memory in use: " << size << ", RSS=" << rss << endl;

[edit] Using mallinfo

The mallinfo() function is a GNU function (it may only exist on Linux) that lets you query statistics for memory allocated via malloc(). Don't worry if your program is C++ because all C++ new operators eventually call malloc(). Here is a function that you can include in your code to log memory usage:

#include <string>
#include <malloc.h>
#include <DebugStream.h>
 
void
log_memory_usage(const std::string &label = std::string())
{
    struct mallinfo m = mallinfo();
    debug5 << "mallinfo:" << label << endl;
    debug5 << "  non-mmapped space allocated from system ="<<m.arena<<endl;
    debug5 << "  number of free chunks ="<<m.ordblks<<endl;
    debug5 << "  number of fastbin blocks ="<<m.smblks<<endl;
    debug5 << "  number of mmapped regions ="<<m.hblks<<endl;
    debug5 << "  space in mmapped regions ="<<m.hblkhd<<endl;
    debug5 << "  maximum total allocated space ="<<m.usmblks<<endl;
    debug5 << "  space available in freed fastbin blocks ="<<m.fsmblks<<endl;
    debug5 << "  total allocated space ="<<m.uordblks<<endl;
    debug5 << "  total free space ="<<m.fordblks<<endl;
}

The key numbers to look at are: non-mmapped space allocated from system, which seems to indicate the total amount that malloc() has allocated from the system so far. The next important lines are: total allocated space and total free space. These lines, give the amount of the malloc'd memory that is in use by the program and the amount of that space that is currently free, respectively.

[edit] External sampling

On systems where you can run additional programs to sample a program's memory footprint, you have various options. Note that for parallel programs, you should use the code instrumentation approaches since you will likely want to profile memory consumption for several hundreds or thousands of processors. That is not typically feasible using sampling tools.

[edit] Using top

Most UNIX systems provide a program called top that lists the programs that are using the most CPU cycles. The table that gets printed usually contains metrics of the memory used by a process.

[edit] Using Totalview

Totalview has a memory profiler that can keep track of the allocations for each processor in your parallel job. Note when you set up the process to be debugged in Totalview, there is an "Enable memory checking" check box that you must enable running your program.

(more here later)

In my experience, using Totalview for memory debugging is hard to get working with VisIt in a parallel setting.