EZTrace
 

Supported architectures/features

EZTrace is mainly tested on Linux and Mac OS systems. It has not been tested on other Unix systems, but if you try, please give us some feedback !

EZTrace is shipped with several modules and can analyze programs based on:

EZTrace is also able to collect hardware performance counters if PAPI is installed.

Of course you can mix these libraries and EZTrace supports hybrid programming models ( MPI+PThread or MPI+OpenMP).

The default trace format generated by EZTrace is Paje. However if you enabled the OTF format in GTG, EZTrace can generate OTF files. In both case, you can visualize the output traces with ViTE.





Documentation

Getting Help

 

Installing EZTrace

Requirements

In order to run EZTrace, you need the following software:

For 'old' versions (ie. before v0.6) of EZTrace, you also need these software:

Building EZTrace

First, you need to configure EZTrace by invoking the configure script.

$ ./configure --prefix=<WHERE_YOU_INSTALL_EZTRACE>

You also need a tool for the trace visualization. We encourage you to try ViTE.

&nbp;

Options to configure

You can pass several options to the configure script for specifying where it should find the needed libraries:

Once EZTrace is configured, just type:

$ make
 $ make install


&nbp;

Using EZTrace

Choosing the functions to instrument

First, you should select the functions you want to instrument in your application. For this, you can set the EZTRACE_TRACE environment variable to the list of plugins that should be used:

$ export EZTRACE_TRACE="pthread mpi"

You can get the list of the currently-selected plugins with the eztrace_loaded command:

$ eztrace_loaded
2 pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
4 mpi Module for MPI functions

By default (ie. if you don't set EZTRACE_TRACE) all the plugins are selected.

You can get the list of the eztrace plugins with the eztrace_avail command:

$ eztrace_avail
3 stdio Module for stdio functions (read, write, select, poll, etc.)
2 pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
6 papi Module for PAPI Performance counters
1 omp Module for OpenMP parallel regions
4 mpi Module for MPI functions
5 memory Module for memory functions (malloc, free, etc.)
&nbp;

Creating execution traces

Once the list of plugins is selected, you can run your application with eztrace. For example:

$ eztrace ./my_program my_arg1 my_arg2

This command line executes your program and generates a trace file in the /tmp directory (usually the file is named /tmp/<username>_eztrace_log_rank_#rank).

Using EZTrace with MPI

EZTrace only needs to set a LD_PRELOAD variable for each MPI process. You can specify to mpiexec the value of LD_PRELOAD using the eztrace -e command.

For example, you can run the following command:

$ mpirun -np 16 `eztrace -e my_app`

When your application ends. Each process writes a file named /tmp/<username>_eztrace_log_rank_<rank>.

Changing The output directory

By default, each process saves its trace in the local /tmp directory. You can change this by setting EZTRACE_TRACE_DIR.

Merging execution traces

Once the execution traces are recorded, you can merge and convert them into a file format that can be read by your visualization software:

$ eztrace_convert -o my_paje.trace /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1

This converts the trace files into the Paje format. If gtg is installed with OTF support, you can choose to convert into the OTF file format with the -t OTF option:

$ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1
Filtering events

You can select the plugins to use for the conversion phase by using the EZTRACE_TRACE variable: if your traces contains MPI and OpenMP events, setting EZTRACE_TRACE to mpi and calling eztrace_convert creates an output trace that only contains mpi events.

Computing statistics

Instead of creating a merged trace file, you can tell EZTrace to compute statistics on the recorded traces:


$ eztrace_stats /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1
[...]
PThread:
-------
6 locks acquired

MPI:
---
27 messages sent
MPI_RECV :10 calls
MPI_BARRIER :11 calls
163 events handled

Defining custom plugins

Since EZTrace works with plugins, you can create one and instrument the functions that you want. An example of plugin is available in the example directory.

Once your plugin is created, you should tell EZTrace where to find it. For this, just set the EZTRACE_LIBRARY_PATH to the appropriate director(ies)y:

$ export EZTRACE_LIBRARY_PATH=path_to_plugin1:path_to_plugin2
$ eztrace_avail
2 pthread Module for PThread synchronization functions (mutex, semaphore, spinlock, etc.)
4 mpi Module for MPI functions
99 plugin1 Example module for libplugin1
98 plugin2 Example module for the plugin2 library

Generating custom plugins

You can generate one plugin and instrument the functions you want to. In order to generate your plugin, you need to create a file containing:

Basically, the file should look like that:

BEGIN_MODULE
NAME example_lib
DESC "module for the example library"
ID 99
int example_do_event(int n)
double example_function1(double* array, int array_size)
END_MODULE

Now use eztrace_create_plugin to generate the plugin source code:

$ eztrace_create_plugin example.tpl
New Module
Module name : 'example_lib'
Module description : '"module for the example library"'
Module id : '99'
emulate record_state for 'example_do_event'
Function 'example_do_event' done
emulate record_state for 'example_function1'
Function 'example_function1' done
End of Module example_lib

The source code is generated in the output directory. Just type:

$ make

Now set the EZTRACE_LIBRARY_PATH to the appropriate directory and you are good to go.

You can also specify (in the example.tpl file) the way a fonction is depicted in the output trace. For instance:

int submit_job(int* array, int array_size)
BEGIN
ADD_VAR("job counter", 1)
END

Specifies that when the submit_job function is called, the output trace should increment the "job counter" variable. You can now track the value of a variable!

The test/module_generator directory contains several scripts that demonstrate the various commands available.

Environment variables

Here is a list of the environment variables that you can use for tuning EZTrace.

Known limitations

 

Frequently Asked Questions

Q: When I run my MPI application with EZTrace, all the processes generate /tmp/<username>_eztrace_log_rank_1 files. What's going wrong ?

This happens when EZTrace fails to intercept calls to MPI_Init (or MPI_Init_thread). There may be several reasons for that:

If you still experience problems, please contact EZTrace development team so that we can fix your problem.

Q: What if I don't want to trace my whole application, but only a small part of it ?

Then, you can call eztrace_start() and eztrace_stop() in your code to specify the part to be traced. You will need to #include <eztrace.h> and to link with libeztrace. Then, just run your application as usual (ie. $ ./my_program my_arg1)

Q: I need to trace my program while using gdb, how can I do that ?

Just add the -d option to eztrace to enable gdb:

$ eztrace -d ./my_program my_arg1 my_arg2

Please note that this should be used only when a bug only happens while using EZTrace

Q: I want my trace to be saved in a specific directory, how can I do that ?

By default, EZTrace saves the trace in the /tmp directory. You can change this by setting EZTRACE_TRACE_DIR.

Q: What if I don't care about OpenMP and I only want to see MPI communication ?

You can set EZTRACE_TRACE to the list of "modules" you want to activate. By default, all the available modules are enabled, but you can tell EZTrace to trace only MPI or OpenMP functions:

$ export EZTRACE_TRACE=mpi
$ export EZTRACE_TRACE=omp
$ export EZTRACE_TRACE="omp mpi"

Q: Can EZTrace generate an OTF trace file so that I can visualize it with VampirTrace ?

Yes, since EZTrace relies on GTG for writing output traces, it can generate OTF trace files. When converting the trace with eztrace_convert, just add the -t OTF option:

$ eztrace_convert -t OTF /tmp/<username>_eztrace_log_rank_0 /tmp/<username>_eztrace_log_rank_1