eViz: a map-generating tool

The plotting tool, eviz.py or eViz for short, is a highly configurable CLI-driven tool that can be used to generate a variety of plots used to diagnose Earth system model output. What follows is a description of several use cases that allow users to quickly generate maps using eViz.

Visualize one field from a single netCDF file

Most Earth system model output is in NetCDF format. Therefore, it is important to try to access and visualize such data quickly. Thus, running the following will print the file’s NetCDF metadata to the screen

python eviz.py --file /path/to/model_output.nc

If you know the name of the FIELD that you want to quickly visualize, then run

python eviz.py --file /path/to/model_output.nc --var FIELD

To run eViz for visualizing netCDF model data you need to provide a configuration file written in YAML format. This file, hereafter referred to as the APP file, contains instructions that specify what data file(s) to use, what field(s) to plot and what type of plot(s) we want. This file can be stored anywhere and can be specified as an argument to eViz. In addition, you have to specify the model template (or APP) to be used. For example:

python eviz.py --configfile /path/to/my_config.yaml -s my_model

In eviz, my_model can be one of the following: generic, geos, ccm, cf, wrf ,or lis. These are the ‘supported models’. Supported models are data sources for which predefined configurations have been made available within eViz.

You can also specify a config directory that contains all of the YAML files stored in sub-directories named after the source of the data. For example:

python eviz.py --config /path/to/config -s my_model

will search for my_model.yaml in /path/to/config/my_model.

The –config /path/to/config option is equivalent to setting the environment variable EVIZ_CONFIG_PATH as follows:

export EVIZ_CONFIG_PATH=/path/to/config

In that case you could run eViz as follows:

python eviz.py -s my_model

Note that running eViz without arguments will generate an error:

python eviz.py    # <<< error

Now, let’s describe the contents of the YAML file.

In order to use eViz we need to tell it where to find the input data. The APP file must specify the filename, the fields to plot and the corresponding plot types. Thus, the simplest YAML file must contain the following information:

inputs:
   - name: /path/to/data.nc4
     to_plot:
        SLP: xy  # assume data.nc4 contains a field named SLP

Note: In this guide we will only consider netCDF input data.

The formatting of a YAML file is important or else syntax errors will result in eViz crashing unexpectedly. In particular, correct indentation is very important in a YAML file. A YAML file is basically a text representation of a Python dictionary with “keys” and “values” separated by a colon “:”. For instance, the “inputs” key above has values specified in the next line, in this case a list of elements starting with a “-” that includes the filename (name) and the fields to plot (to_plot). Keep in mind that for a given input file the values to “to_plot” must be specified as defined in the data source metadata, and it is case-sensitive. For example if the metadata describes a field named SLP then you must enter SLP and not slp.

Also, if one specifies only an APP file, a “generic” model is assumed. A “generic” model is a file model abstraction to represent a generic netCDF data source. In the current example, eViz will attempt to generate the simplest possible 2D map. In this simple case a “good looking” plot is not guaranteed. Finally, note that the plot will be displayed in a pop-up window.

Notes

A sample APP YAML file, sample_app.yaml, with all the available options, can be found in the config/ directory.

2. On DISCOVER you can try this example by copying the file /discover/nobackup/projects/jh_tutorials/eviz/config/my_config.yaml to a directory of your choice.

Metadump: creating the configuration files

Creating the APP and SPECS files can be time consuming. To expedite the creation of such files you can run the metadump utility as follows:

python metadump.py /path/to/model_output.nc --specs model_output.yaml --app model_output_specs.yaml

The above command will create configuration files containing information about all the plottable variables as well as the allowable plot types for each variable. Note that most settings will default to “reasonable” values and will probably need to be tweaked if you wish to generate different output than that provided byt the defaults.

Additionally, you can create configuration files to contain details about selected variables. For example:

python metadump.py /path/to/model_output.nc --specs model_output.yaml --app model_output_specs.yaml --vars VAR1 VAR2

Visualize multiple fields field from a single netCDF file

Here we repeat the above example but specifying more fields to plot. With multiple fields, the plots will be shown one at a time, in a window.

It is more convenient to save the image plots to files. We can control our output options in the “outputs” section of the APP file. For example:

inputs:
  - name: /path/to/data/my_data.nc4
     to_plot:
        SLP: xy
        H: yz


outputs:
   print_to_file: yes

You can also specify an output directory as follows:

outputs:
   print_to_file: yes
   output_dir: /some/other/path/output

Note that if output_dir is not specified then the images will be stored in the top-level EViz directory under outputs. You may also specify the output_dir value from the shell by setting the EVIZ_OUTPUT_PATH environment variable. For example:

export EVIZ_OUTPUT_PATH=/some/other/path/output

which has the same effect as the entry in the APP file.

For multidimensional fields the cases above only display the “first available” field slice. So, if the field is 4D, simple lat-lon plots will display the first vertical level and the first-time level. A simple zonal mean plot will display the first-time level. How do you select vertical or time slices? That’s next!

Visualize fields with a SPECS file

There is another YAML file that is used to configure the plotted fields. We call that file the SPECS file. In it, we provide fine-grained specifications for the field to be plotted. For example, if it’s a 4D field, we may want to specify what vertical level to plot as well as what time level - or perhaps we want a time average. The SPECS file is located in the same path as the APP file, and it must have the same basename with “_specs” appended to it. For example the “my_config.yaml” SPECS file must be named “my_config_specs.yaml”.

Note that we still run eviz as before:

python eviz.py --config /path/to/config -s my_model

but now, the SPECS file, if found, will be used and the configurations therein will be applied to the maps specified in the APP file. What does the SPECS file contain? As an example consider the sea-level pressure field. A possible SLP entry in the SPECS file could be

SLP:
    xyplot:
       levels:
          0: []

Here we specify the field name, the type of plot (xy or latlon), the vertical level we wish to plot, and the contour levels we wish to use in [].

Notes

The field name is, again, case-sensitive and must conform with the netCDF field name metadata.
The type of plot is “xyplot”, not “xy”.
For 2D plots the convention is to use “0” to specify that it is a 2D field and thus a single vertical level.
For 3D fields, the vertical level must be one of the levels specified in the netCDF metadata, for example 1000.
The list [] following “0” can be used to specify a list of contour values that we want to display. In our example the contour list is empty, and so eViz will generate levels based on the range values of the SLP data. Using [] can be particularly useful if the range of data values is not known in advance.

A field such as sea-level pressure can be defined using units of Pa. it can be converted to mb units by defining a “unitconversion” field as shown below (also specifying a data contour range):

SLP:
   unitconversion: 0.01
   units: mb
   xyplot:
      levels:
         0: [700, 800, 900, 950, 975, 985, 990, 995, 1000, 1005, 1010]

For more information about the available options look at the sample SPECS YAML file, sample_specs.yaml, available in the config/ directory.

Notes

Pre-defined SPECS files are included in the config directory and are already configure to work with sample data on the DISCOVER system. These can be used with the “supported” data sources that include generic, geos, ccm, cf, lis and wrf. So, for example, we can use the predefined generic model to create plots from various data sources – as long as they are in netCDF format (that’s what generic refers to).

Use your own APP/SPECS

To use your own APP/SPECS you need to change the file paths in the APP file. These paths specify the eViz input data and output paths in your system. For example, in your APP file:

inputs:
   - name         : my_file.nc4
     location     : /discover/nobackup/$USER/data

outputs:
    output_dir: /discover/nobackup/$USER/output

The location entry is optional as you can specify the file location with the name as follows:

inputs:
   - name         : /discover/nobackup/$USER/data/my_file.nc4

Use configurations from a config directory

The configurations described above are stored with the eViz code base, under config. In some cases you may want to maintain files in a separate config directory and use those when running eViz or iViz. In that case you can specify the location of the config files using the EVIZ_CONFIG_PATH environment variable. For example:

export EVIZ_CONFIG_PATH=/home/$USER/projects/config

Also, note that on DISCOVER there are sample config files that you can copy to your work space. For example:

cp -r /discover/nobackup/projects/jh_tutorials/eviz/config $NOBACKUP

Visualize with a predefined generic model

In this case, eViz will use the predefined SPECS file corresponding to the generic APP file. These files can be found in the config/generic directory. The generated plots will be as specified in the generic APP and SPECS files. The sample generic SPECS file provided with eViz contains settings for multiple files including: 3D data (2D+time) from the Climatic Research Unit, 3D data (2D+time) from the CESM model, and 4D data (3D+time) from ERA5-REAN.

For this case we run eViz as follows:

python eviz.py --source generic

or

python eviz.py -s generic

The terminal output will look something like this:

INFO :: config (__init__:56) : Start init
INFO :: config (__init__:58) : model name: generic
INFO :: data_source (__init__:26) : Start init
INFO :: config (__init__:86) : eviz was successfully configured
INFO :: generic (__init__:56) : Start init
INFO :: root (__init__:40) : Start init
INFO :: frame (__init__:38) : Start init
INFO :: generic (plot:66) : Creating GENERIC plots.
INFO :: generic (plot:77) : ta, xyplot, level:100000
INFO :: generic (plot:90) : GENERIC plots are done.
INFO :: generic (plot:92) : Output files are in /Users/ccruz/scratch/eviz/output/generic
Time taken = 0:00:03.950652

This will produce various plots in the path specified in output_dir, including

Lat-lon plots for all of them, xy
Time-series plots for CRU and CESM
Hovmoller plots for CRU and CESM
Same as (2) but with subsets
Zonal mean plots for ERA5-REAN
Zonal mean Hovmoller plots for ERA5-REAN
Polar plot for CRU

Visualize with a predefined GEOS model

In this case we produce plots generated by the GEOS model simulations including the Chemistry Climate Model (CCM), Composition Forecast (CF), simulations and MERRA2.

For this case we run eViz as follows:

python eviz.py -s geos
python eviz.py -s ccm
python eviz.py -s cf

These commands will produce various plots in the path specified in output_dir including:

Various image plots from multiple GEOS files
CCM comparison zonal mean plots with tropopause overlay in one field
CF comparison plots

For details look in the corresponding YAML files in config/geos, config/ccm, and config/cf and feel free to change the settings.

Visualize NU-WRF output

NU-WRF is a modeling framework that incorporates, among others, the WRF and LIS modeling systems. Therefore, NU-WRF simulations produce both WRF and LIS output.

Visualize with a predefined WRF model

WRF configurations can be found in the config/wrf folder. Feel free to modify the filename (and location) entries. To generate the sample eViz plots run:

python eviz.py -s wrf

This will produce simple LatLon and zonal mean plots for some fields specified in the config/wrf.yaml file.

Visualize with a predefined LIS model

LIS configurations can be found in the config/lis folder. To generate the plots run:

python eviz.py -s lis

This will produce simple 2D (LatLon) plots for the fields specified in the config/lis.yaml file.

Using eViz from a Jupyter notebook

Start up Jupyter notebook and open examples/eviz.ccm_example.ipynb. The notebook contains the commands to run both the iViz and eViz tools within the notebook. Feel free to experiment!