iViz: the interactive tool
The interactive tool, iViz, can be launched from the top level directory using a CLI or directly in a Jupyter notebook.
This section covers the following:
How to run iViz
iViz can be run from the command line or from a Jupyter notebook. The inputs required to run the tool are the same, regardless of the system or manner of use.
All arguments are optional but either an input, a source, or both, must be provided
Flag |
Name |
Description |
|---|---|---|
-i |
–input |
A path to a file or directory |
-s |
–source |
Source of data input (model or observation) with accompanying config files |
-d |
–dtype |
If iViz is unable to determine data type, enter data type of input (netcdf, hdf, tabular) |
-se |
–sesssion |
Restore a previous visualization session with a yaml file saved in iViz |
-p |
–plot_options |
Provide plot options to appl to plots with a yaml file saved in iViz |
Use iViz on Discover
Log into Discover JupyterHub system and start a session at:
https://jh-discover.nccs.nasa.gov/
Open a notebook and select the Python viz environment. iViz and eViz code is installed on Discover. In a notebook cell, run the following command in order to import and instantiate iViz.
from eviz.lib.iviz_base import Iviz
iviz = Iviz()
Once iViz is imported, start the tool with the make_notebook function. Provide input to the tool using one or more of the arguments listed in How to run iViz . All of the commands in the cell below are correct and runnable.
iviz.make_notebook(s='ccm')
iviz.make_notebook(s='wrf')
iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/lis/LIS_HIST_201507121200.d01.nc', s='lis')
iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/cf')
After running one of the commands above in a cell, the visualization dashboard will be output in the cell below. A session file from a previous iViz session, or a plot options file saved in iViz, can also be input to the make_notebook function to restore a session in iViz or apply a set of plot options to plots.
iviz.make_notebook(s='ccm', p='my_plot_options.yaml')
iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/wrf/',
s='wrf', p='my_plot_options.yaml')
iviz.make_notebook(s='cf', se='/discover/nobackup/vvalenti/22.01.2023_18:39:20.iViz_session.yaml')
Data sources and types currently supported
Inputs currently supported:
A supported model or observation data source with configuration files (-s –source)
Path to a directory with netcdf files in it (-i –-input)
Path to a file that is either netcdf, hdf, or tabular data type (-i –-input)
Currently supported configurable data sources:
-s flag keywords |
Source (model or observations) |
Data type |
|---|---|---|
ccm |
GEOS-CCM (Chemistry Climate Model) |
netcdf |
cf |
GEOS-CF (Composite Forecast) |
netcdf |
wrf |
WRF (Weather Research Forecast) |
netcdf |
lis |
LIS (Land Information System) |
netcdf |
airnow |
EPA Airnow |
tabular |
Configurable data sources contain additional information regarding model or observations specific fields available, like unit conversions, and ability to add information like experiment name, and where to save output plots. If you would like to add a data source, either a model or observations, please contact the developers.
Additional data sources supported without configuration:
Source (model or observations) |
Data product |
Data type |
Where to access or download |
|---|---|---|---|
Landsat |
Level 3 global gridded |
hdf |
|
Ozone Monitoring Instrument (OMI) |
Level 3 global gridded |
hdf |
Examples for each of these data sources and types below
Visualize configuration supported data sources
Configuration files are stored in the ‘config’ directory of the codebase. iViz and eViz automatically use read-only config files stored on Discover. To edit these files for personal use, refer to Customize configuration files on Discover Run any of the commands below
Jupter notebook (Discover) |
CLI |
|---|---|
|
|
|
|
|
|
|
|
|
|
The source argument can be used with or without the input argument. If a file or directory path is provided as input without a source, iViz will use the coordinates of the data file to try to determine the source. If iViz is unable to determine the source, an -s input is required.
Jupter notebook (Discover) |
CLI |
|---|---|
|
|
|
|
|
|
|
|
|
|
Visualize satellite data (beta)
Jupter notebook (Discover) |
CLI |
|---|---|
|
|
|
|
Visualize tabular data formats (beta)
Tabular data is data that is stored in rows and columns. This includes most CSV file formats. Tabular data is opened with pandas into a DataFrame.
Jupter notebook (Discover) |
CLI |
|---|---|
|
|
Customize configuration files on Discover
The config directory containing data source specific configuration files is read-only on Discover. To edit and customize configuration files, copy the config directory to your personal $NOBACKUP space. For example:
cp -r /discover/nobackup/projects/jh_tutorials/eviz/config /discover/nobackup/$USER/eviz
Then, set the environment variable ` EVIZ_CONFIG_PATH ` to the location of this config directory.
To do this within the Jupyter notebook, use the os module to set the environment variable. For example:
import os
os.environ['EVIZ_CONFIG_PATH'] = '/discover/nobackup/$USER/eviz/config/'
In a bash shell, set the environment variable with the following command:
export EVIZ_CONFIG_PATH=/discover/nobackup/$USER/eviz/config
iViz layout and controls
Header
Sidebar
Plotting area
Bokeh based plotting
Plotting in iViz is bokeh based. All plots come with the Bokeh interactive toolbar.
From left to right, the toolbar allows panning in the plot by clicking and dragging, drawing a box on the plot to zoom to, using the mouse scroll for zooming, saving the plot with the system saving method, resetting the zoom and view of the plot, and turning the hover on/off that shows exact lat/lons and field values when hovering the plot.
All plot types available
x/y (Lat/lon)
y/z (Lat/lev) “zonal mean”
x/t (Lon/time) “hov molar”
y/t (Lat/time) “hov molar”
avg/z (averaged dims/lev) “vertical profile”
avg/t (averaged dims/time) “time series”
polar coordinates (lat/lon) “polar”
tavg x/y (Lat/lon) “time averaged”
iViz will evaluate what dimensions are available in the provided dataset and determine what plot types are available.
Plot type 2, and 5 requires multiple vertical levels in data; type 3, 4, and 6 require multiple time values in data. Type 8 can be created within the tool (index # __).
iViz will by default only create plot types 1 and 2 if present. Additional plot types available will appear in ‘plot types available’ in the tool GUI.
All plot kinds available
For netcdf sources, the plot kinds available include:
quadmesh
contours
filled contours
image
For csv file formats and tabular data like Airnow:
histogram
bar
scatter
points
polygons
box
hexagon
bivariate
Comparing two netcdf files
Launch iViz with desired configuration or input file(s). Click ‘Create comparisons’ to pop out the comparison panel. If you have placed multiple files in the model yaml configuration, they will appear automatically as comparison file options. If not, enter a path to where your data for comparison is located. All netcdf files in that directory will appear in the selection box below.
After selecting a file for comparison, click ‘Add secondary file’ button to create plots for comparison file, and click ‘Create comparison’ button to create difference plots between the two files. Once ‘create comparison’ is clicked, iViz will automatically check the shape and size of the data being compared and apply a scip-py regridding algorithm if needed. iViz will then difference (subtract left to right) the two datasets and plot the difference. This may take 1-2 minutes. Please look to the app loading indicator in the header to see if iViz is loading.
Difference plots available: 1. Difference (left-right) 2. Percent difference (left-right) / ((left+right)/2.0)) 3. Percent change ( ((left-right) / right) * 100 ) 4. Ratio (left/right)
To change the diff type, use the ‘Comparison type’ selection box. The difference plot will automatically update.
If the comparison files variables or coordinates are difference than the first file, the tool will add new selectors for variable, time, lev (if needed) that control the comparison file.
Click ‘Clear comparison’ and ‘Clear secondary file’ buttons to clear plots.
Use plot options to manipulate comparison visualizations, add coastlines and borders, etc.
Visualizing multiple files at once
Click ‘Explore Files’ in the iViz header to pop-out the file explorer tool. Enter a path to a directory or navigate using the file explorer to select multiple netcdf files to visualize. Using your mouse, you can select a file and press the ‘>>’ to move it to ‘Selected options’ or ‘<<’ to a move a file from Selected options to Available options. Click and drag with the mouse to select multiple files at once. When the files desired for plotting are in ‘Selected options’, click ‘Plot all files’ button.
In the main plotting tabs, additional tabs will appear for each file selected. Each file will contain a mini control panel with basic visualization options and interactive features.
When done visualizing a file, press the red ‘X’ in the tabs to close a file.
Time series and time averaging
Click the same ‘Explore Files’ button to pop out the file explorer. With the file explorer, select files to either time average or create a time series plot using the same method as above. To create a time series plot or average, the data source should be the same as the currently selected data source. For example, if you have launched the tool with a GEOS-CCM data source, attempting to create a LIS time series or time average may cause an error.
1. Time series To create a time series plot, enter the variable you would like to plot in the ‘Enter variable(s)’ entry box. Then click ‘Create Time Series Plot’. The time series plot will appear to the right of the main plotting column.
2. Time average To conduct a time average of the data, either leave ‘Use all variables’ checked to use all variables in the datafiles or uncheck it and enter which variable(s) to use int he time averaging. You can enter multiple variables separated by a comma to use multiple selected variables. The time averaged file will be added to the tool’s file selector once finished averaging. The resulting time averaged file will also be saved to the code base location in a folder titled ‘time_average_files’.
Saving and restoring a user session
When done using iViz, to preserve your visualization session, press ‘Save session’ button from the tool header. A date and time stamped yaml file with iViz session will appear in the top level directory of the source code. This is the session file.
At next CLI startup of iViz, pass in the desired session file with the -s flag. Example
python iviz.py -m ccm -s 07.06.2022_16/00/03.iViz_session.yaml
In the notebook, input the session file as an argument to .notebook(). Example:
Import iviz as Iviz
I = iViz()
i.notebook(‘ccm’, session_file=‘07.06.2022_16/00/03.iViz_session.yaml’)
Plot settings available
Plot type + Settings
Plot types available: contains the currently selected plot types in the main layout. Click in the box to see other available plot types.
Transparency: adjust transparency of all plots
Contour color levels: adjust number of contour levels on all plots
Show grid: add gridlines to all plots
Custom title: Enable the custom title, using text entered in ‘Plot title”
Plot title: enter a customized plot title
Number of columns in plot layout: Adjust the number of columns in the main layout. Defaults to 1 - lines all the plots up in one column. 4 plots with 2 columns would be (2,2). Must turn off plot tabs toggle to see effect.
Show coastlines: add Cartopy feature coastlines to all lat/lon plots (includes country borders)
Show states: Add US states (this could be adjusted to add other countries ‘states’ as well if we wanted)
Show lakes: Add large global lakes to all lat/lon plots.
Scale of Cartopy features: the available scales for the coastlines, states, and lakes. Increase the scale to see smaller lakes and states.
Axes + Colorbar
Colorbar location: move all plots colorbars.
Share colorbar range w/ 2nd file – if a second file is present, share the colorbar limits with it.
Start colorbar at zero – plots create a colorbar range based on the data values. This often means that the colorbar
does start at 0 or very close to 0 already, but not always. This will force the colorbar minimum to 0 by force.
XY tab controls only the lat/lon plots + Y axis range: adjust y axis + X axis range: adjust x axis + Invert y axis + Invert x axis + Colorbar normalization: ‘linear’ default, normalization scheme appled to colorbar
Zonal tab controls only YZ [lat/lev] zonal plots + Y axis range + X axis range + Invert y axis + Invert x axis + Additional option to log the y axis + Zonal colorbar normalization
Explore colormaps
All colormaps in matplotlib, bokeh, and colorcet, all reversible, are available in iViz.
Filter colormap category: select different types of colormaps to filter options shown (diverging, rainbow, mono sequential, etc.)
Colormap provider: matplotlib, bokeh, colorcet
Reverse the colormap: all colormaps are reversible
Select from filtered cmaps: make selection of new colormap to be applied to all plots from filtered cmaps
Advanced
Apply basic operation to data: + Enter value to be applied + Select whether to multiply, divide, add, subtract + And Click ‘Apply operation to data’ to apply to data and re-plot
Spatial select + Make regional selections + Click Apply zoom to plot to zoom to that region
Projection + Select from the available Cartopy projections + Click ‘Set projection’ to re-plot the data with the supplied projection.
