iViz: the interactive tool ========================== The interactive tool, **iViz**, can be launched from the top level directory using a CLI or directly in a Jupyter notebook. This section covers the following: 1. `How to run iViz`_ 2. `Use iViz on Discover`_ 3. `Data sources and types currently supported`_ 4. `Visualize configuration supported data sources`_ 5. `Visualize satellite data (beta)`_ 6. `Visualize tabular data formats (beta)`_ 7. `Customize configuration files on Discover`_ 8. `iViz layout and controls`_ 9. `Bokeh based plotting`_ 10. `All plot types available`_ 11. `All plot kinds available`_ 12. `Comparing two netcdf files`_ 13. `Visualizing multiple files at once`_ 14. `Time series and time averaging`_ 15. `Saving and restoring a user session`_ 16. `Plot settings available`_ How to run iViz --------------- iViz can be run from the command line or from a Jupyter notebook. The inputs required to run the tool are the same, regardless of the system or manner of use. **All arguments are optional but either an input, a source, or both, must be provided** .. list-table:: :header-rows: 1 * - Flag - Name - Description * - -i - --input - A path to a file or directory * - -s - --source - Source of data input (model or observation) with accompanying config files * - -d - --dtype - If iViz is unable to determine data type, enter data type of input (netcdf, hdf, tabular) * - -se - --sesssion - Restore a previous visualization session with a yaml file saved in iViz * - -p - --plot_options - Provide plot options to appl to plots with a yaml file saved in iViz Use iViz on Discover -------------------- Log into Discover JupyterHub system and start a session at: .. code-block:: https://jh-discover.nccs.nasa.gov/ Open a notebook and select the Python `viz` environment. iViz and eViz code is installed on Discover. In a notebook cell, run the following command in order to import and instantiate iViz. .. code-block:: from eviz.lib.iviz_base import Iviz iviz = Iviz() Once iViz is imported, start the tool with the make_notebook function. Provide input to the tool using one or more of the arguments listed in `How to run iViz`_ . All of the commands in the cell below are correct and runnable. .. code-block:: iviz.make_notebook(s='ccm') iviz.make_notebook(s='wrf') iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/lis/LIS_HIST_201507121200.d01.nc', s='lis') iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/cf') After running one of the commands above in a cell, the visualization dashboard will be output in the cell below. A session file from a previous iViz session, or a plot options file saved in iViz, can also be input to the make_notebook function to restore a session in iViz or apply a set of plot options to plots. .. code-block:: iviz.make_notebook(s='ccm', p='my_plot_options.yaml') iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/wrf/', s='wrf', p='my_plot_options.yaml') iviz.make_notebook(s='cf', se='/discover/nobackup/vvalenti/22.01.2023_18:39:20.iViz_session.yaml') Data sources and types currently supported ------------------------------------------ Inputs currently supported: 1. A supported model or observation data source with configuration files (-s --source) 2. Path to a directory with netcdf files in it (-i –-input) 3. Path to a file that is either netcdf, hdf, or tabular data type (-i –-input) Currently supported configurable data sources: .. list-table:: :header-rows: 1 * - -s flag keywords - Source (model or observations) - Data type * - ccm - GEOS-CCM (Chemistry Climate Model) - netcdf * - cf - GEOS-CF (Composite Forecast) - netcdf * - wrf - WRF (Weather Research Forecast) - netcdf * - lis - LIS (Land Information System) - netcdf * - airnow - EPA Airnow - tabular Configurable data sources contain additional information regarding model or observations specific fields available, like unit conversions, and ability to add information like experiment name, and where to save output plots. If you would like to add a data source, either a model or observations, please contact the developers. Additional data sources supported without configuration: .. list-table:: :header-rows: 1 * - Source (model or observations) - Data product - Data type - Where to access or download * - Landsat - Level 3 global gridded - hdf - https://earthexplorer.usgs.gov/ * - Ozone Monitoring Instrument (OMI) - Level 3 global gridded - hdf - https://www.airnow.gov/ **Examples for each of these data sources and types below** Visualize configuration supported data sources ---------------------------------------------- Configuration files are stored in the 'config' directory of the codebase. iViz and eViz automatically use read-only config files stored on Discover. To edit these files for personal use, refer to `Customize configuration files on Discover`_ Run any of the commands below .. list-table:: :header-rows: 1 * - Jupter notebook (Discover) - CLI * - :code:`iviz.make_notebook(s='ccm')` - :code:`python iviz.py -s ccm` * - :code:`iviz.make_notebook(s='wrf')` - :code:`python iviz.py -s wrf` * - :code:`iviz.make_notebook(s='cf')` - :code:`python iviz.py -s cf` * - :code:`iviz.make_notebook(s='lis')` - :code:`python iviz.py -s lis` * - :code:`iviz.make_notebook(s='airnow')` - :code:`python iviz.py -s airnow` The source argument can be used with or without the input argument. If a file or directory path is provided as input without a source, iViz will use the coordinates of the data file to try to determine the source. If iViz is unable to determine the source, an -s input is required. .. list-table:: :header-rows: 1 * - Jupter notebook (Discover) - CLI * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/lis/LIS_HIST_201507121200.d01.nc', s='lis')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/lis/LIS_HIST_201507121200.d01.nc -s lis` * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/wrf', s='wrf')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/wrf -s wrf` * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/cf')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/cf` * - :code:`iviz.make_notebook(i='/discover/nobackup/vvalenti/data/')` - :code:`python iviz.py -i /discover/nobackup/vvalenti/data/` * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/airnow/', s='airnow')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/airnow -s airnow` Visualize satellite data (beta) ------------------------------- .. list-table:: :header-rows: 1 * - Jupter notebook (Discover) - CLI * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/satellite_data/LT50830152011198GLC00.hdf')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/satellite_data/LT50830152011198GLC00.hdf` * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/satellite_data/OMI-Aura_L3-OMTO3e_2022m0709_v003-2022m0711t031807.he5')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/satellite_data/OMI-Aura_L3-OMTO3e_2022m0709_v003-2022m0711t031807.he5` Visualize tabular data formats (beta) ------------------------------------- Tabular data is data that is stored in rows and columns. This includes most CSV file formats. Tabular data is opened with pandas into a DataFrame. .. list-table:: :header-rows: 1 * - Jupter notebook (Discover) - CLI * - :code:`iviz.make_notebook(i='/discover/nobackup/projects/jh_tutorials/eviz/sample_data/tabular/airports.csv')` - :code:`python iviz.py -i /discover/nobackup/projects/jh_tutorials/eviz/sample_data/tabular/airports.csv` Customize configuration files on Discover ----------------------------------------- The config directory containing data source specific configuration files is read-only on Discover. To edit and customize configuration files, copy the config directory to your personal $NOBACKUP space. For example: :code:`cp -r /discover/nobackup/projects/jh_tutorials/eviz/config /discover/nobackup/$USER/eviz` Then, set the environment variable ``` EVIZ_CONFIG_PATH ``` to the location of this config directory. To do this within the Jupyter notebook, use the os module to set the environment variable. For example: .. code-block:: import os os.environ['EVIZ_CONFIG_PATH'] = '/discover/nobackup/$USER/eviz/config/' In a bash shell, set the environment variable with the following command: :code:`export EVIZ_CONFIG_PATH=/discover/nobackup/$USER/eviz/config` iViz layout and controls ------------------------ Header ^^^^^^ .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/header_iviz.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/headre_iviz.png :width: 700 Sidebar ^^^^^^^ .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/sidebar_iviz.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/sidebar_iviz.png :width: 400 Plotting area ^^^^^^^^^^^^^ .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plotting_area_iviz.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plotting_area_iviz.png :width: 800 Bokeh based plotting -------------------- Plotting in **iViz** is bokeh based. All plots come with the Bokeh interactive toolbar. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/bokeh_plot.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/bokeh_plot.png :width: 400 .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/bokeh_bar.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/bokeh_bar.png :width: 400 From left to right, the toolbar allows panning in the plot by clicking and dragging, drawing a box on the plot to zoom to, using the mouse scroll for zooming, saving the plot with the system saving method, resetting the zoom and view of the plot, and turning the hover on/off that shows exact lat/lons and field values when hovering the plot. All plot types available ------------------------ 1. x/y (Lat/lon) 2. y/z (Lat/lev) “zonal mean” 3. x/t (Lon/time) “hov molar” 4. y/t (Lat/time) “hov molar” 5. avg/z (averaged dims/lev) “vertical profile” 6. avg/t (averaged dims/time) “time series” 7. polar coordinates (lat/lon) “polar” 8. tavg x/y (Lat/lon) “time averaged” **iViz** will evaluate what dimensions are available in the provided dataset and determine what plot types are available. Plot type 2, and 5 requires multiple vertical levels in data; type 3, 4, and 6 require multiple time values in data. Type 8 can be created within the tool (index # __). **iViz** will by default only create plot types 1 and 2 if present. Additional plot types available will appear in ‘plot types available’ in the tool GUI. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_types_avail1.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_types_avail1.png :width: 400 .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_types_avail2.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_types_avail2.png :width: 400 All plot kinds available ------------------------ For netcdf sources, the plot kinds available include: 1. quadmesh 2. contours 3. filled contours 4. image For csv file formats and tabular data like Airnow: 1. histogram 2. bar 3. scatter 4. points 5. polygons 6. box 7. hexagon 8. bivariate Comparing two netcdf files -------------------------- Launch **iViz** with desired configuration or input file(s). Click ‘Create comparisons’ to pop out the comparison panel. If you have placed multiple files in the model yaml configuration, they will appear automatically as comparison file options. If not, enter a path to where your data for comparison is located. All netcdf files in that directory will appear in the selection box below. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files.png :width: 500 After selecting a file for comparison, click ‘Add secondary file’ button to create plots for comparison file, and click ‘Create comparison’ button to create difference plots between the two files. Once ‘create comparison’ is clicked, **iViz** will automatically check the shape and size of the data being compared and apply a scip-py regridding algorithm if needed. **iViz** will then difference (subtract left to right) the two datasets and plot the difference. This may take 1-2 minutes. Please look to the app loading indicator in the header to see if **iViz** is loading. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files2.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files2.png :width: 500 Difference plots available: 1. Difference (left-right) 2. Percent difference (left-right) / ((left+right)/2.0)) 3. Percent change ( ((left-right) / right) * 100 ) 4. Ratio (left/right) .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files3.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/compare_files3.png :width: 500 To change the diff type, use the ‘Comparison type’ selection box. The difference plot will automatically update. If the comparison files variables or coordinates are difference than the first file, the tool will add new selectors for variable, time, lev (if needed) that control the comparison file. Click ‘Clear comparison’ and ‘Clear secondary file’ buttons to clear plots. Use plot options to manipulate comparison visualizations, add coastlines and borders, etc. Visualizing multiple files at once ---------------------------------- Click 'Explore Files' in the iViz header to pop-out the file explorer tool. Enter a path to a directory or navigate using the file explorer to select multiple netcdf files to visualize. Using your mouse, you can select a file and press the ‘>>’ to move it to ‘Selected options’ or ‘<<’ to a move a file from Selected options to Available options. Click and drag with the mouse to select multiple files at once. When the files desired for plotting are in ‘Selected options’, click ‘Plot all files’ button. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file1.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file1.png :width: 550 In the main plotting tabs, additional tabs will appear for each file selected. Each file will contain a mini control panel with basic visualization options and interactive features. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file2.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file2.png :width: 750 When done visualizing a file, press the red ‘X’ in the tabs to close a file. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file3.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/multi_file3.png :width: 700 Time series and time averaging ---------------------------------- Click the same ‘Explore Files’ button to pop out the file explorer. With the file explorer, select files to either time average or create a time series plot using the same method as above. To create a time series plot or average, the data source should be the same as the currently selected data source. For example, if you have launched the tool with a GEOS-CCM data source, attempting to create a LIS time series or time average may cause an error. 1. Time series To create a time series plot, enter the variable you would like to plot in the 'Enter variable(s)' entry box. Then click 'Create Time Series Plot'. The time series plot will appear to the right of the main plotting column. 2. Time average To conduct a time average of the data, either leave 'Use all variables' checked to use all variables in the datafiles or uncheck it and enter which variable(s) to use int he time averaging. You can enter multiple variables separated by a comma to use multiple selected variables. The time averaged file will be added to the tool's file selector once finished averaging. The resulting time averaged file will also be saved to the code base location in a folder titled 'time_average_files'. Saving and restoring a user session ----------------------------------- When done using **iViz**, to preserve your visualization session, press ‘Save session’ button from the tool header. A date and time stamped yaml file with **iViz** session will appear in the top level directory of the source code. This is the session file. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/save_session.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/save_session.png :width: 650 At next CLI startup of **iViz**, pass in the desired session file with the -s flag. Example .. code-block:: python iviz.py -m ccm -s 07.06.2022_16/00/03.iViz_session.yaml In the notebook, input the session file as an argument to .notebook(). Example: .. code-block:: Import iviz as Iviz I = iViz() i.notebook(‘ccm’, session_file=‘07.06.2022_16/00/03.iViz_session.yaml’) Plot settings available ----------------------- Plot type + Settings ^^^^^^^^^^^^^^^^^^^^ + Plot types available: contains the currently selected plot types in the main layout. Click in the box to see other available plot types. + Transparency: adjust transparency of all plots + Contour color levels: adjust number of contour levels on all plots + Show grid: add gridlines to all plots + Custom title: Enable the custom title, using text entered in ‘Plot title” + Plot title: enter a customized plot title + Number of columns in plot layout: Adjust the number of columns in the main layout. Defaults to 1 - lines all the plots up in one column. 4 plots with 2 columns would be (2,2). Must turn off plot tabs toggle to see effect. + Show coastlines: add Cartopy feature coastlines to all lat/lon plots (includes country borders) + Show states: Add US states (this could be adjusted to add other countries ‘states’ as well if we wanted) + Show lakes: Add large global lakes to all lat/lon plots. + Scale of Cartopy features: the available scales for the coastlines, states, and lakes. Increase the scale to see smaller lakes and states. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts1.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts1.png :width: 400 Axes + Colorbar ^^^^^^^^^^^^^^^ + Colorbar location: move all plots colorbars. + Share colorbar range w/ 2nd file – if a second file is present, share the colorbar limits with it. + Start colorbar at zero – plots create a colorbar range based on the data values. This often means that the colorbar + does start at 0 or very close to 0 already, but not always. This will force the colorbar minimum to 0 by force. + XY tab controls only the lat/lon plots + Y axis range: adjust y axis + X axis range: adjust x axis + Invert y axis + Invert x axis + Colorbar normalization: ‘linear’ default, normalization scheme appled to colorbar + Zonal tab controls only YZ [lat/lev] zonal plots + Y axis range + X axis range + Invert y axis + Invert x axis + Additional option to log the y axis + Zonal colorbar normalization .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts2.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts2.png :width: 400 Explore colormaps ^^^^^^^^^^^^^^^^^ + All colormaps in matplotlib, bokeh, and colorcet, all reversible, are available in **iViz**. + Filter colormap category: select different types of colormaps to filter options shown (diverging, rainbow, mono sequential, etc.) + Colormap provider: matplotlib, bokeh, colorcet + Reverse the colormap: all colormaps are reversible + Select from filtered cmaps: make selection of new colormap to be applied to all plots from filtered cmaps .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts3.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts3.png :width: 400 Advanced ^^^^^^^^ + Apply basic operation to data: + Enter value to be applied + Select whether to multiply, divide, add, subtract + And Click ‘Apply operation to data’ to apply to data and re-plot + Spatial select + Make regional selections + Click Apply zoom to plot to zoom to that region + Projection + Select from the available Cartopy projections + Click ‘Set projection’ to re-plot the data with the supplied projection. .. image:: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts4.png :alt: https://git.smce.nasa.gov/astg/scratch/static/-/raw/main/ug/plot_opts4.png :width: 400