[[TOC]] = GRASS GIS Mini Project 2022: Space-Time Dataset Visualization and Improved Interactive Maps for grass.jupyter= For more information about `grass.jupyter` and to learn more about the current API, please visit the [https://grass.osgeo.org/grass81/manuals/libpython/grass.jupyter.html manual page]. || Title: || '''Space-Time Dataset Visualization and Improved Interactive Maps for grass.jupyter'' || || Student Name: || Caitlin Haedrich, [https://cnr.ncsu.edu/geospatial/ North Carolina State University] || || Mentor Name: || Vaclav Petras, Helena Mitasova, Stephan Blumentrath || || !GitHub Fork: || [https://github.com/chaedri/grass/tree/master/python/grass/jupyter View Repo] || || Proposal: || [https://docs.google.com/document/d/1X9gBeDIU8-zDRx-AVmGrkhYw3TwQ_g9WAInG9e4u1SA/edit?usp=sharing View Proposal] || || Budget: || 1000 € (via GRASS GIS [https://grasswiki.osgeo.org/wiki/Student_Grants Student Grants]) || == Abstract == During GSoC 2021, we created “grass.jupyter”, a package that improves the integration of GRASS GIS and Jupyter with a set of functions for displaying GRASS data in Jupyter Notebooks. In its current state, “grass.jupyter” allows users to create static visuals and simple interactive maps. However, several additional features are needed to allow Jupyter users to fully and easily access the power GRASS, including space-time dataset visualization and more options for interactive mapping (such as color or vector attribute access). == Goal == This project had three main objectives: 1. create a class for visualizing space time datasets allowing users to create interactive time-sliders and non-interactive animations such as GIFs. 2. improve the integration of grass and folium, the library used for !InteractiveMap, so that users can access all of folium’s functionality. Currently, !InteractiveMap allows users to add rasters and vectors to folium maps. Users can toggle between layers and export the map in HTML. However, folium allows for much more sophisticated mapping as well: users can control color, vector symbology, create heatmaps (point density maps) and view vector attributes with a click or hover. I propose to create new grass-folium objects that allow users to directly call folium (thus avoiding the problem of needing to wrap the entire folium library and continue to update as folium features change, depreciate and expand). 3. create a function to display vector attribute data in nicely-formatted Pandas or !GeoPandas tables (as opposed to text output which is currently possible with “v.db.select”). == Timeline == {{{#!th style="background: #ddd" rowspan=2 '''Time Period''' }}} {{{#!th style="background: #ddd" colspan=2 '''Milestones''' }}} |----------------------- {{{#!th style="background: #ddd" Tasks }}} {{{#!th style="background: #ddd" Status }}} |----------------------- {{{#!td January 10th - January 14th\\ ''Week 1'' }}} {{{#!td Project Intro Email \\ Start !TimeSeries class }}} {{{#!td Ok \\ Ok }}} |----------------------- {{{#!td January 17th - January 21st \\ ''Week 2'' }}} {{{#!td Draft of !TimeSlider method }}} {{{#!td Ok }}} |----------------------- {{{#!td January 24th - January 28th \\ ''Week 3'' }}} {{{#!td Draft of time_slider and animate }}} {{{#!td Ok }}} |----------------------- {{{#!td January 31st - February 4th \\ ''Break'' }}} {{{#!td -- }}} {{{#!td -- }}} |----------------------- {{{#!td February 7th - February 11th \\ ''Week 4'' }}} {{{#!td Docstrings and comments for time_slider and animate \\ Solution for varying timesteps in !TimeSeries \\ Started GRASS-folium }}} {{{#!td Ok \\ Ok \\ Ok }}} |----------------------- {{{#!td February 14th - February 18th \\ ''Week 5'' }}} {{{#!td Fixed region handling for !InteractiveMap \\ Clean-up of interact_display.py \\ Update `grass_jupyter` Notebook with GRASS-folium }}} {{{#!td Ok \\ Ok \\ Ok }}} |----------------------- {{{#!td February 21st - February 25th \\ ''Week 6'' }}} {{{#!td Write tests for !TimeSeries and GRASS-folium objects }}} {{{#!td Ok }}} |----------------------- {{{#!td February 28th - March 4th\\ ''Week 7'' }}} {{{#!td Write test for !ReprojectionRenderer \\ Merge existing pull requests for GRASS-folium and !TimeSeriesMap }}} {{{#!td Ok \\ Ok }}} |----------------------- {{{#!td March 7th - March 11th\\ ''Week 8'' }}} {{{#!td Implement changes to grass.jupyter class names\\ Write final report }}} {{{#!td Ok \\ Ok }}} == Weekly reports == == Week 2 == Here's a quick update of what I worked on last week for my Mini Project on improving grass.jupyter. **1) What did I accomplish this week?** I started on visualization functions for space time datasets (strds and stvds). I created a `grass.jupyter` class called `TimeSeries` that will eventually have two methods: `TimeSlider` for creating an interactive time slider where users can scroll through the rendered maps and `animate` for creating animations of the space time datasets. I have a draft version of the `TimeSlider` working - currently, users have to switch through the maps with a drop down menu instead of a slider (or scroll with arrow keys) . You can test it out on Binder [https://mybinder.org/v2/gh/chaedri/grass/Jupyter-Timeseries?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2FTemporal.ipynb here] and view the PR [https://github.com/OSGeo/grass/pull/2010 on GitHub]. **2) What do I plan to do next week?** * Switch from a dropdown menu to a !SelectionSlider for the `TimeSlider` visualization * Create more robust ways to add background and overlay layers to the `TimeSlider` * Start the `TimeSeries.animate()` function for animating space time datasets **3) Am I blocked on anything?** No, I'm not currently blocked on anything. Feedback, thoughts and comments welcome! == Week 3 == Here's a quick update on my Mini Project for improving `grass.jupyter`. **1) What did I accomplish this week?** * Switched from a Dropdown menu to a !SelectionSlider for the `time_slider` visualization. I found a bug in ipywidgets which slowed me down a bit this week. I ended up [https://github.com/jupyter-widgets/ipywidgets/issues/3363 opening an issue on GitHub] and it was quickly addressed (go ipywidgets!). * Wrote `TimeSeries.animate()` function for animating space time datasets You can [https://mybinder.org/v2/gh/chaedri/grass/Jupyter-Timeseries?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2FTemporal.ipynb test it out on Binder] and [https://github.com/OSGeo/grass/pull/2010 view the PR]. **2) What do I plan to do next week?** * Cleanup and write nice docstrings for !TimeSeries visualizations * The current visualizations are not great for multitemporal datasets (i.e. datasets with varying time steps), since the steps in the animation and time slider are all regularly spaced. Next week, I'll work on a way to address this. * Maybe start working on GRASS-folium integration for interactive mapping **3) Am I blocked on anything?** * No, I'm not currently blocked on anything. Feedback, thoughts and comments welcome! == Week 4 == Here's an update on what I've been working on over the last two weeks for my Mini Project on improving `grass.jupyter`. **1) What did I accomplish over the past week?** * Took a week off to catch up on other work * Made minor edits, cleaned-up and wrote docstrings for !TimeSeries visualizations ([https://github.com/OSGeo/grass/pull/2010 view the PR]) * Addressed visualizations issues with datasets that have varying time steps ([https://github.com/OSGeo/grass/pull/2010 view the PR]) * Started improving GRASS-folium integration for interactive mapping by restructuring interact_display.py and adding classes that can be passed to folium ([https://github.com/OSGeo/grass/pull/2208 view the PR]) You can test it out the !TimeSeries visualizations (animate and time_slider) [https://mybinder.org/v2/gh/chaedri/grass/Jupyter-Timeseries?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Ftemporal.ipynb in Binder] **2) What do I plan to do next week?** * Work on cleaning up temporary files created by grass.jupyter * Region handling for !InteractiveMap (as you can see in the [https://mybinder.org/v2/gh/chaedri/grass/GRASS-folium?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fgrass_jupyter.ipynb Binder], the map is no longer automatically zoomed to the computation region) * Docstrings, comments, cleanup interact_display.py * Update grass_jupyter Notebook to include GRASS-folium example * Begin functions for viewing vector attributes in Pandas **3) Am I blocked on anything?** * No, I'm not currently blocked on anything. Feedback, thoughts and comments welcome! == Week 5 == Here's my weekly update on my Mini Project on improving `grass.jupyter`. **1) What did I accomplish over the past two weeks?** * Fixed region handling for !InteractiveMap (now, it automatically zooms to the computation region or layer added!) and added tiles keyword argument so you can use a tile of your choice in the background by passing one of folium's defaults or a URL address (cool!) ([https://github.com/OSGeo/grass/pull/2208 view the PR]) * Docstrings, comments, Pylint cleanup, and tempfile clean-up of interact_display.py ([https://github.com/OSGeo/grass/pull/2208 view the PR]) * Update `grass_jupyter` Notebook to include GRASS-folium example ([https://github.com/OSGeo/grass/pull/2208 view the PR]) You can test the restructured !InteractiveMap and GRASS-folium integration in Binder at [https://mybinder.org/v2/gh/chaedri/grass/GRASS-folium?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fgrass_jupyter.ipynb in Binder]. **2) What do I plan to do next week?** * Merge existing PRs * Begin functions for viewing vector attributes in Pandas **3) Am I blocked on anything?** * No, I'm not currently blocked on anything. Feedback, thoughts and comments welcome! == Week 6/7 == I'm just sending out an update on my GRASS Mini Project on improving grass.jupyter. **1) What have I accomplished since my last update?** * I've been having frequent meetings with Vashek Petras to discuss the structure of grass.jupyter and to improve the code (thanks to Vashek, I'm learning a lot of better coding practices and I think the current code is a lot more robust than before) * Improved region handling, method for filling gaps in space-time datasets, baselayer/overlay methods for !TimeSeries ([https://github.com/OSGeo/grass/pull/2010 view the PR]) **2) What do I plan to do next week?** * Write tests for !TimeSeries and GRASS-folium objects * Functions for viewing vector attributes in Pandas **3) Am I blocked on anything?** * No, I'm not currently blocked on anything. Feedback, thoughts and comments welcome! == Week 7 == I'm just sending out an update on my GRASS Mini Project on improving grass.jupyter. **1) What have I accomplished since my last update?** * I continue to have regular meetings with Vashek Petras. This past week we discussed how to decrease the rendering time for !TimeSeries objects and using the pytest package for writing tests. * Wrote tests for !TimeSeries ([https://github.com/OSGeo/grass/pull/2010 view the PR]) * Took a week off for our Spring Break. **2) What do I plan to do next week?** * Write test for ([https://github.com/OSGeo/grass/pull/2208 ReprojectionRenderer]) * Merge existing pull requests ([https://github.com/OSGeo/grass/pull/2010 here] and [https://github.com/OSGeo/grass/pull/2208 here]) **3) Am I blocked on anything?** * One thing Vashek Petras and I have been discussing is the naming of classes and methods in grass.jupyter. With the release of 8.2 coming up, we're hoping to produce a relatively stable API and want to solicit some feedback and ideas on naming conventions so they could be changed, if needed, before the release. I've opened an Issue on [https://github.com/OSGeo/grass/issues/2283 GitHub]. Feedback, thoughts and comments welcome! == Week 8 == This is my last update before the final report! I'm finally wrapping up work on the grass.jupyter mini grant. We're just putting the final changes in the two PRs that I've been working on for a while ([https://github.com/OSGeo/grass/pull/2010 here] and [https://github.com/OSGeo/grass/pull/2208 here]) and getting ready to implement the grass.jupyter naming conventions before the release of 8.2 ([https://github.com/OSGeo/grass/issues/2283 GitHub Issue]). Here's a quick summary: **1) What have I accomplished since my last update?** * I continue to have regular meetings with Vashek Petras. This week, we talked about testing and some minor edits for the existing pull requests ([https://github.com/OSGeo/grass/pull/2010 here] and [https://github.com/OSGeo/grass/pull/2208 here]) * Wrote tests for ReprojectionRenderer - a class that handles reprojecting vector and raster data for folium ([https://github.com/OSGeo/grass/pull/2208 PR]) * Had some productive discussions about naming conventions for grass.jupyter ([https://github.com/OSGeo/grass/issues/2283 GitHub Issue]) **2) What do I plan to do next week?** * Merge existing pull requests ([https://github.com/OSGeo/grass/pull/2010 here] and [https://github.com/OSGeo/grass/pull/2208 here]) * Implement changes to grass.jupyter class names ([https://github.com/OSGeo/grass/issues/2283 GitHub Issue]) * Write final report **3) Am I blocked on anything?** * No Feedback, thoughts and comments welcome! == Final Report == **Title:** Space-Time Dataset Visualization and Improved Interactive Maps for `grass.jupyter` Last week, we wrapped a final push on `grass.jupyter` and are excited for its official release with GRASS 8.2. Here's my final report along with a summary of `grass.jupyter` changes introduced as part of the project. Thank you all for your support, feedback and testing over the past few months! **The state of the art BEFORE the start of the Mini Project:** During GSoC 2021, we created `grass.jupyter`, a package that improves the integration of GRASS GIS and Jupyter with a set of functions for displaying GRASS data in Jupyter Notebooks. In its previous state, `grass.jupyter` allows users to create static visuals and simple interactive maps. However, several additional features are needed to allow Jupyter users to fully and easily access the power GRASS, including space-time dataset visualization and more options for interactive mapping. **Project Goals:** In preparation for the stable release of `grass.jupyter` with GRASS 8.2, this project had three main goals: (1) create space time dataset visualizations for use in Jupyter Notebooks, (2) improve the integration of GRASS with folium (leaflet library for Python) and (3) write a function for displaying vector attributes in nicely-formatted tables (using Pandas or !GeoPandas). Along the way, we also wanted to finalize the naming of `grass.jupyter` classes and create documentation (thank you Vaclav Petras). **The state of the art AFTER the Mini Project:** 1. New !TimeSeriesMap class that creates ipywidget time sliders of space time datasets (see attached timeseriesmap.png) and [https://mybinder.org/v2/gh/OSGeo/grass/main?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Ftemporal.ipynb a notebook] documenting it's usage 2. Improved GRASS-folium integration allowing rasters and vectors to be added to existing folium maps (see attached grass-folium.png) and an updated [https://mybinder.org/v2/gh/OSGeo/grass/main?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fgrass_jupyter.ipynb notebook] demonstrating its usage 3. Updated class names: * !GrassRenderer -> Map * !Grass3dRenderer -> Map3D * !InteractiveMap * !TimeSeries -> !TimeSeriesMap 4. Thanks to Vaclav Petras, we also have [https://grass.osgeo.org/grass81/manuals/libpython/grass.jupyter.html a manual page] for `grass.jupyter` 5. I didn't end up working to integrate GRASS and Pandas. It seems that it is quite straightforward to display vector attributes in nicely-formatted Pandas tables. For example: {{{ #!div style="font-size: 80%" {{{#!python import json import pandas as pd pd.DataFrame(json.loads(gs.read_command("v.db.select", map="viewpoints", columns="cat,height", layer=2, format="json"))["records"]) }}} }}} There are other outputs that would be nice to display in nice Pandas tables, like text output from r.univar, r.stats, or t.vect.list. However, this is difficult since there is no standard output that is easily parse-able to pandas. I think the best way would be to create a standard json or csv output for all modules that return text. Then, it would be simple to take any module output and convert to a nice-looking Pandas table. **Next Steps:** 1. Bug: !InteractiveMap does not honor `use_region=True` for vectors. 2. Bug: !InteractiveMap for Jupyter does not handle not existing data properly ([https://github.com/OSGeo/grass/issues/2302 GitHub Issue]) 3. !InteractiveMap: add legend options for rasters, support !simpleCRS for faster rendering, !ToolTip integration for vector attributes 4. Continue Pandas integration by adding standard json or csv output to modules that return text 5. ... And many others! I think there's still lots of ways we can improve and expand the integration of GRASS and Jupyter. Ideas welcome. You can follow next steps for grass.jupyter on our project page on [https://github.com/OSGeo/grass/projects/7 GitHub]. Thank you again to Vaclav Petras and Anna Petrasova for their guidance and contributions to grass.jupyter. And, another thank you to Vero Andreo, Stefan Blumentrath and Markus Neteler for their feedback and testing! [[Image(timeseriesmap.PNG, 600px, align=left)]] [[Image(grass-folium.PNG, 600px, align=center)]]