GRASS GIS Mini Project 2022: Space-Time Dataset Visualization and Improved Interactive Maps for grass.jupyter

For more information about grass.jupyter and to learn more about the current API, please visit the manual page.

Title:	Space-Time Dataset Visualization and Improved Interactive Maps for grass.jupyter
Student Name:	Caitlin Haedrich, ​North Carolina State University
Mentor Name:	Vaclav Petras, Helena Mitasova, Stephan Blumentrath
GitHub Fork:	​View Repo
Proposal:	​View Proposal
Budget:	1000 € (via GRASS GIS ​Student Grants)

Abstract

During GSoC 2021, we created “grass.jupyter”, a package that improves the integration of GRASS GIS and Jupyter with a set of functions for displaying GRASS data in Jupyter Notebooks. In its current state, “grass.jupyter” allows users to create static visuals and simple interactive maps. However, several additional features are needed to allow Jupyter users to fully and easily access the power GRASS, including space-time dataset visualization and more options for interactive mapping (such as color or vector attribute access).

Goal

This project had three main objectives:

1. create a class for visualizing space time datasets allowing users to create interactive time-sliders and non-interactive animations such as GIFs.

2. improve the integration of grass and folium, the library used for InteractiveMap, so that users can access all of folium’s functionality. Currently, InteractiveMap allows users to add rasters and vectors to folium maps. Users can toggle between layers and export the map in HTML. However, folium allows for much more sophisticated mapping as well: users can control color, vector symbology, create heatmaps (point density maps) and view vector attributes with a click or hover. I propose to create new grass-folium objects that allow users to directly call folium (thus avoiding the problem of needing to wrap the entire folium library and continue to update as folium features change, depreciate and expand).

3. create a function to display vector attribute data in nicely-formatted Pandas or GeoPandas tables (as opposed to text output which is currently possible with “v.db.select”).

Timeline

Time Period	Milestones
	Tasks	Status
January 10th - January 14th Week 1	Project Intro Email Start TimeSeries class	Ok Ok
January 17th - January 21st Week 2	Draft of TimeSlider method	Ok
January 24th - January 28th Week 3	Draft of time_slider and animate	Ok
January 31st - February 4th Break	--	--
February 7th - February 11th Week 4	Docstrings and comments for time_slider and animate Solution for varying timesteps in TimeSeries Started GRASS-folium	Ok Ok Ok
February 14th - February 18th Week 5	Fixed region handling for InteractiveMap Clean-up of interact_display.py Update grass_jupyter Notebook with GRASS-folium	Ok Ok Ok
February 21st - February 25th Week 6	Write tests for TimeSeries and GRASS-folium objects	Ok
February 28th - March 4th Week 7	Write test for ReprojectionRenderer Merge existing pull requests for GRASS-folium and TimeSeriesMap	Ok Ok
March 7th - March 11th Week 8	Implement changes to grass.jupyter class names Write final report	Ok Ok

Weekly reports

Week 2

Here's a quick update of what I worked on last week for my Mini Project on improving grass.jupyter.

1) What did I accomplish this week? I started on visualization functions for space time datasets (strds and stvds). I created a grass.jupyter class called TimeSeries that will eventually have two methods: TimeSlider for creating an interactive time slider where users can scroll through the rendered maps and animate for creating animations of the space time datasets. I have a draft version of the TimeSlider working - currently, users have to switch through the maps with a drop down menu instead of a slider (or scroll with arrow keys) . You can test it out on Binder ​here and view the PR ​on GitHub.

2) What do I plan to do next week?

• Switch from a dropdown menu to a SelectionSlider for the TimeSlider visualization
• Create more robust ways to add background and overlay layers to the TimeSlider
• Start the TimeSeries.animate() function for animating space time datasets

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Feedback, thoughts and comments welcome!

Week 3

Here's a quick update on my Mini Project for improving grass.jupyter.

1) What did I accomplish this week?

• Switched from a Dropdown menu to a SelectionSlider for the time_slider visualization. I found a bug in ipywidgets which slowed me down a bit this week. I ended up ​opening an issue on GitHub and it was quickly addressed (go ipywidgets!).
• Wrote TimeSeries.animate() function for animating space time datasets You can ​test it out on Binder and ​view the PR.

2) What do I plan to do next week?

• Cleanup and write nice docstrings for TimeSeries visualizations
• The current visualizations are not great for multitemporal datasets (i.e. datasets with varying time steps), since the steps in the animation and time slider are all regularly spaced. Next week, I'll work on a way to address this.
• Maybe start working on GRASS-folium integration for interactive mapping

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, thoughts and comments welcome!

Week 4

Here's an update on what I've been working on over the last two weeks for my Mini Project on improving grass.jupyter.

1) What did I accomplish over the past week?

• Took a week off to catch up on other work
• Made minor edits, cleaned-up and wrote docstrings for TimeSeries visualizations (​view the PR)
• Addressed visualizations issues with datasets that have varying time steps (​view the PR)
• Started improving GRASS-folium integration for interactive mapping by restructuring interact_display.py and adding classes that can be passed to folium (​view the PR)

You can test it out the TimeSeries visualizations (animate and time_slider) ​in Binder

2) What do I plan to do next week?

• Work on cleaning up temporary files created by grass.jupyter
• Region handling for InteractiveMap (as you can see in the ​Binder, the map is no longer automatically zoomed to the computation region)
• Docstrings, comments, cleanup interact_display.py
• Update grass_jupyter Notebook to include GRASS-folium example
• Begin functions for viewing vector attributes in Pandas

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, thoughts and comments welcome!

Week 5

Here's my weekly update on my Mini Project on improving grass.jupyter.

1) What did I accomplish over the past two weeks?

• Fixed region handling for InteractiveMap (now, it automatically zooms to the computation region or layer added!) and added tiles keyword argument so you can use a tile of your choice in the background by passing one of folium's defaults or a URL address (cool!) (​view the PR)
• Docstrings, comments, Pylint cleanup, and tempfile clean-up of interact_display.py (​view the PR)
• Update grass_jupyter Notebook to include GRASS-folium example (​view the PR)

You can test the restructured InteractiveMap and GRASS-folium integration in Binder at ​in Binder.

2) What do I plan to do next week?

• Merge existing PRs
• Begin functions for viewing vector attributes in Pandas

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, thoughts and comments welcome!

Week 6/7

I'm just sending out an update on my GRASS Mini Project on improving grass.jupyter.

1) What have I accomplished since my last update?

• I've been having frequent meetings with Vashek Petras to discuss the structure of grass.jupyter and to improve the code (thanks to Vashek, I'm learning a lot of better coding practices and I think the current code is a lot more robust than before)
• Improved region handling, method for filling gaps in space-time datasets, baselayer/overlay methods for TimeSeries (​view the PR)

2) What do I plan to do next week?

• Write tests for TimeSeries and GRASS-folium objects
• Functions for viewing vector attributes in Pandas

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, thoughts and comments welcome!

Week 7

I'm just sending out an update on my GRASS Mini Project on improving grass.jupyter.

1) What have I accomplished since my last update?

• I continue to have regular meetings with Vashek Petras. This past week we discussed how to decrease the rendering time for TimeSeries objects and using the pytest package for writing tests.
• Wrote tests for TimeSeries (​view the PR)
• Took a week off for our Spring Break.

2) What do I plan to do next week?

• Write test for (​ReprojectionRenderer)
• Merge existing pull requests (​here and ​here)

3) Am I blocked on anything?

• One thing Vashek Petras and I have been discussing is the naming of classes and methods in grass.jupyter. With the release of 8.2 coming up, we're hoping to produce a relatively stable API and want to solicit some feedback and ideas on naming conventions so they could be changed, if needed, before the release. I've opened an Issue on ​GitHub.

Feedback, thoughts and comments welcome!

Week 8

This is my last update before the final report! I'm finally wrapping up work on the grass.jupyter mini grant. We're just putting the final changes in the two PRs that I've been working on for a while (​here and ​here) and getting ready to implement the grass.jupyter naming conventions before the release of 8.2 (​GitHub Issue). Here's a quick summary:

1) What have I accomplished since my last update?

• I continue to have regular meetings with Vashek Petras. This week, we talked about testing and some minor edits for the existing pull requests (​here and ​here)
• Wrote tests for ReprojectionRenderer - a class that handles reprojecting vector and raster data for folium (​PR)
• Had some productive discussions about naming conventions for grass.jupyter (​GitHub Issue)

2) What do I plan to do next week?

• Merge existing pull requests (​here and ​here)
• Implement changes to grass.jupyter class names (​GitHub Issue)
• Write final report

3) Am I blocked on anything?

• No

Feedback, thoughts and comments welcome!

Final Report

Title: Space-Time Dataset Visualization and Improved Interactive Maps for grass.jupyter

Last week, we wrapped a final push on grass.jupyter and are excited for its official release with GRASS 8.2. Here's my final report along with a summary of grass.jupyter changes introduced as part of the project. Thank you all for your support, feedback and testing over the past few months!

The state of the art BEFORE the start of the Mini Project:

During GSoC 2021, we created grass.jupyter, a package that improves the integration of GRASS GIS and Jupyter with a set of functions for displaying GRASS data in Jupyter Notebooks. In its previous state, grass.jupyter allows users to create static visuals and simple interactive maps. However, several additional features are needed to allow Jupyter users to fully and easily access the power GRASS, including space-time dataset visualization and more options for interactive mapping.

Project Goals:

In preparation for the stable release of grass.jupyter with GRASS 8.2, this project had three main goals: (1) create space time dataset visualizations for use in Jupyter Notebooks, (2) improve the integration of GRASS with folium (leaflet library for Python) and (3) write a function for displaying vector attributes in nicely-formatted tables (using Pandas or GeoPandas). Along the way, we also wanted to finalize the naming of grass.jupyter classes and create documentation (thank you Vaclav Petras).

The state of the art AFTER the Mini Project:

1. New TimeSeriesMap class that creates ipywidget time sliders of space time datasets (see attached timeseriesmap.png) and ​a notebook documenting it's usage
2. Improved GRASS-folium integration allowing rasters and vectors to be added to existing folium maps (see attached grass-folium.png) and an updated ​notebook demonstrating its usage
3. Updated class names:

• GrassRenderer -> Map
• !Grass3dRenderer -> Map3D
• InteractiveMap
• TimeSeries -> TimeSeriesMap

4. Thanks to Vaclav Petras, we also have a manual page for grass.jupyter
5. I didn't end up working to integrate GRASS and Pandas. It seems that it is quite straightforward to display vector attributes in nicely-formatted Pandas tables. For example:

   import json
   import pandas as pd
   pd.DataFrame(json.loads(gs.read_command("v.db.select", map="viewpoints", columns="cat,height", layer=2, format="json"))["records"])
   

There are other outputs that would be nice to display in nice Pandas tables, like text output from r.univar, r.stats, or t.vect.list. However, this is difficult since there is no standard output that is easily parse-able to pandas. I think the best way would be to create a standard json or csv output for all modules that return text. Then, it would be simple to take any module output and convert to a nice-looking Pandas table.

Next Steps:

1. Bug: InteractiveMap does not honor use_region=True for vectors.
2. Bug: InteractiveMap for Jupyter does not handle not existing data properly (​GitHub Issue)
3. InteractiveMap: add legend options for rasters, support !simpleCRS for faster rendering, ToolTip integration for vector attributes
4. Continue Pandas integration by adding standard json or csv output to modules that return text
5. ... And many others! I think there's still lots of ways we can improve and expand the integration of GRASS and Jupyter. Ideas welcome.

You can follow next steps for grass.jupyter on our project page on ​GitHub. Thank you again to Vaclav Petras and Anna Petrasova for their guidance and contributions to grass.jupyter. And, another thank you to Vero Andreo, Stefan Blumentrath and Markus Neteler for their feedback and testing!