GSoC 2021 Improved Integration of GRASS GIS and Jupyter Notebooks

Title:	Improved Integration of GRASS GIS and Jupyter Notebooks
Student Name:	Caitlin Haedrich, ​North Carolina State University
Organization:	​OSGeo - Open Source Geospatial Foundation
Mentor Name:	Vaclav Petras, Helena Mitasova, Stephan Blumentrath
GitHub Repository:	​View Repo
GSoC proposal:	​View Proposal

Abstract

The current integration of GRASS GIS and Jupyter Notebooks requires a cumbersome environment variable setup after launching GRASS from within the notebook. It also only allows for simple, non-interactive map displays. My project will address both these issues and improve the GRASS-Jupyter integration with a new package called grass.jupyter.

Goal

My goal is to design a Python/PyGRASS library which will have 3 main components for improving the GRASS-Jupyter integration: (1) creating new initiation functions for the launch of GRASS GIS in Jupyter Notebooks, (2) creating functions for more intuitive map display and, (3) introducing an interactive map display function.

Timeline

Time Period	Milestones
	Tasks	Status
May 17th - June 7th Community Bonding	Introduce myself in dev and SOC mailing lists Get in contact with mentors and discuss project Prepare the wiki page Set up the GitHub repository for project Set up developer environment	Ok Ok Ok Ok Ok
June 7th - June 11th Week 1	Finish Binder Set Up Write grass.jupyter initiation functions	Ok Ok
June 14th - June 18th Week 2	Write non-interactive display functions	Ok
June 21st - June 25 Week 3	Begin writing interactive/folium map functions Edit non-interactive display functions	Ok Ok
June 28 - July 2 Week 4	Continue writing interactive/folium map functions for vectors	Ok
July 5th - July 9th Week 5	Continue writing interactive vector functions Begin writing interactive raster functions	Ok Ok
July 12th - July 16th Week 6: Evaluations	Receive feedback and write evaluation Switch to temporary files and directories Continue working on folium functionality	Ok Ok Ok
July 19th - July 23rd Week 7	Improve documentation and error handling Draft test module for non-interactive rendering Continue raster functions for folium	Ok Ok Ok
July 26th - July 30th Week 8	Finish draft of raster methods with folium Improve test module	Ok Ok
August 2nd - August 6th Week 9	Improve raster handling in folium Add save as HTML option Improve docstrings, readability	Ok Ok Ok
August 9th - August 13th 'Week 10'	Write project summary
August 16th- August 23rd Week 11: Final Evaluation and Code Submission	Submit code and final evaluation.

Bonding period report

With the bonding period wrapped up, here is a brief update on what I've been working on so far.

1. What did I accomplish during the bonding period?

• I introduced myself in dev and SOC mailing lists (​1).
• I had a productive meeting with my mentors, Vashek Petras, Helena Mitasova and Stefan Blumentrath last Friday. We discussed the best development environment (see second to last bullet point) and setting up the main GRASS repository to run in Binder (see last bullet point).
• Prepared the wiki page.
• Set up the GitHub repository for project (​2).
• Set up developer environment. Since development will be easier on Linux, I set up a VirtualBox with Linux (Ubuntu) on my Windows machine. I compiled and installed GRASS and installed Jupyter Notebooks in Ubuntu.
• Opened my first PR (​3) which contains binder dependencies.

2. What do you plan on doing next week?

I've started working on setting up the GRASS repository to run in Binder (​3). This week I plan to continue working on that, set up branches for each of the PRs, and begin writing functions for the grass.jupyter library.

3. Are you blocked on anything?

No, I'm not currently blocked on anything.

Evaluation Status

Weekly reports

Week 1

With Week 1 wrapping up, here is an update on what I worked on this week and where I'm headed next week.

1) What did I get done this week?

• Finished binder setup for GRASS main repo. PR was approved. There's a new folder in the main GRASS repo that contains the necessary files to launch the repository in binder.
  
• Added Binder Button. PR under review. This button will go in the README.md and allow users to launch a Jupyter Lab of the repository in binder. Users can run GRASS (i.e. a compiled version of the main GRASS repo) in Jupyter Notebooks there.
  
• Created an ​example notebook that is linked from the Binder Button. The notebook was copied from ​here and demonstrates the existing Jupyter/GRASS integration.
  
• Wrote draft of GRASS initiation functions for Jupyter (​PR).
  
• Created Makefile and init.py file for grass.jupyter (​PR).
  
• Created a Jupyter Notebook where others can view and test the grass.jupyter functions (​PR).
  

2) What do I plan on doing next week? I plan to start working on display functions and familiarizing myself with Folium.

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Week 2

Here is a brief update from Week 2. You can also check out how the new grass.jupyter module works so far in Binder at ​this link

1) What did I accomplish this week?

• Had a productive discussion with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath.
• Edited and merged Binder button into README ​PR and GRASS session initiation functions for Jupyter ​PR.
• Wrote functions for displaying non-interactive maps in Jupyter and opened PR for discussion ​PR.
• Began researching and drafting interactive display options with folium. In particular, figuring out how to best pass GRASS data/layers to folium.

2) What do I plan on doing next week?

• Edit non-interactive display functions.
• Continue working on passing GRASS data/layers to folium.

3) Am I blocked on anything? No, I'm not blocked but I didn't get much done this week because I had a family event.

Any suggestions and feedback are welcome!

Week 3

Here is a brief update on what I accomplished during Week 3 of Google Summer of Code for the new GRASS library grass.jupyter. I have also started posting links to Binder Jupyter Notebooks in each PR if you would like to try any of the features and functions I've been working on.

1) What did I accomplish this week?

• Edited and discussed functions for displaying non-interactive maps in Jupyter ​PR.
• Drafted an interactive display function for rasters with folium ​PR.
• Made a few minor edits to the example_notebook.ipynb that is linked in the README ​PR.

2) What do I plan on doing next week?

• I have a meeting planned with my mentors.
• I will continue working on interactive functions for Jupyter. In particular, I'm working on passing GRASS rasters to folium.
• I'll begin writing functions for displaying GRASS vectors with folium.

3) Am I blocked on anything? I'm not blocked at the moment. But, I have been working on finding a good way to pass raster data between GRASS and Jupyter and I'm not sure my current method is particularly fast or robust [3]. Luckily, I have a meeting tomorrow with my mentors where we will discuss this.

Any suggestions and feedback are welcome!

Week 4

With week 4 wrapping up, here is a brief update of what I've been working on. Per usual, you can try proposed changes out in the Binder link in the description of each PR.

1) What did I accomplish during Week 1?

• I had a productive meeting with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath. We discussed how environments work and the purpose of copying the environment each time an instance of GrassRenderer is called.
• I finished and merged a class called GrassRenderer for non-interactive map displays ​PR.
• I continued to work on interactive mapping with folium. In particular, I changed how data is passed to folium so that it is first reprojected to WGS84 (required by folium/leaflet) in a new location/mapset ​PR.

2) What do I plan on doing next week?

• Continue to work on interactive functions with folium.

3) Am I blocked on anything?

• No, I'm not blocked on anything at the moment.

Any suggestions and feedback welcome!

Week 5

Here is a weekly update on my project, Improved Integration of GRASS GIS and Jupyter Notebooks.

1) What did I accomplish this week?

• This week, I spent most of my time improving and editing a new class called InteractiveMap to display vectors interactively with folium. You can test the current functionality in ​Binder.
• I had a productive meeting with my mentor, Vaclav Petras on Thursday. We discussed several things that I was a little stuck on and had questions about.
• Updated grass.jupyter demonstration ​Notebook.
• Resolved merge conflicts with master - now, the interactive and non-interactive classes are functional together ​PR.
• Found unexpected behavior in the legend display. Calling folium.map.LayerControl() adds a legend to the map but also seems to prevent further modification of the map.

2) What do I plan to do next week?

• Meet with my mentors tomorrow.
• GSoC Evaluations.
• Finish and merge interactive vector methods.
• Begin interactive maps for rasters.

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 6

Here's a weekly update on my GSoC project, Improved Integration of GRASS GIS and Jupyter Notebooks.

1) What did I accomplish this week?

• This week, I had two meetings with my mentor, Vaclav Petras. We've been discussing how to improve my workflow, best practices and conventions for GRASS and Python and questions that come up as I'm working.
• Instituted the use of temporary PNG files for displaying static/non-interactive maps ​PR and temporary directories for passing vectors between GRASS and folium ​link (PR coming soon).
• Continued to edit the InteractiveMap class and mapping GRASS vectors in folium ​PR
• Created a shortcut for calling GRASS display modules using getattr ​PR
• Completed evaluation and received feedback.

2) What do I plan to do next week?

• I have another meeting planning with Vaclav tomorrow.
• Finish and merge existing PRs
• Continue writing test module for Non-Interactive Vectors
• Support Raster display in folium map
• Clean-up function

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 7

Here's a quick update on what I worked on this past week.

1) What did I accomplish this week?

• This week, I had another two meetings with my mentor, Vaclav Petras. We've been discussing best practices and conventions for GRASS and Python and questions that come up as I'm working.
• Merged 'add_vector' method for InteractiveMaps in folium [2].
• Improved error handling and documentation on the shortcut for calling GRASS display modules for "GrassRenderer" [3].
• Wrote a test module for the "GrassRenderer" class - a class which displays GRASS maps in Jupyter Notebooks as PNG images. I haven't opened a PR since there are a couple things I need to add after some other PRs are merged.
• Continued to work on overlaying rasters in folium - this is taking me a while because the raster and raster bounds have to be reprojected since folium only takes WGS84 and Pseudo-Mercator projections.

2) What do I plan to do next week?

• I have another meeting planning with Vaclav on Monday.
• Finish and merge existing PRs
• Continue writing test module for GrassRenderer

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 8

Here's a quick update on what I worked on this past week. I came down with a stomach bug this week and didn't get as much as I hoped done but I did wrap up some things I'd been working on for a while.

1) What did I accomplish this week?

• I had a brief meeting with my mentor, Vaclav Petras. We discussed some of the existing PR's I had open and goals for the rest of the summer.
• Got rasters to display with folium (finally)! You can check out the functionality in Binder ​here and see the PR ​here. It takes a while to reproject the raster (folium only excepts rasters in Pseudo-mercator). As Stefan recommended, I'm hoping to write a "simple CRS" option that won't require reproject and will be faster. But that's for next week...
• Merged shortcuts and temporary files for GrassRenderer (​PR and ​PR).
• Continued working on test script for GrassRenderer, the non-interactive display class ​PR.

2) What do I plan to do next week?

• I have a meeting with my mentors planned for tomorrow
• Add simple CRS option for raster display in folium
• Edit and finish exiting PRs
• Add option to save/export folium map as HTML

3) Am I blocked on anything?

• No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 9

Happy last week of coding! Here's an update of my GSoC project, "Improved Integration of GRASS GIS and Jupyter Notebooks."

) What did I accomplish this week?

• I had a meeting with my mentors, Vaclav Petras and Helena Mitasova. We discussed some of the existing PR's and goals for the rest of the summer.
• Rasters reproject and display much faster now! You can test it out in ​Binder. As Vaclav pointed out, it was oversampling the raster when it reprojected - adding a method to estimate the resolution in target location fixed this.
• Re-arranged code for InteractiveMap and improved documentation ​PR
• Added save as HTML method for InteractiveMap ​PR

2) What do I plan to do next week?

• I have a meeting with my mentors planned for today
• Add simple CRS option for raster display in folium
• Finish test script ​PR
• Improve/finish GRASS-Jupyter Demonstration notebook

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome!

Week 10

Last week of coding! Here's an update on how the project is progressing

) What did I accomplish this week?

• I had a meeting with my mentors, Vaclav Petras and Stephan Blumentrath. We discussed goals for the rest of the summer and decided to leave the simple CRS for future work.
• Merged raster support in folium to main! It was a pretty substantial PR in the end ​PR
• Began improving example notebooks and adding additional tutorials.

2) What do I plan to do next week?

• Finalize docstrings
• Finish example notebooks
• Evaluations and final submission

3) Am I blocked on anything? No, I'm not currently blocked on anything.

Feedback, comments, questions and ideas welcome - especially for Future Work!

Future Work