= GSoC 2021 Improved Integration of GRASS GIS and Jupyter Notebooks= || Title: || '''Improved Integration of GRASS GIS and Jupyter Notebooks'' || || Student Name: || Caitlin Haedrich, [https://cnr.ncsu.edu/geospatial/ North Carolina State University] || || Organization: || [http://www.osgeo.org OSGeo - Open Source Geospatial Foundation] || || Mentor Name: || Vaclav Petras, Helena Mitasova, Stephan Blumentrath || || !GitHub Repository: || [https://github.com/chaedri/grass/ View Repo] || || GSoC proposal: || [https://docs.google.com/document/d/1ZT0cZobd87YCb3Ogis7RzWPj02XZkCpAHbC3VBGh7gc/edit?usp=sharing View Proposal] || == Abstract == The current integration of GRASS GIS and Jupyter Notebooks requires a cumbersome environment variable setup after launching GRASS from within the notebook. It also only allows for simple, non-interactive map displays. My project will address both these issues and improve the GRASS-Jupyter integration with a new package called grass.jupyter. == Goal == My goal is to design a Python/PyGRASS library which will have 3 main components for improving the GRASS-Jupyter integration: (1) creating new initiation functions for the launch of GRASS GIS in Jupyter Notebooks, (2) creating functions for more intuitive map display and, (3) introducing an interactive map display function. == Timeline == {{{#!th style="background: #ddd" rowspan=2 '''Time Period''' }}} {{{#!th style="background: #ddd" colspan=2 '''Milestones''' }}} |----------------------- {{{#!th style="background: #ddd" Tasks }}} {{{#!th style="background: #ddd" Status }}} |----------------------- {{{#!td May 17th - June 7th\\ ''Community Bonding'' }}} {{{#!td - Introduce myself in dev and SOC mailing lists - Get in contact with mentors and discuss project - Prepare the wiki page - Set up the !GitHub repository for project - Set up developer environment }}} {{{#!td Ok \\ Ok \\ Ok \\ Ok \\ Ok }}} |----------------------- {{{#!td June 7th - June 11th \\ ''Week 1'' }}} {{{#!td - Finish Binder Set Up - Write grass.jupyter initiation functions }}} {{{#!td Ok\\ Ok }}} |----------------------- {{{#!td June 14th - June 18th \\ ''Week 2'' }}} {{{#!td - Write non-interactive display functions }}} {{{#!td Ok }}} |----------------------- {{{#!td June 21st - June 25 \\ ''Week 3'' }}} {{{#!td - Begin writing interactive/folium map functions - Edit non-interactive display functions }}} {{{#!td Ok\\ Ok }}} |----------------------- {{{#!td June 28 - July 2 \\ ''Week 4'' }}} {{{#!td - Continue writing interactive/folium map functions for vectors }}} {{{#!td Ok }}} |----------------------- {{{#!td July 5th - July 9th \\ ''Week 5'' }}} {{{#!td - Continue writing interactive vector functions - Begin writing interactive raster functions }}} {{{#!td Ok\\ Ok }}} |----------------------- {{{#!td style="background: #ddd" July 12th - July 16th \\ ''Week 6: Evaluations'' }}} {{{#!td style="background: #ddd" - Receive feedback and write evaluation - Switch to temporary files and directories - Continue working on folium functionality }}} {{{#!td Ok\\ Ok\\ Ok }}} |----------------------- {{{#!td July 19th - July 23rd \\ ''Week 7'' }}} {{{#!td - Improve documentation and error handling - Draft test module for non-interactive rendering - Continue raster functions for folium }}} {{{#!td Ok\\ Ok\\ Ok }}} |----------------------- {{{#!td July 26th - July 30th \\ ''Week 8'' }}} {{{#!td - Finish draft of raster methods with folium - Improve test module }}} {{{#!td Ok \\ Ok }}} |----------------------- {{{#!td August 2nd - August 6th \\ ''Week 9'' }}} {{{#!td - Improve raster handling in folium - Add save as HTML option - Improve docstrings, readability }}} {{{#!td Ok \\ Ok \\ Ok }}} |----------------------- {{{#!td August 9th - August 13th \\ 'Week 10' }}} {{{#!td - Finish InteractiveMap - Example Notebooks }}} {{{#!td Ok \\ Ok }}} |----------------------- {{{#!td style="background: #ddd" August 16th- August 23rd \\ ''Week 11: Final Evaluation and Code Submission'' }}} {{{#!td style="background: #ddd" - Submit code and final evaluation. - Example Notebooks - Final docstring improvements }}} {{{#!td }}} == Bonding period report == With the bonding period wrapped up, here is a brief update on what I've been working on so far. 1. What did I accomplish during the bonding period? - I introduced myself in dev and SOC mailing lists ([https://lists.osgeo.org/pipermail/grass-dev/2021-May/095165.html 1]). - I had a productive meeting with my mentors, Vashek Petras, Helena Mitasova and Stefan Blumentrath last Friday. We discussed the best development environment (see second to last bullet point) and setting up the main GRASS repository to run in Binder (see last bullet point). - Prepared the wiki page. - Set up the !GitHub repository for project ([https://github.com/chaedri/grass 2]). - Set up developer environment. Since development will be easier on Linux, I set up a !VirtualBox with Linux (Ubuntu) on my Windows machine. I compiled and installed GRASS and installed Jupyter Notebooks in Ubuntu. - Opened my first PR ([https://github.com/OSGeo/grass/pull/1603 3]) which contains binder dependencies. 2. What do you plan on doing next week? I've started working on setting up the GRASS repository to run in Binder ([https://github.com/OSGeo/grass/pull/1603 3]). This week I plan to continue working on that, set up branches for each of the PRs, and begin writing functions for the grass.jupyter library. 3. Are you blocked on anything? No, I'm not currently blocked on anything. == Evaluation Status == == Weekly reports == == Week 1 == With Week 1 wrapping up, here is an update on what I worked on this week and where I'm headed next week. '''1) What did I get done this week?''' * Finished binder setup for GRASS main repo. PR was approved. There's a new folder in the main GRASS repo that contains the necessary files to launch the repository in binder.\\ * Added Binder Button. PR under review. This button will go in the README.md and allow users to launch a Jupyter Lab of the repository in binder. Users can run GRASS (i.e. a compiled version of the main GRASS repo) in Jupyter Notebooks there.\\ * Created an [https://github.com/OSGeo/grass/pull/1628 example notebook] that is linked from the Binder Button. The notebook was copied from [https://github.com/wenzeslaus/try-grass-in-jupyter here] and demonstrates the existing Jupyter/GRASS integration.\\ * Wrote draft of GRASS initiation functions for Jupyter ([https://github.com/OSGeo/grass/pull/1629 PR]).\\ * Created Makefile and __init__.py file for grass.jupyter ([https://github.com/OSGeo/grass/pull/1629 PR]).\\ * Created a Jupyter Notebook where others can view and test the grass.jupyter functions ([https://github.com/OSGeo/grass/pull/1629 PR]).\\ '''2) What do I plan on doing next week?''' I plan to start working on display functions and familiarizing myself with Folium.\\ '''3) Am I blocked on anything?''' No, I'm not currently blocked on anything. == Week 2 == Here is a brief update from Week 2. You can also check out how the new grass.jupyter module works so far in Binder at [https://mybinder.org/v2/gh/chaedri/grass/non-interactive-display?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fjupyter_integration.ipynb this link] '''1) What did I accomplish this week?''' * Had a productive discussion with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath. * Edited and merged Binder button into README [https://github.com/OSGeo/grass/pull/1628 PR] and GRASS session initiation functions for Jupyter [https://github.com/OSGeo/grass/pull/1629 PR]. * Wrote functions for displaying non-interactive maps in Jupyter and opened PR for discussion [https://github.com/OSGeo/grass/pull/1668 PR]. * Began researching and drafting interactive display options with folium. In particular, figuring out how to best pass GRASS data/layers to folium. '''2) What do I plan on doing next week?''' * Edit non-interactive display functions. * Continue working on passing GRASS data/layers to folium. '''3) Am I blocked on anything?''' No, I'm not blocked but I didn't get much done this week because I had a family event. Any suggestions and feedback are welcome! == Week 3 == Here is a brief update on what I accomplished during Week 3 of Google Summer of Code for the new GRASS library grass.jupyter. I have also started posting links to Binder Jupyter Notebooks in each PR if you would like to try any of the features and functions I've been working on. '''1) What did I accomplish this week?''' * Edited and discussed functions for displaying non-interactive maps in Jupyter [https://github.com/OSGeo/grass/pull/1668 PR]. * Drafted an interactive display function for rasters with folium [https://github.com/OSGeo/grass/pull/1684 PR]. * Made a few minor edits to the example_notebook.ipynb that is linked in the README [https://github.com/OSGeo/grass/pull/1686 PR]. '''2) What do I plan on doing next week?''' * I have a meeting planned with my mentors. * I will continue working on interactive functions for Jupyter. In particular, I'm working on passing GRASS rasters to folium. * I'll begin writing functions for displaying GRASS vectors with folium. '''3) Am I blocked on anything?''' I'm not blocked at the moment. But, I have been working on finding a good way to pass raster data between GRASS and Jupyter and I'm not sure my current method is particularly fast or robust [3]. Luckily, I have a meeting tomorrow with my mentors where we will discuss this. Any suggestions and feedback are welcome! == Week 4 == With week 4 wrapping up, here is a brief update of what I've been working on. Per usual, you can try proposed changes out in the Binder link in the description of each PR. '''1) What did I accomplish during Week 1?''' * I had a productive meeting with my mentors: Vaclav Petras, Helena Mitasova and Stefan Blumentrath. We discussed how environments work and the purpose of copying the environment each time an instance of GrassRenderer is called. * I finished and merged a class called GrassRenderer for non-interactive map displays [https://github.com/OSGeo/grass/pull/1668 PR]. * I continued to work on interactive mapping with folium. In particular, I changed how data is passed to folium so that it is first reprojected to WGS84 (required by folium/leaflet) in a new location/mapset [https://github.com/OSGeo/grass/pull/1710 PR]. '''2) What do I plan on doing next week?''' * Continue to work on interactive functions with folium. '''3) Am I blocked on anything?''' * No, I'm not blocked on anything at the moment. Any suggestions and feedback welcome! == Week 5 == Here is a weekly update on my project, Improved Integration of GRASS GIS and Jupyter Notebooks. '''1) What did I accomplish this week?''' * This week, I spent most of my time improving and editing a new class called `InteractiveMap` to display vectors interactively with folium. You can test the current functionality in [https://mybinder.org/v2/gh/chaedri/grass/interactive-vectors?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fjupyter_integration.ipynb Binder]. * I had a productive meeting with my mentor, Vaclav Petras on Thursday. We discussed several things that I was a little stuck on and had questions about. * Updated `grass.jupyter` demonstration [https://mybinder.org/v2/gh/chaedri/grass/interactive-vectors?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fjupyter_integration.ipynb Notebook]. * Resolved merge conflicts with master - now, the interactive and non-interactive classes are functional together [https://github.com/OSGeo/grass/pull/1710 PR]. * Found unexpected behavior in the legend display. Calling folium.map.LayerControl() adds a legend to the map but also seems to prevent further modification of the map. '''2) What do I plan to do next week?''' * Meet with my mentors tomorrow. * GSoC Evaluations. * Finish and merge interactive vector methods. * Begin interactive maps for rasters. '''3) Am I blocked on anything?''' * No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome! == Week 6 == Here's a weekly update on my GSoC project, Improved Integration of GRASS GIS and Jupyter Notebooks. '''1) What did I accomplish this week?''' * This week, I had two meetings with my mentor, Vaclav Petras. We've been discussing how to improve my workflow, best practices and conventions for GRASS and Python and questions that come up as I'm working. * Instituted the use of temporary PNG files for displaying static/non-interactive maps [https://github.com/OSGeo/grass/pull/1727 PR] and temporary directories for passing vectors between GRASS and folium [https://github.com/chaedri/grass/blob/InteractiveMaps-add-temp-files/python/grass/jupyter/interact_display.py link (PR coming soon)]. * Continued to edit the InteractiveMap class and mapping GRASS vectors in folium [https://github.com/OSGeo/grass/pull/1710 PR] * Created a shortcut for calling GRASS display modules using __getattr__ [https://github.com/OSGeo/grass/pull/1723 PR] * Completed evaluation and received feedback. '''2) What do I plan to do next week?''' * I have another meeting planning with Vaclav tomorrow. * Finish and merge existing PRs * Continue writing test module for Non-Interactive Vectors * Support Raster display in folium map * Clean-up function '''3) Am I blocked on anything?''' * No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome! == Week 7 == Here's a quick update on what I worked on this past week. '''1) What did I accomplish this week?''' * This week, I had another two meetings with my mentor, Vaclav Petras. We've been discussing best practices and conventions for GRASS and Python and questions that come up as I'm working. * Merged 'add_vector' method for InteractiveMaps in folium [2]. * Improved error handling and documentation on the shortcut for calling GRASS display modules for "GrassRenderer" [3]. * Wrote a test module for the "GrassRenderer" class - a class which displays GRASS maps in Jupyter Notebooks as PNG images. I haven't opened a PR since there are a couple things I need to add after some other PRs are merged. * Continued to work on overlaying rasters in folium - this is taking me a while because the raster and raster bounds have to be reprojected since folium only takes WGS84 and Pseudo-Mercator projections. '''2) What do I plan to do next week?''' * I have another meeting planning with Vaclav on Monday. * Finish and merge existing PRs * Continue writing test module for GrassRenderer '''3) Am I blocked on anything?''' * No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome! == Week 8 == Here's a quick update on what I worked on this past week. I came down with a stomach bug this week and didn't get as much as I hoped done but I did wrap up some things I'd been working on for a while. '''1) What did I accomplish this week?''' * I had a brief meeting with my mentor, Vaclav Petras. We discussed some of the existing PR's I had open and goals for the rest of the summer. * Got rasters to display with folium (finally)! You can check out the functionality in Binder [https://mybinder.org/v2/gh/chaedri/grass/interactive-rasters?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fjupyter_integration.ipynb here] and see the PR [https://github.com/OSGeo/grass/pull/1769 here]. It takes a while to reproject the raster (folium only excepts rasters in Pseudo-mercator). As Stefan recommended, I'm hoping to write a "simple CRS" option that won't require reproject and will be faster. But that's for next week... * Merged shortcuts and temporary files for `GrassRenderer` ([https://github.com/OSGeo/grass/pull/1723 PR] and [https://github.com/OSGeo/grass/pull/1727 PR]). * Continued working on test script for `GrassRenderer`, the non-interactive display class [https://github.com/OSGeo/grass/pull/1739 PR]. '''2) What do I plan to do next week?''' * I have a meeting with my mentors planned for tomorrow * Add simple CRS option for raster display in folium * Edit and finish exiting PRs * Add option to save/export folium map as HTML '''3) Am I blocked on anything?''' * No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome! == Week 9 == Happy last week of coding! Here's an update of my GSoC project, "Improved Integration of GRASS GIS and Jupyter Notebooks." '''1) What did I accomplish this week?''' * I had a meeting with my mentors, Vaclav Petras and Helena Mitasova. We discussed some of the existing PR's and goals for the rest of the summer. * Rasters reproject and display much faster now! You can test it out in [https://mybinder.org/v2/gh/chaedri/grass/interactive-rasters?urlpath=lab%2Ftree%2Fdoc%2Fnotebooks%2Fjupyter_integration.ipynb Binder]. As Vaclav pointed out, it was oversampling the raster when it reprojected - adding a method to estimate the resolution in target location fixed this. * Re-arranged code for InteractiveMap and improved documentation [https://github.com/OSGeo/grass/pull/1769 PR] * Added save as HTML method for InteractiveMap [https://github.com/OSGeo/grass/pull/1769 PR] '''2) What do I plan to do next week?''' * I have a meeting with my mentors planned for today * Add simple CRS option for raster display in folium * Finish test script [https://github.com/OSGeo/grass/pull/1739 PR] * Improve/finish GRASS-Jupyter Demonstration notebook '''3) Am I blocked on anything?''' No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome! == Week 10 == Last week of coding! Here's an update on how the project is progressing '''1) What did I accomplish this week?''' * I had a meeting with my mentors, Vaclav Petras and Stephan Blumentrath. We discussed goals for the rest of the summer and decided to leave the simple CRS for future work. * Merged raster support in folium to main! It was a pretty substantial PR in the end [https://github.com/OSGeo/grass/pull/1769 PR] * Began improving example notebooks and adding additional tutorials. '''2) What do I plan to do next week?''' * Finalize docstrings * Finish example notebooks * Evaluations and final submission '''3) Am I blocked on anything?''' No, I'm not currently blocked on anything. Feedback, comments, questions and ideas welcome - especially for Future Work! == Future Work ==