= GSoC 2018 GRASS GIS module for Sentinel-2 cloud and shadow detection = || Title: || '''GRASS GIS module for Sentinel-2 cloud and shadow detection''' || ||Student Name: || Roberta Fagandini, Ph.D. student at [https://www.polimi.it/en/ Politecnico of Milano] || ||Organization: || [http://www.osgeo.org OSGeo - Open Source Geospatial Foundation] || || Mentor Name: || [https://wiki.osgeo.org/wiki/User:Robertomarzocchi Roberto Marzocchi], [https://wiki.osgeo.org/wiki/User:Mlennert Moritz Lennert] || || GSoC proposal: || [https://docs.google.com/document/d/1mQyouqbHGHugn5DjEDZI3hc1eU_bT_b0qLJPPwkt_hk/edit?usp=sharing GRASS GIS module for Sentinel-2 cloud and shadow detection] || || Repositories: || Github for development: https://github.com/RobiFag/GRASS_clouds_and_shadows || {{{#!comment|| || GRASS SVN !AddOns for final modules: || }}} == Abstract == Unlike Landsat images, Sentinel-2 datasets do not include thermal and Quality Assessment bands that simplify the detection of clouds avoiding erroneous classification. Moreover, also clouds shadows on the ground lead to anomalous reflectance values which have to be taken into account during the image processing. To date, only a specific module for Landsat automatic cloud coverage assessment is available within GRASS GIS [https://grass.osgeo.org/grass74/manuals/i.landsat.acca.html (i.landsat.acca)] while regarding shadows, no specific module is available. Therefore to date, the detection of clouds and shadows has to be manually performed for Sentinel-2 images. == Workflow == The workflow of the algorithm is described [https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing HERE] == Goal == The aim is the coding of a specific module for GRASS GIS application which implements an automatic procedure for clouds and shadow detection within Sentinel 2 images. The module has to provide a tool which can be easily used by inexpert users, taking advantage of the suggested parameters, or by more expert users that can modify default values according to their needs. == Timeline == {{{#!th '''Time Period''' }}} {{{#!th '''Tasks''' }}} {{{#!th '''Deliverables''' }}} {{{#!th '''Status''' }}} |----------------------- {{{#!td Community Bonding }}} {{{#!td - Introduce myself in soc and dev lists - Get in contact with my mentors - Discuss and share ideas, suggestions, etc. with my mentors and the dev list - Prepare the wiki page about the project and keep it up to date - Set up the github repository of the project (https://github.com/RobiFag/GRASS_clouds_and_shadows) - Get familiar with development environment - Follow the procedure for access to the GRASS-Addons-SVN repository - Familiarize myself with GRASS Python scripting library and the writing of GRASS module - Elaborate a schema of the algorithm share and discuss it with the community list }}} {{{#!td style="text-align: center" Full Report [https://docs.google.com/document/d/177amKStJMCvbQMU8TneoR89So4AzlWGGH5co3s0dyWA/edit?usp=sharing HERE] }}} {{{#!td style="text-align: center" X }}} |----------------------- {{{#!td MAY 14 - 20 }}} {{{#!td - Make any changes to the algorithm deriving from the discussions and feedback and define the final version of the algorithm - Test the final version of the algorithm with several datasets - Start translating the final algorithm about clouds and shadows detection in a GRASS Python script }}} {{{#!td }}} {{{#!td style="text-align: center" X }}} |----------------------- {{{#!td MAY 21 - 27 }}} {{{#!td - Translate the final algorithm about clouds and shadows detection in a GRASS Python script - Improve the GRASS python script (manage temporary file, clean up the code, etc.) - Test and validate the script and fix bugs - Share the script with the community }}} {{{#!td }}} {{{#!td style="text-align: center" X }}} |----------------------- {{{#!td MAY 28 - JUNE 3 }}} {{{#!td - Implement any change from discussions and feedback - Start implementing the GUI - Test and validate the script and fix bugs - Share the script with the community }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JUNE 4 - JUNE 10 }}} {{{#!td - Implement any change from discussions and feedback - Finish the implementation of the GUI - Test and validate the script with several datasets and fix bugs - Share the script with the community }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JUNE 11 - JUNE 15\\ '''First evaluation''' }}} {{{#!td - Final tests - Check the script with mentors - Share the script with the community - Discuss with mentors about the next coding steps - Prepare deliverable for the evaluation }}} {{{#!td '''Working GRASS Python Script with GUI''' }}} {{{#!td }}} |----------------------- {{{#!td JUNE 16 - JUNE 24 }}} {{{#!td - Implement any change from discussions and feedback - Start integrating the atmospheric correction (i.atcorr) within the clouds and shadows detection algorithm - Define an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr - Share and discuss it with the community }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JUNE 25 - JULY 1 }}} {{{#!td - Implement any change from discussions and feedback - Translate the automatic procedure for the retrieving of i.atcorr parameters in a GRASS Python script - Test and fix bugs }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JULY 2 - JULY 8 }}} {{{#!td - Finish the implementation of i.atcorr with a working automated procedure for retrieving requested input parameters - Set up the GUI for this part of the script - Share the script with the community }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JULY 9 - JULY 13\\ '''Second evaluation''' }}} {{{#!td - Implement any change from discussions and feedback - Check the module with mentors - Share the module with the community - Prepare deliverable for the evaluation }}} {{{#!td '''Complete GRASS module prototype''' }}} {{{#!td }}} |----------------------- {{{#!td JULY 14 - JULY 22 }}} {{{#!td - Implement any change from discussions and feedback - Start compiling the user documentation (GRASS GIS hel page) - Evaluate other existing algorithm of clouds and shadows detection to be implemented in the module }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JULY 23 - JULY 29 }}} {{{#!td - Depending on the availability of other algorithms, evaluate their integration within the module - Finish compiling the user documentation (GRASS GIS hel page) - Share the module with the community }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JULY 30 - AUGUST 5 }}} {{{#!td - Depending on previous evaluations, integration of at least another algorithm - Improvement of the user documentation (following feedback from community) }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td JULY 30 - AUGUST 5 }}} {{{#!td - Test and fix bugs - Share the module with the community - Implement any change from discussions and feedback }}} {{{#!td }}} {{{#!td }}} |----------------------- {{{#!td AUGUST 9 - AUGUST 13\\ '''Final evaluation''' }}} {{{#!td - Test and fix bugs - Publication of the final module prototype as a GRASS add-on (svn official repository) - Prepare deliverable for the final evaluation }}} {{{#!td '''Final GRASS add-on module''' }}} {{{#!td }}} |----------------------- == Reports == === Bonding Period === '''Introduce myself in soc and dev lists'''\\ After being accepted as a student for GSoC 2018, I introduced my self in SOC and grass-dev lists on 24th of April (0). '''Get in contact with my mentors'''\\ At the same time, I got in contact with my mentors, Moritz Lennert and Roberto Marzocchi. I corresponded with them by email more or less every day from the beginning to the end of the bonding period describing all the activities that I was carrying out (0)(1). I asked them suggestions about the best tool to set-up my dev environment, how to do that, etc. I also better detailed my proposal explaining to them the starting point of the project, the procedure, the algorithm and I asked them for hints and feedback. I shared immediately with them and the grass-dev list all the necessary information and all the new features like links to the wiki page, !GitHub repository, schemas and description of the project, etc. '''Prepare the wiki page about the project and keep it up to date'''\\ I requested my OSGeo user ID on 26th of April and, once received, I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3). The wiki page includes: General information about the project (title, mentors, links to the proposal and !GitHub repository, etc.); A brief description of the project; Link to general and detailed schemas of the procedure to better describe the whole workflow; Goal; Timeline of tasks and deliverables. After discussing with my mentors, I have better detailed the timeline adding specific tasks for each week and the deliverables for the evaluations. I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4). '''Set up the !GitHub repository of the project'''\\ I chose !GitHub as a public repository for the development of the GRASS GIS module for the automatic detection of cloud and shadow in Sentinel 2 images, under the guidance of my mentors. I studied the !GitHub documentation (5) and followed tutorials (6) in order to understand how it works and how to use it. After that, I set up my !GitHub repository (7) and I started committing some test files. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors and the dev community. My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS. '''Follow the procedure for access to the GRASS-Addons-SVN repository'''\\ I requested the write access to the GRASS-Addons-SVN repository following the official procedure and now I have been added to the grass-addons contributors' list. This step was necessary to submit the code as a GRASS GIS addon at the end of the GSoC 2018. '''Developer environment'''\\ I checked and updated my developer environment and installed all the necessary libraries and dependencies so as to be ready to start coding after the bonding period. '''Familiarize myself with GRASS Python scripting library and the writing of GRASS module'''\\ During the bonding period, I took lots of time to get familiar with GRASS Python scripting library and the writing of a GRASS module. First of all, I started from the community practices about how to contribute to GRASS GIS (8). Then I studied technical issues like function definition following the official documentation of GRASS Python scripting library (9). I also investigated the existing scripts (10)(11) to better understand how they work and how to structure addon code. During this phase, I made some coding tests that I shared with my mentors and the dev community through my repository. '''Elaborate a schema of the algorithm'''\\ In order to better explain and define all the steps and procedures of the algorithm to be implemented as a GRASS addon, I elaborate a general workflow schema and specific diagrams for each procedure (cloud detection, shadow detection, etc.) (12). For the proper and complete consultation of the schemas, please use the google drive app diagram.io. '''Share and discuss it with the community list'''\\ Obviously, I shared everything with my mentors and the dev community. I started a new thread on the list adding all the necessary and useful links (wiki, repository, schema and full description of the project) and asking explicitly for feedback and hints (13). (0) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088254.html \\ (1) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088314.html \\ (2) https://trac.osgeo.org/grass/wiki/GSoC/2018/CloudsAndShadowsDetection \\ (3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted \\ (4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students \\ (5) https://guides.github.com/ \\ (6) https://www.youtube.com/githubguides \\ (7) https://github.com/RobiFag/GRASS_clouds_and_shadows \\ (8) https://trac.osgeo.org/grass/wiki/HowToContribute \\ (9) https://grass.osgeo.org/grass75/manuals/libpython/script_intro.html \\ (10) https://trac.osgeo.org/grass/browser/grass/trunk/scripts \\ (11) https://trac.osgeo.org/grass/browser/grass-addons/grass7 \\ (12) https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing \\ (13) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088369.html \\ === Week 01 === Since my progress has been a bit faster than expected I have already discussed with my mentors about further additions to the project. Therefore I have already updated my wiki page changing the timeline. In this way, I will be able to release a more complete module. '''1) What did I complete this week?''' * Checked the prototype version of the algorithm for cloud and shadow detection\\ * Tested it \\ * I better studied the documentation of the GRASS Python Scripting Library and started coding the GRASS python script following the workflow schema prepared during the bonding period\\ * Coded the 'data preparation' phase\\ * Coded the clouds detection procedure\\ * Added this first part of python script to my !GitHub repository\\ * Discussed with my mentors about further additions\\ * Discussed with the community about testing phase. I received some feedback and some developers stepped forward to help me in testing the algorithm \\ * Shared progress with the community\\ Unfortunately, I wasn't able to code on Thursday because of a PhD meeting with my supervisor. '''2) What am I going to achieve for next week?''' * Finish the translation of the algorithm in a basic python script adding the shadows detection procedure and the steps for checking and cleaning the final outputs\\ * Improve the python script so as to have a "light" working python script (manage temporary file, clean up the code, etc.)\\ * Test it and fix bugs\\ * Start implementing the GUI\\ '''3) Is there any blocking issue?'''\\ No at the moment. === Week 02 === '''1) What did I complete this week?''''\\ * Added a basic version of the shadow detection procedure * Implemented some changes from dev feedback * Cleaned up the cloud detection code reducing the r.mapcalc calls and other computations in order to improve performance * Cleaned up the shadow detection code reducing the r.mapcalc calls and other computations in order to improve performance * Integrated the cloud and shadow intersection part * Managed temporary file * Tested the python script and fixed bugs * Frequently added the python script to my !GitHub repository * Shared progress with the community '''2) What am I going to achieve for next week?''' * Implement any change from discussions and feedback * Test it and fix bugs * Start implementing the GUI '''3) Is there any blocking issue?'''\\ No at the moment.