| Version 22 (modified by , 6 years ago) ( diff ) |
|---|
GSoC 2018 GRASS GIS module for Sentinel-2 cloud and shadow detection
| Title: | GRASS GIS module for Sentinel-2 cloud and shadow detection |
| Student Name: | Roberta Fagandini, Ph.D. student at Politecnico of Milano |
| Organization: | OSGeo - Open Source Geospatial Foundation |
| Mentor Name: | Roberto Marzocchi, Moritz Lennert |
| GSoC proposal: | GRASS GIS module for Sentinel-2 cloud and shadow detection |
| Repositories: | Github for development: https://github.com/RobiFag/GRASS_clouds_and_shadows |
Abstract
Unlike Landsat images, Sentinel-2 datasets do not include thermal and Quality Assessment bands that simplify the detection of clouds avoiding erroneous classification. Moreover, also clouds shadows on the ground lead to anomalous reflectance values which have to be taken into account during the image processing. To date, only a specific module for Landsat automatic cloud coverage assessment is available within GRASS GIS (i.landsat.acca) while regarding shadows, no specific module is available. Therefore to date, the detection of clouds and shadows has to be manually performed for Sentinel-2 images.
Workflow
The workflow of the algorithm is described HERE
Goal
The aim is the coding of a specific module for GRASS GIS application which implements an automatic procedure for clouds and shadow detection within Sentinel 2 images. The module has to provide a tool which can be easily used by inexpert users, taking advantage of the suggested parameters, or by more expert users that can modify default values according to their needs.
Timeline
Time Period | Tasks | Deliverables | Status |
|---|---|---|---|
Community Bonding |
| Full Report HERE | X |
MAY 14 - 20 |
| X | |
MAY 21 - 27 |
| X | |
MAY 28 - JUNE 3 |
| X | |
JUNE 4 - JUNE 10 |
| X | |
JUNE 11 - JUNE 15 |
| Working GRASS Python Script with GUI | X |
JUNE 16 - JUNE 24 |
| X | |
JUNE 25 - JULY 1 |
| ||
JULY 2 - JULY 8 |
| ||
JULY 9 - JULY 13 |
| Complete GRASS module prototype | |
JULY 14 - JULY 22 |
| ||
JULY 23 - JULY 29 |
| ||
JULY 30 - AUGUST 5 |
| ||
JULY 30 - AUGUST 5 |
| ||
AUGUST 9 - AUGUST 13 |
| Final GRASS add-on module | |
Reports
Bonding Period
Introduce myself in soc and dev lists
After being accepted as a student for GSoC 2018, I introduced my self in SOC and grass-dev lists on 24th of April (0).
Get in contact with my mentors
At the same time, I got in contact with my mentors, Moritz Lennert and Roberto Marzocchi.
I corresponded with them by email more or less every day from the beginning to the end of the bonding period describing all the activities that I was carrying out (0)(1). I asked them suggestions about the best tool to set-up my dev environment, how to do that, etc.
I also better detailed my proposal explaining to them the starting point of the project, the procedure, the algorithm and I asked them for hints and feedback.
I shared immediately with them and the grass-dev list all the necessary information and all the new features like links to the wiki page, GitHub repository, schemas and description of the project, etc.
Prepare the wiki page about the project and keep it up to date
I requested my OSGeo user ID on 26th of April and, once received, I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3).
The wiki page includes:
General information about the project (title, mentors, links to the proposal and GitHub repository, etc.);
A brief description of the project;
Link to general and detailed schemas of the procedure to better describe the whole workflow;
Goal;
Timeline of tasks and deliverables.
After discussing with my mentors, I have better detailed the timeline adding specific tasks for each week and the deliverables for the evaluations.
I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4).
Set up the GitHub repository of the project
I chose GitHub as a public repository for the development of the GRASS GIS module for the automatic detection of cloud and shadow in Sentinel 2 images, under the guidance of my mentors. I studied the GitHub documentation (5) and followed tutorials (6) in order to understand how it works and how to use it. After that, I set up my GitHub repository (7) and I started committing some test files. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors and the dev community.
My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS.
Follow the procedure for access to the GRASS-Addons-SVN repository
I requested the write access to the GRASS-Addons-SVN repository following the official procedure and now I have been added to the grass-addons contributors' list. This step was necessary to submit the code as a GRASS GIS addon at the end of the GSoC 2018.
Developer environment
I checked and updated my developer environment and installed all the necessary libraries and dependencies so as to be ready to start coding after the bonding period.
Familiarize myself with GRASS Python scripting library and the writing of GRASS module
During the bonding period, I took lots of time to get familiar with GRASS Python scripting library and the writing of a GRASS module. First of all, I started from the community practices about how to contribute to GRASS GIS (8). Then I studied technical issues like function definition following the official documentation of GRASS Python scripting library (9). I also investigated the existing scripts (10)(11) to better understand how they work and how to structure addon code.
During this phase, I made some coding tests that I shared with my mentors and the dev community through my repository.
Elaborate a schema of the algorithm
In order to better explain and define all the steps and procedures of the algorithm to be implemented as a GRASS addon, I elaborate a general workflow schema and specific diagrams for each procedure (cloud detection, shadow detection, etc.) (12). For the proper and complete consultation of the schemas, please use the google drive app diagram.io.
Share and discuss it with the community list
Obviously, I shared everything with my mentors and the dev community. I started a new thread on the list adding all the necessary and useful links (wiki, repository, schema and full description of the project) and asking explicitly for feedback and hints (13).
(0) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088254.html
(1) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088314.html
(2) https://trac.osgeo.org/grass/wiki/GSoC/2018/CloudsAndShadowsDetection
(3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted
(4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students
(5) https://guides.github.com/
(6) https://www.youtube.com/githubguides
(7) https://github.com/RobiFag/GRASS_clouds_and_shadows
(8) https://trac.osgeo.org/grass/wiki/HowToContribute
(9) https://grass.osgeo.org/grass75/manuals/libpython/script_intro.html
(10) https://trac.osgeo.org/grass/browser/grass/trunk/scripts
(11) https://trac.osgeo.org/grass/browser/grass-addons/grass7
(12) https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing
(13) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088369.html
Week 01
Since my progress has been a bit faster than expected I have already discussed with my mentors about further additions to the project. Therefore I have already updated my wiki page changing the timeline. In this way, I will be able to release a more complete module.
1) What did I complete this week?
- Checked the prototype version of the algorithm for cloud and shadow detection
- Tested it
- I better studied the documentation of the GRASS Python Scripting Library and started coding the GRASS python script following the workflow schema prepared during the bonding period
- Coded the 'data preparation' phase
- Coded the clouds detection procedure
- Added this first part of python script to my GitHub repository
- Discussed with my mentors about further additions
- Discussed with the community about testing phase. I received some feedback and some developers stepped forward to help me in testing the algorithm
- Shared progress with the community
Unfortunately, I wasn't able to code on Thursday because of a PhD meeting with my supervisor.
2) What am I going to achieve for next week?
- Finish the translation of the algorithm in a basic python script adding the shadows detection procedure and the steps for checking and cleaning the final outputs
- Improve the python script so as to have a "light" working python script (manage temporary file, clean up the code, etc.)
- Test it and fix bugs
- Start implementing the GUI
3) Is there any blocking issue?
No at the moment.
Week 02
1) What did I complete this week?'
- Added a basic version of the shadow detection procedure
- Implemented some changes from dev feedback
- Cleaned up the cloud detection code reducing the r.mapcalc calls and other computations in order to improve performance
- Cleaned up the shadow detection code reducing the r.mapcalc calls and other computations in order to improve performance
- Integrated the cloud and shadow intersection part
- Managed temporary file
- Tested the python script and fixed bugs
- Frequently added the python script to my GitHub repository
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test it and fix bugs
- Start implementing the GUI
3) Is there any blocking issue?
No at the moment.
Week 03
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. r.univar instead of r.stats.zonal)
- Tested the modified python script and fixed bugs
- Prepared the python script in order to start implementing the GUI
- Started implementing the GUI
- Made some changes to the code depending on the GUI requirements (add controls and check on input, output and temporary file, etc.)
- Cleaned up the whole code
- Tested the GUI and fixed bugs
- Frequently added the basic version of the GUI to my GitHub repository [0]
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Finish the implementation of the GUI
- Start writing the manual page
3) Is there any blocking issue?
No at the moment.
Week 04
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. added a new flag to manage the two procedure separately and added the text file option to specify input bands)
- Tested the modified python script and fixed bugs (e.g. solved a bug for -s flag)
- Created a real complete GRASS GIS module that can be installed with g.extension
- Finished implementing the GUI
- Tested the GUI and fixed bugs
- Cleaned up the code from the style point of view in order to make it more readable (followed PEP8 style guide and GRASS Python Scripting Library rules, converted python lists into dictionaries, added comments, messages and warnings, etc.)
- Started writing the manual page
- Solved a problem with g.extension thanks to dev community suggestions
- Discussed future improvements with the dev community
- Frequently added the new version of the code to my GitHub repository
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Finish writing the manual page
- Check the code with mentors
- Prepare deliverable for the evaluation
3) Is there any blocking issue?
No at the moment.
Week 05 - first evaluation
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. simplified the code for reading the input text file and the handling of temporary maps, small changes to make the code more readable, etc.)
- Finished writing the first version of the manual page
- Cleaned up further the code from the style point of view in order to prepare it for the submission to the official GRASS-SVN repository (followed PEP8 style guide and GRASS Python Scripting Library rules.)
- Frequently added the new version of the code to my GitHub repository
- Read and applied all the rules for submitting a GRASS GIS Addon
- Submitted the code to the official GRASS-SVN repository and informed the dev community
- Discussed possible improvements with the dev community and mentors
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Start integrating the atmospheric correction (i.atcorr) within the clouds and shadows detection algorithm
- Start defining an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr
3) Is there any blocking issue?
No at the moment.
Week 06
1) What did I complete this week?
- Better investigated the library ElementTree for parsing xml file (sentinel metadata)
- Started defining an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr
- Started writing the basic code for the automatic compilation of the control file
- Wrote a basic python script for retrieving AOT value from AERONET data
- Discussed with dev community about the implementation of the topographic correction too
- Made some tests about topographic correction using the GRASS GIS module i.topo.cor
Unfortunately, I wasn't able to code for two days because of some commitments related to the PhD.
2) What am I going to achieve for next week?
- Understand how to automatically retrieve some parameters for the control file like aerosol and atmospheric models
- Finish defining the automatic procedure for retrieve all the necessary parameters for the control file
- Start integrating the atmospheric correction (i.atcorr)
3) Is there any blocking issue?
No at the moment.
