| Version 29 (modified by , 6 years ago) ( diff ) |
|---|
GSoC 2018 GRASS GIS module for Sentinel-2 cloud and shadow detection
| Title: | GRASS GIS module for Sentinel-2 cloud and shadow detection |
| Student Name: | Roberta Fagandini, Ph.D. student at Politecnico of Milano |
| Organization: | OSGeo - Open Source Geospatial Foundation |
| Mentor Name: | Roberto Marzocchi, Moritz Lennert |
| GSoC proposal: | GRASS GIS module for Sentinel-2 cloud and shadow detection |
| Repositories: | Github for development: https://github.com/RobiFag/GRASS_clouds_and_shadows |
Abstract
Unlike Landsat images, Sentinel-2 datasets do not include thermal and Quality Assessment bands that simplify the detection of clouds avoiding erroneous classification. Moreover, also clouds shadows on the ground lead to anomalous reflectance values which have to be taken into account during the image processing. To date, only a specific module for Landsat automatic cloud coverage assessment is available within GRASS GIS (i.landsat.acca) while regarding shadows, no specific module is available. Therefore to date, the detection of clouds and shadows has to be manually performed for Sentinel-2 images.
Workflow
The workflow of the algorithm is described HERE
Goal
The aim is the coding of a specific module for GRASS GIS application which implements an automatic procedure for clouds and shadow detection within Sentinel 2 images. The module has to provide a tool which can be easily used by inexpert users, taking advantage of the suggested parameters, or by more expert users that can modify default values according to their needs.
Timeline
Time Period | Tasks | Deliverables | Status |
|---|---|---|---|
Community Bonding |
| Full Report HERE | X |
MAY 14 - 20 |
| X | |
MAY 21 - 27 |
| X | |
MAY 28 - JUNE 3 |
| X | |
JUNE 4 - JUNE 10 |
| X | |
JUNE 11 - JUNE 15 |
| Working GRASS Python Script with GUI | X |
JUNE 16 - JUNE 24 |
| X | |
JUNE 25 - JULY 1 |
| X | |
JULY 2 - JULY 8 |
| X | |
JULY 9 - JULY 13 |
| Complete GRASS module prototype | X |
JULY 14 - JULY 22 |
| X | |
JULY 23 - JULY 29 |
| X | |
JULY 30 - AUGUST 5 |
| X | |
AUGUST 6 - AUGUST 14 |
| Final GRASS add-on module | |
Reports
Bonding Period
Introduce myself in soc and dev lists
After being accepted as a student for GSoC 2018, I introduced my self in SOC and grass-dev lists on 24th of April (0).
Get in contact with my mentors
At the same time, I got in contact with my mentors, Moritz Lennert and Roberto Marzocchi.
I corresponded with them by email more or less every day from the beginning to the end of the bonding period describing all the activities that I was carrying out (0)(1). I asked them suggestions about the best tool to set-up my dev environment, how to do that, etc.
I also better detailed my proposal explaining to them the starting point of the project, the procedure, the algorithm and I asked them for hints and feedback.
I shared immediately with them and the grass-dev list all the necessary information and all the new features like links to the wiki page, GitHub repository, schemas and description of the project, etc.
Prepare the wiki page about the project and keep it up to date
I requested my OSGeo user ID on 26th of April and, once received, I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3).
The wiki page includes:
General information about the project (title, mentors, links to the proposal and GitHub repository, etc.);
A brief description of the project;
Link to general and detailed schemas of the procedure to better describe the whole workflow;
Goal;
Timeline of tasks and deliverables.
After discussing with my mentors, I have better detailed the timeline adding specific tasks for each week and the deliverables for the evaluations.
I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4).
Set up the GitHub repository of the project
I chose GitHub as a public repository for the development of the GRASS GIS module for the automatic detection of cloud and shadow in Sentinel 2 images, under the guidance of my mentors. I studied the GitHub documentation (5) and followed tutorials (6) in order to understand how it works and how to use it. After that, I set up my GitHub repository (7) and I started committing some test files. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors and the dev community.
My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS.
Follow the procedure for access to the GRASS-Addons-SVN repository
I requested the write access to the GRASS-Addons-SVN repository following the official procedure and now I have been added to the grass-addons contributors' list. This step was necessary to submit the code as a GRASS GIS addon at the end of the GSoC 2018.
Developer environment
I checked and updated my developer environment and installed all the necessary libraries and dependencies so as to be ready to start coding after the bonding period.
Familiarize myself with GRASS Python scripting library and the writing of GRASS module
During the bonding period, I took lots of time to get familiar with GRASS Python scripting library and the writing of a GRASS module. First of all, I started from the community practices about how to contribute to GRASS GIS (8). Then I studied technical issues like function definition following the official documentation of GRASS Python scripting library (9). I also investigated the existing scripts (10)(11) to better understand how they work and how to structure addon code.
During this phase, I made some coding tests that I shared with my mentors and the dev community through my repository.
Elaborate a schema of the algorithm
In order to better explain and define all the steps and procedures of the algorithm to be implemented as a GRASS addon, I elaborate a general workflow schema and specific diagrams for each procedure (cloud detection, shadow detection, etc.) (12). For the proper and complete consultation of the schemas, please use the google drive app diagram.io.
Share and discuss it with the community list
Obviously, I shared everything with my mentors and the dev community. I started a new thread on the list adding all the necessary and useful links (wiki, repository, schema and full description of the project) and asking explicitly for feedback and hints (13).
(0) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088254.html
(1) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088314.html
(2) https://trac.osgeo.org/grass/wiki/GSoC/2018/CloudsAndShadowsDetection
(3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted
(4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students
(5) https://guides.github.com/
(6) https://www.youtube.com/githubguides
(7) https://github.com/RobiFag/GRASS_clouds_and_shadows
(8) https://trac.osgeo.org/grass/wiki/HowToContribute
(9) https://grass.osgeo.org/grass75/manuals/libpython/script_intro.html
(10) https://trac.osgeo.org/grass/browser/grass/trunk/scripts
(11) https://trac.osgeo.org/grass/browser/grass-addons/grass7
(12) https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing
(13) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088369.html
Week 01
Since my progress has been a bit faster than expected I have already discussed with my mentors about further additions to the project. Therefore I have already updated my wiki page changing the timeline. In this way, I will be able to release a more complete module.
1) What did I complete this week?
- Checked the prototype version of the algorithm for cloud and shadow detection
- Tested it
- I better studied the documentation of the GRASS Python Scripting Library and started coding the GRASS python script following the workflow schema prepared during the bonding period
- Coded the 'data preparation' phase
- Coded the clouds detection procedure
- Added this first part of python script to my GitHub repository
- Discussed with my mentors about further additions
- Discussed with the community about testing phase. I received some feedback and some developers stepped forward to help me in testing the algorithm
- Shared progress with the community
Unfortunately, I wasn't able to code on Thursday because of a PhD meeting with my supervisor.
2) What am I going to achieve for next week?
- Finish the translation of the algorithm in a basic python script adding the shadows detection procedure and the steps for checking and cleaning the final outputs
- Improve the python script so as to have a "light" working python script (manage temporary file, clean up the code, etc.)
- Test it and fix bugs
- Start implementing the GUI
3) Is there any blocking issue?
No at the moment.
Week 02
1) What did I complete this week?'
- Added a basic version of the shadow detection procedure
- Implemented some changes from dev feedback
- Cleaned up the cloud detection code reducing the r.mapcalc calls and other computations in order to improve performance
- Cleaned up the shadow detection code reducing the r.mapcalc calls and other computations in order to improve performance
- Integrated the cloud and shadow intersection part
- Managed temporary file
- Tested the python script and fixed bugs
- Frequently added the python script to my GitHub repository
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test it and fix bugs
- Start implementing the GUI
3) Is there any blocking issue?
No at the moment.
Week 03
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. r.univar instead of r.stats.zonal)
- Tested the modified python script and fixed bugs
- Prepared the python script in order to start implementing the GUI
- Started implementing the GUI
- Made some changes to the code depending on the GUI requirements (add controls and check on input, output and temporary file, etc.)
- Cleaned up the whole code
- Tested the GUI and fixed bugs
- Frequently added the basic version of the GUI to my GitHub repository [0]
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Finish the implementation of the GUI
- Start writing the manual page
3) Is there any blocking issue?
No at the moment.
Week 04
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. added a new flag to manage the two procedure separately and added the text file option to specify input bands)
- Tested the modified python script and fixed bugs (e.g. solved a bug for -s flag)
- Created a real complete GRASS GIS module that can be installed with g.extension
- Finished implementing the GUI
- Tested the GUI and fixed bugs
- Cleaned up the code from the style point of view in order to make it more readable (followed PEP8 style guide and GRASS Python Scripting Library rules, converted python lists into dictionaries, added comments, messages and warnings, etc.)
- Started writing the manual page
- Solved a problem with g.extension thanks to dev community suggestions
- Discussed future improvements with the dev community
- Frequently added the new version of the code to my GitHub repository
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Finish writing the manual page
- Check the code with mentors
- Prepare deliverable for the evaluation
3) Is there any blocking issue?
No at the moment.
Week 05 - first evaluation
1) What did I complete this week?
- Implemented some changes from dev feedback (e.g. simplified the code for reading the input text file and the handling of temporary maps, small changes to make the code more readable, etc.)
- Finished writing the first version of the manual page
- Cleaned up further the code from the style point of view in order to prepare it for the submission to the official GRASS-SVN repository (followed PEP8 style guide and GRASS Python Scripting Library rules.)
- Frequently added the new version of the code to my GitHub repository
- Read and applied all the rules for submitting a GRASS GIS Addon
- Submitted the code to the official GRASS-SVN repository and informed the dev community
- Discussed possible improvements with the dev community and mentors
- Shared progress with the community
2) What am I going to achieve for next week?
- Implement any change from discussions and feedback
- Test and fix bugs
- Start integrating the atmospheric correction (i.atcorr) within the clouds and shadows detection algorithm
- Start defining an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr
3) Is there any blocking issue?
No at the moment.
Week 06
1) What did I complete this week?
- Better investigated the library ElementTree for parsing xml file (sentinel metadata)
- Started defining an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr
- Started writing the basic code for the automatic compilation of the control file
- Wrote a basic python script for retrieving AOT value from AERONET data
- Discussed with dev community about the implementation of the topographic correction too
- Made some tests about topographic correction using the GRASS GIS module i.topo.cor
Unfortunately, I wasn't able to code for two days because of some commitments related to the PhD.
2) What am I going to achieve for next week?
- Understand how to automatically retrieve some parameters for the control file like aerosol and atmospheric models
- Finish defining the automatic procedure for retrieve all the necessary parameters for the control file
- Start integrating the atmospheric correction (i.atcorr)
3) Is there any blocking issue?
No at the moment.
Week 07
1) What did I complete this week?
- Continued coding the GRASS python script to execute i.atcorr for all bands of the input image changing accordingly the requested input parameters and control file (0)
- Changed some lines of the code according to the i.atcorr manual page (e.g. lon/lat retrieved from the computational region and not from metadata file) (0)
- Added the automatic computation of mean target elevation from the input dem (0)
- Wrote and added the python script for retrieving the aerosol optical thickness (AOT) value from AERONET data using Py6S library (1)
- Discussed with dev community about the need to write my own routine to retrieve and compute AOT at 550 nm avoiding too many dependencies (2)
- Started writing my own routine to extract and compute AOT at 550 (one of the parameters of the i.atcorr control file) (3)
2) What am I going to achieve for next week?
- Finish writing the routine for AOT computation
- Understand how to automatically retrieve some parameters for the control file like aerosol and atmospheric models
- Finish defining the automatic procedure for retrieving all the last missing parameters for the control file
3) Is there any blocking issue?
No at the moment but the coding of the routine takes more time than expected.
(0) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/52ede2e0e8d157f5b19fbb414c29bbde0e728b13#diff-d42beca70d363fcee5a6ec17260c5129
(1) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/a8757bfaae04c283daab1b03216b3e0d7ed4c885
(2) https://lists.osgeo.org/pipermail/grass-dev/2018-June/088814.html
(3) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/568164ea4f093a3045b2562404ece544d1ff6f0f
Week 08
1) What did I complete this week?
- Finished writing the routine to read AERONET file and compute the value of Aerosol Optical Thickness at 550 nm (one of the parameters of the i.atcorr control file) (0)
- Integrated the routine for AOT computation into the GRASS python script to execute i.atcorr for all bands of the input image changing accordingly the requested input parameters and control file (1)
- Added the automatic and simplified procedure to identify the atmospheric model (one of the parameters of the i.atcorr control file) (1)
- Managed the use of visibility as input parameter instead of AOT value (1)
- Finished defining the automatic procedure for retrieving all the last missing parameters for the control file
- Finished and added to my GitHub repository the complete GRASS python script to execute i.atcorr for all bands (1)
- Started implementing the GUI (2)
2) What am I going to achieve for next week?
- Finish the implementation of the GUI
- Integrate the download and import of Sentinel 2 images into the new GRASS module (i.sentinel.preproc)
- Test and fix bugs
- Make the new module available through my GitHub repository
3) Is there any blocking issue?
No at the moment even though I'm a little late on the roadmap because the coding of the routine to compute AOT took slightly more time than expected.
(0) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/33d0832a0f8dc4aa0c45d37ba6598988579c0d9a
(1) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/1617184154ead85f83ce7f1a8750f00f100dc2cb
(2) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/35175c68a35e01941faead903ec8f0380b317aa6
Week 09 - second evaluation
1) What did I complete this week?
- Finished implementing the GUI (0)
- Made some changes to the code depending on the GUI requirements (add controls and check on input, output, etc.) (0)
- Tested the module and fixed bugs
- Discussed with mentors about next steps and we decided to not implement the cloud and shadow detection procedure in the new module i.sentinel.preproc
- Added to i.sentinel.preproc a flag option to automatically create the text file to be used as input in the module i.sentinel.mask (0)
- Made small changes to i.sentinel.mask in order to make it compatible with the output of i.sentinel.preproc (1)
2) What am I going to achieve for next week?
- Integrate the download and import of Sentinel 2 images into the new GRASS module (i.sentinel.preproc)
- Start writing the manual page
- Create a real complete GRASS GIS module that can be installed with g.extension from my GitHub repository
3) Is there any blocking issue?
No at the moment even though I'm a little late on the roadmap because also the implementation of the GUI took slightly more time than expected (made some changes to the code depending on the GUI requirements).
(0) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/f60952c701e87d21a7010494dec10ec603446a1f
(1) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/e2b6978651efe063d1beed603ae2bd8341b9264e
Week 10
1) What did I complete this week?
- Submitted a new version of i.sentinel.mask to the official GRASS-SVN repository (0)(1)
- Integrated the import of Sentinel 2 images into i.sentinel.preproc (2)
- Tested the module and fixed bugs
- Discussed with mentors about next steps and we decided to not implement the download procedure in the new module i.sentinel.preproc in order to avoid unnecessary and undesirable dependency as the dev community suggested
- Studied other possible python libraries for the download of Sentinel images in order to reduce the number of dependencies (e.g. sentinelhub instead of sentinelsat)
- Created a real complete GRASS GIS module that can be installed with g.extension from my GitHub repository (3)(4)
Unfortunately, I wasn't able for two days because of an unexpected personal commitment.
2) What am I going to achieve for next week?
- Write the manual page
- Start implementing a new procedure for downloading Sentinel image avoiding too many dependencies
3) Is there any blocking issue?
No at the moment.
(0) https://trac.osgeo.org/grass/changeset?reponame=&new=72992%40grass-addons%2Fgrass7%2Fimagery%2Fi.sentinel.mask%2Fi.sentinel.mask.html&old=72816%40grass-addons%2Fgrass7%2Fimagery%2Fi.sentinel.mask%2Fi.sentinel.mask.html
(1) https://trac.osgeo.org/grass/changeset?reponame=&new=72992%40grass-addons%2Fgrass7%2Fimagery%2Fi.sentinel.mask%2Fi.sentinel.mask.py&old=72816%40grass-addons%2Fgrass7%2Fimagery%2Fi.sentinel.mask%2Fi.sentinel.mask.py
(2) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/e7151f1a73895f8c0acd5692444e4fd79d83b6a2
(3) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/d22a33b8af2848ce9b479deb26cd10bc15a98133
(4) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/303592e6a73ec8b4d4178519fe507e6027040e3f
Week 11
1) What did I complete this week?
- Added a control check on input folder to check if it belongs to a L1C image (0)
- Fixed relevant bugs on the -a flag (0)
- Fixed a bug on the control file writing in case of a visibility map is provided (0)
- Small changes to the code and the GUI (0)
- Discussed with dev community the implementation of a new download procedure in order to avoid too many dependencies and as the community suggested I decided to not write a new procedure but to use the existent addon i.sentinel.download
- Wrote the manual page (1)
- Tested i.sentinel.preproc both with sentinel 2A and 2B scenes (for Sentinel 2B it works only with GRASS GIS 7.5)
2) What am I going to achieve for next week?
- Implement i.sentinel.download
- Write the manual page for the download procedure
- Test and fix bugs for both i.sentinel.preproc and i.sentinel.mask
- Start preparing deliverables for the final evaluation
3) Is there any blocking issue?
No at the moment.
(0) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/1ad8f397b1eb6b2b76cf9d037e50a3dfe1a74c0c
(1) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/5758b6c796f8b32c35f35e02a581247ed9ba48a5
Week 12
1) What did I complete this week?
- Taking the suggestions of the dev community into account and after a long discussion with my mentor, I decided to not implement i.sentinel.download in order to avoid dependencies due to the implementation of an existing module that users can use separately
- Added support for the Old format Naming Convention of Sentinel 2 images in i.sentinel.preproc (0)
- Improved the manual page of i.sentinel.preproc (1)
- Small changes to the code and the GUI of i.sentinel. preproc in order to make it more congruent with the manual page of i.atcorr (e.g. change AOT in AOD) (2)
- Improved the manual page of i.sentinel.mask adding a more detailed example and other information (3)
- Started preparing deliverables for the final evaluation (updated the readme file of my GitHub repository) (4)(5)(6)
- Tested both modules i.sentinel.mask and i.sentinel.preproc in order to improve and debug the code (small changes to the codes and the GUIs)
2) What am I going to achieve for next week?
- Last tests and possible changes
- Clean up the code of i.sentinel.preproc (PEP8 style guide and GRASS Python Scripting Library rules)
- Final check to both modules, i.sentinel.preproc and i.sentinel.mask (codes and manual pages)
- Preparing deliverables for the final evaluation
- Check everything (codes, manuals and deliverables ) with my mentor
3) Is there any blocking issue?
No.
(0) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/3bb503d2eb860dede36710ba7fb913de4fad17bf
(1) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/b7e69b1bdbaa8e9c91579091febf46718cbaf4ee#diff-15b47cd1e07802ebc64446c2fce5fef0
(2) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/949100564d79895bd992e62e3d214752e704faf7#diff-84085907e0e00fead4cef44afe6eb4e7
(3) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/fe2aa29bc656d436e28579f1086d6c60bce678aa
(4) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/fbcaa32d9fa2bbd92e85f16ec6a828aaa575a85b
(5) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/9ad83e286594012f58e44de63c9af9c9d8ce2c07
(6) https://github.com/RobiFag/GRASS_clouds_and_shadows/commit/48f3747888eed2bcbde3ea95c318be3a2ef17d11
