wiki:GSoC/2018/CloudsAndShadowsDetection

Version 19 (modified by Robifag, 6 years ago) ( diff )

--

GSoC 2018 GRASS GIS module for Sentinel-2 cloud and shadow detection

Title: GRASS GIS module for Sentinel-2 cloud and shadow detection
Student Name: Roberta Fagandini, Ph.D. student at Politecnico of Milano
Organization: OSGeo - Open Source Geospatial Foundation
Mentor Name: Roberto Marzocchi, Moritz Lennert
GSoC proposal: GRASS GIS module for Sentinel-2 cloud and shadow detection
Repositories: Github for development: https://github.com/RobiFag/GRASS_clouds_and_shadows

Abstract

Unlike Landsat images, Sentinel-2 datasets do not include thermal and Quality Assessment bands that simplify the detection of clouds avoiding erroneous classification. Moreover, also clouds shadows on the ground lead to anomalous reflectance values which have to be taken into account during the image processing. To date, only a specific module for Landsat automatic cloud coverage assessment is available within GRASS GIS (i.landsat.acca) while regarding shadows, no specific module is available. Therefore to date, the detection of clouds and shadows has to be manually performed for Sentinel-2 images.

Workflow

The workflow of the algorithm is described HERE

Goal

The aim is the coding of a specific module for GRASS GIS application which implements an automatic procedure for clouds and shadow detection within Sentinel 2 images. The module has to provide a tool which can be easily used by inexpert users, taking advantage of the suggested parameters, or by more expert users that can modify default values according to their needs.

Timeline

Time Period

Tasks

Deliverables

Status

Community Bonding

  • Introduce myself in soc and dev lists
  • Get in contact with my mentors
  • Discuss and share ideas, suggestions, etc. with my mentors and the dev list
  • Prepare the wiki page about the project and keep it up to date
  • Set up the github repository of the project (https://github.com/RobiFag/GRASS_clouds_and_shadows)
  • Get familiar with development environment
  • Follow the procedure for access to the GRASS-Addons-SVN repository
  • Familiarize myself with GRASS Python scripting library and the writing of GRASS module
  • Elaborate a schema of the algorithm share and discuss it with the community list

Full Report HERE

X

MAY 14 - 20

  • Make any changes to the algorithm deriving from the discussions and feedback and define the final version of the algorithm
  • Test the final version of the algorithm with several datasets
  • Start translating the final algorithm about clouds and shadows detection in a GRASS Python script

X

MAY 21 - 27

  • Translate the final algorithm about clouds and shadows detection in a GRASS Python script
  • Improve the GRASS python script (manage temporary file, clean up the code, etc.)
  • Test and validate the script and fix bugs
  • Share the script with the community

X

MAY 28 - JUNE 3

  • Implement any change from discussions and feedback
  • Start implementing the GUI
  • Test and validate the script and fix bugs
  • Share the script with the community

X

JUNE 4 - JUNE 10

  • Implement any change from discussions and feedback
  • Finish the implementation of the GUI
  • Test and validate the script with several datasets and fix bugs
  • Share the script with the community

X

JUNE 11 - JUNE 15
First evaluation

  • Final tests
  • Check the script with mentors
  • Share the script with the community
  • Discuss with mentors about the next coding steps
  • Prepare deliverable for the evaluation

Working GRASS Python Script with GUI

JUNE 16 - JUNE 24

  • Implement any change from discussions and feedback
  • Start integrating the atmospheric correction (i.atcorr) within the clouds and shadows detection algorithm
  • Define an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr
  • Share and discuss it with the community

JUNE 25 - JULY 1

  • Implement any change from discussions and feedback
  • Translate the automatic procedure for the retrieving of i.atcorr parameters in a GRASS Python script
  • Test and fix bugs

JULY 2 - JULY 8

  • Finish the implementation of i.atcorr with a working automated procedure for retrieving requested input parameters
  • Set up the GUI for this part of the script
  • Share the script with the community

JULY 9 - JULY 13
Second evaluation

  • Implement any change from discussions and feedback
  • Check the module with mentors
  • Share the module with the community
  • Prepare deliverable for the evaluation

Complete GRASS module prototype

JULY 14 - JULY 22

  • Implement any change from discussions and feedback
  • Start compiling the user documentation (GRASS GIS hel page)
  • Evaluate other existing algorithm of clouds and shadows detection to be implemented in the module

JULY 23 - JULY 29

  • Depending on the availability of other algorithms, evaluate their integration within the module
  • Finish compiling the user documentation (GRASS GIS hel page)
  • Share the module with the community

JULY 30 - AUGUST 5

  • Depending on previous evaluations, integration of at least another algorithm
  • Improvement of the user documentation (following feedback from community)

JULY 30 - AUGUST 5

  • Test and fix bugs
  • Share the module with the community
  • Implement any change from discussions and feedback

AUGUST 9 - AUGUST 13
Final evaluation

  • Test and fix bugs
  • Publication of the final module prototype as a GRASS add-on (svn official repository)
  • Prepare deliverable for the final evaluation

Final GRASS add-on module

Reports

Bonding Period

Introduce myself in soc and dev lists
After being accepted as a student for GSoC 2018, I introduced my self in SOC and grass-dev lists on 24th of April (0).

Get in contact with my mentors
At the same time, I got in contact with my mentors, Moritz Lennert and Roberto Marzocchi. I corresponded with them by email more or less every day from the beginning to the end of the bonding period describing all the activities that I was carrying out (0)(1). I asked them suggestions about the best tool to set-up my dev environment, how to do that, etc. I also better detailed my proposal explaining to them the starting point of the project, the procedure, the algorithm and I asked them for hints and feedback. I shared immediately with them and the grass-dev list all the necessary information and all the new features like links to the wiki page, GitHub repository, schemas and description of the project, etc.

Prepare the wiki page about the project and keep it up to date
I requested my OSGeo user ID on 26th of April and, once received, I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3). The wiki page includes: General information about the project (title, mentors, links to the proposal and GitHub repository, etc.); A brief description of the project; Link to general and detailed schemas of the procedure to better describe the whole workflow; Goal; Timeline of tasks and deliverables. After discussing with my mentors, I have better detailed the timeline adding specific tasks for each week and the deliverables for the evaluations. I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4).

Set up the GitHub repository of the project
I chose GitHub as a public repository for the development of the GRASS GIS module for the automatic detection of cloud and shadow in Sentinel 2 images, under the guidance of my mentors. I studied the GitHub documentation (5) and followed tutorials (6) in order to understand how it works and how to use it. After that, I set up my GitHub repository (7) and I started committing some test files. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors and the dev community. My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS.

Follow the procedure for access to the GRASS-Addons-SVN repository
I requested the write access to the GRASS-Addons-SVN repository following the official procedure and now I have been added to the grass-addons contributors' list. This step was necessary to submit the code as a GRASS GIS addon at the end of the GSoC 2018.

Developer environment
I checked and updated my developer environment and installed all the necessary libraries and dependencies so as to be ready to start coding after the bonding period.

Familiarize myself with GRASS Python scripting library and the writing of GRASS module
During the bonding period, I took lots of time to get familiar with GRASS Python scripting library and the writing of a GRASS module. First of all, I started from the community practices about how to contribute to GRASS GIS (8). Then I studied technical issues like function definition following the official documentation of GRASS Python scripting library (9). I also investigated the existing scripts (10)(11) to better understand how they work and how to structure addon code. During this phase, I made some coding tests that I shared with my mentors and the dev community through my repository.

Elaborate a schema of the algorithm
In order to better explain and define all the steps and procedures of the algorithm to be implemented as a GRASS addon, I elaborate a general workflow schema and specific diagrams for each procedure (cloud detection, shadow detection, etc.) (12). For the proper and complete consultation of the schemas, please use the google drive app diagram.io.

Share and discuss it with the community list
Obviously, I shared everything with my mentors and the dev community. I started a new thread on the list adding all the necessary and useful links (wiki, repository, schema and full description of the project) and asking explicitly for feedback and hints (13).

(0) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088254.html
(1) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088314.html
(2) https://trac.osgeo.org/grass/wiki/GSoC/2018/CloudsAndShadowsDetection
(3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted
(4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students
(5) https://guides.github.com/
(6) https://www.youtube.com/githubguides
(7) https://github.com/RobiFag/GRASS_clouds_and_shadows
(8) https://trac.osgeo.org/grass/wiki/HowToContribute
(9) https://grass.osgeo.org/grass75/manuals/libpython/script_intro.html
(10) https://trac.osgeo.org/grass/browser/grass/trunk/scripts
(11) https://trac.osgeo.org/grass/browser/grass-addons/grass7
(12) https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing
(13) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088369.html

Week 01

Since my progress has been a bit faster than expected I have already discussed with my mentors about further additions to the project. Therefore I have already updated my wiki page changing the timeline. In this way, I will be able to release a more complete module.

1) What did I complete this week?

  • Checked the prototype version of the algorithm for cloud and shadow detection
  • Tested it
  • I better studied the documentation of the GRASS Python Scripting Library and started coding the GRASS python script following the workflow schema prepared during the bonding period
  • Coded the 'data preparation' phase
  • Coded the clouds detection procedure
  • Added this first part of python script to my GitHub repository
  • Discussed with my mentors about further additions
  • Discussed with the community about testing phase. I received some feedback and some developers stepped forward to help me in testing the algorithm
  • Shared progress with the community

Unfortunately, I wasn't able to code on Thursday because of a PhD meeting with my supervisor.

2) What am I going to achieve for next week?

  • Finish the translation of the algorithm in a basic python script adding the shadows detection procedure and the steps for checking and cleaning the final outputs
  • Improve the python script so as to have a "light" working python script (manage temporary file, clean up the code, etc.)
  • Test it and fix bugs
  • Start implementing the GUI

3) Is there any blocking issue?
No at the moment.

Week 02

1) What did I complete this week?'

  • Added a basic version of the shadow detection procedure
  • Implemented some changes from dev feedback
  • Cleaned up the cloud detection code reducing the r.mapcalc calls and other computations in order to improve performance
  • Cleaned up the shadow detection code reducing the r.mapcalc calls and other computations in order to improve performance
  • Integrated the cloud and shadow intersection part
  • Managed temporary file
  • Tested the python script and fixed bugs
  • Frequently added the python script to my GitHub repository
  • Shared progress with the community

2) What am I going to achieve for next week?

  • Implement any change from discussions and feedback
  • Test it and fix bugs
  • Start implementing the GUI

3) Is there any blocking issue?
No at the moment.

Week 03

1) What did I complete this week?

  • Implemented some changes from dev feedback (e.g. r.univar instead of r.stats.zonal)
  • Tested the modified python script and fixed bugs
  • Prepared the python script in order to start implementing the GUI
  • Started implementing the GUI
  • Made some changes to the code depending on the GUI requirements (add controls and check on input, output and temporary file, etc.)
  • Cleaned up the whole code
  • Tested the GUI and fixed bugs
  • Frequently added the basic version of the GUI to my GitHub repository [0]
  • Shared progress with the community

2) What am I going to achieve for next week?

  • Implement any change from discussions and feedback
  • Test and fix bugs
  • Finish the implementation of the GUI
  • Start writing the manual page

3) Is there any blocking issue?
No at the moment.

Week 04

1) What did I complete this week?

  • Implemented some changes from dev feedback (e.g. added a new flag to manage the two procedure separately and added the text file option to specify input bands)
  • Tested the modified python script and fixed bugs (e.g. solved a bug for -s flag)
  • Created a real complete GRASS GIS module that can be installed with g.extension
  • Finished implementing the GUI
  • Tested the GUI and fixed bugs
  • Cleaned up the code from the style point of view in order to make it more readable (followed PEP8 style guide and GRASS Python Scripting Library rules, converted python lists into dictionaries, added comments, messages and warnings, etc.)
  • Started writing the manual page
  • Solved a problem with g.extension thanks to dev community suggestions
  • Discussed future improvements with the dev community
  • Frequently added the new version of the code to my GitHub repository
  • Shared progress with the community

2) What am I going to achieve for next week?

  • Implement any change from discussions and feedback
  • Test and fix bugs
  • Finish writing the manual page
  • Check the code with mentors
  • Prepare deliverable for the evaluation

3) Is there any blocking issue?
No at the moment.

Note: See TracWiki for help on using the wiki.