wiki:GSoC/2018/CloudsAndShadowsDetection

Context Navigation

Version 21 (modified by Robifag, 7 years ago) ( diff )
--

GSoC 2018 GRASS GIS module for Sentinel-2 cloud and shadow detection

Title:	GRASS GIS module for Sentinel-2 cloud and shadow detection
Student Name:	Roberta Fagandini, Ph.D. student at Politecnico of Milano
Organization:	OSGeo - Open Source Geospatial Foundation
Mentor Name:	Roberto Marzocchi, Moritz Lennert
GSoC proposal:	GRASS GIS module for Sentinel-2 cloud and shadow detection
Repositories:	Github for development: https://github.com/RobiFag/GRASS_clouds_and_shadows

Abstract

Unlike Landsat images, Sentinel-2 datasets do not include thermal and Quality Assessment bands that simplify the detection of clouds avoiding erroneous classification. Moreover, also clouds shadows on the ground lead to anomalous reflectance values which have to be taken into account during the image processing. To date, only a specific module for Landsat automatic cloud coverage assessment is available within GRASS GIS (i.landsat.acca) while regarding shadows, no specific module is available. Therefore to date, the detection of clouds and shadows has to be manually performed for Sentinel-2 images.

Workflow

The workflow of the algorithm is described HERE

Goal

The aim is the coding of a specific module for GRASS GIS application which implements an automatic procedure for clouds and shadow detection within Sentinel 2 images. The module has to provide a tool which can be easily used by inexpert users, taking advantage of the suggested parameters, or by more expert users that can modify default values according to their needs.

Timeline

Time Period	Tasks	Deliverables	Status
Community Bonding	Introduce myself in soc and dev lists Get in contact with my mentors Discuss and share ideas, suggestions, etc. with my mentors and the dev list Prepare the wiki page about the project and keep it up to date Set up the github repository of the project (https://github.com/RobiFag/GRASS_clouds_and_shadows) Get familiar with development environment Follow the procedure for access to the GRASS-Addons-SVN repository Familiarize myself with GRASS Python scripting library and the writing of GRASS module Elaborate a schema of the algorithm share and discuss it with the community list	Full Report HERE	X
MAY 14 - 20	Make any changes to the algorithm deriving from the discussions and feedback and define the final version of the algorithm Test the final version of the algorithm with several datasets Start translating the final algorithm about clouds and shadows detection in a GRASS Python script		X
MAY 21 - 27	Translate the final algorithm about clouds and shadows detection in a GRASS Python script Improve the GRASS python script (manage temporary file, clean up the code, etc.) Test and validate the script and fix bugs Share the script with the community		X
MAY 28 - JUNE 3	Implement any change from discussions and feedback Start implementing the GUI Test and validate the script and fix bugs Share the script with the community		X
JUNE 4 - JUNE 10	Implement any change from discussions and feedback Finish the implementation of the GUI Test and validate the script with several datasets and fix bugs Share the script with the community		X
JUNE 11 - JUNE 15 First evaluation	Final tests Check the script with mentors Share the script with the community Discuss with mentors about the next coding steps Prepare deliverable for the evaluation	Working GRASS Python Script with GUI	X
JUNE 16 - JUNE 24	Implement any change from discussions and feedback Start integrating the atmospheric correction (i.atcorr) within the clouds and shadows detection algorithm Define an automatic procedure in order to retrieve all the necessary parameters for the control file of i.atcorr Share and discuss it with the community
JUNE 25 - JULY 1	Implement any change from discussions and feedback Translate the automatic procedure for the retrieving of i.atcorr parameters in a GRASS Python script Test and fix bugs
JULY 2 - JULY 8	Finish the implementation of i.atcorr with a working automated procedure for retrieving requested input parameters Set up the GUI for this part of the script Share the script with the community
JULY 9 - JULY 13 Second evaluation	Implement any change from discussions and feedback Check the module with mentors Share the module with the community Prepare deliverable for the evaluation	Complete GRASS module prototype
JULY 14 - JULY 22	Implement any change from discussions and feedback Start compiling the user documentation (GRASS GIS hel page) Evaluate other existing algorithm of clouds and shadows detection to be implemented in the module
JULY 23 - JULY 29	Depending on the availability of other algorithms, evaluate their integration within the module Finish compiling the user documentation (GRASS GIS hel page) Share the module with the community
JULY 30 - AUGUST 5	Depending on previous evaluations, integration of at least another algorithm Improvement of the user documentation (following feedback from community)
JULY 30 - AUGUST 5	Test and fix bugs Share the module with the community Implement any change from discussions and feedback
AUGUST 9 - AUGUST 13 Final evaluation	Test and fix bugs Publication of the final module prototype as a GRASS add-on (svn official repository) Prepare deliverable for the final evaluation	Final GRASS add-on module

Reports

Bonding Period

Introduce myself in soc and dev lists
After being accepted as a student for GSoC 2018, I introduced my self in SOC and grass-dev lists on 24th of April (0).

Get in contact with my mentors
At the same time, I got in contact with my mentors, Moritz Lennert and Roberto Marzocchi. I corresponded with them by email more or less every day from the beginning to the end of the bonding period describing all the activities that I was carrying out (0)(1). I asked them suggestions about the best tool to set-up my dev environment, how to do that, etc. I also better detailed my proposal explaining to them the starting point of the project, the procedure, the algorithm and I asked them for hints and feedback. I shared immediately with them and the grass-dev list all the necessary information and all the new features like links to the wiki page, GitHub repository, schemas and description of the project, etc.

Prepare the wiki page about the project and keep it up to date
I requested my OSGeo user ID on 26th of April and, once received, I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3). The wiki page includes: General information about the project (title, mentors, links to the proposal and GitHub repository, etc.); A brief description of the project; Link to general and detailed schemas of the procedure to better describe the whole workflow; Goal; Timeline of tasks and deliverables. After discussing with my mentors, I have better detailed the timeline adding specific tasks for each week and the deliverables for the evaluations. I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4).

Set up the GitHub repository of the project
I chose GitHub as a public repository for the development of the GRASS GIS module for the automatic detection of cloud and shadow in Sentinel 2 images, under the guidance of my mentors. I studied the GitHub documentation (5) and followed tutorials (6) in order to understand how it works and how to use it. After that, I set up my GitHub repository (7) and I started committing some test files. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors and the dev community. My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS.

Follow the procedure for access to the GRASS-Addons-SVN repository
I requested the write access to the GRASS-Addons-SVN repository following the official procedure and now I have been added to the grass-addons contributors' list. This step was necessary to submit the code as a GRASS GIS addon at the end of the GSoC 2018.

Developer environment
I checked and updated my developer environment and installed all the necessary libraries and dependencies so as to be ready to start coding after the bonding period.

Familiarize myself with GRASS Python scripting library and the writing of GRASS module
During the bonding period, I took lots of time to get familiar with GRASS Python scripting library and the writing of a GRASS module. First of all, I started from the community practices about how to contribute to GRASS GIS (8). Then I studied technical issues like function definition following the official documentation of GRASS Python scripting library (9). I also investigated the existing scripts (10)(11) to better understand how they work and how to structure addon code. During this phase, I made some coding tests that I shared with my mentors and the dev community through my repository.

Elaborate a schema of the algorithm
In order to better explain and define all the steps and procedures of the algorithm to be implemented as a GRASS addon, I elaborate a general workflow schema and specific diagrams for each procedure (cloud detection, shadow detection, etc.) (12). For the proper and complete consultation of the schemas, please use the google drive app diagram.io.

Share and discuss it with the community list
Obviously, I shared everything with my mentors and the dev community. I started a new thread on the list adding all the necessary and useful links (wiki, repository, schema and full description of the project) and asking explicitly for feedback and hints (13).

(0) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088254.html
(1) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088314.html
(2) https://trac.osgeo.org/grass/wiki/GSoC/2018/CloudsAndShadowsDetection
(3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted
(4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students
(5) https://guides.github.com/
(6) https://www.youtube.com/githubguides
(7) https://github.com/RobiFag/GRASS_clouds_and_shadows
(8) https://trac.osgeo.org/grass/wiki/HowToContribute
(9) https://grass.osgeo.org/grass75/manuals/libpython/script_intro.html
(10) https://trac.osgeo.org/grass/browser/grass/trunk/scripts
(11) https://trac.osgeo.org/grass/browser/grass-addons/grass7
(12) https://drive.google.com/file/d/1KYEKvNBurBFHw1xUTLjM0PW80Z-7br81/view?usp=sharing
(13) https://lists.osgeo.org/pipermail/grass-dev/2018-May/088369.html

Week 01

Since my progress has been a bit faster than expected I have already discussed with my mentors about further additions to the project. Therefore I have already updated my wiki page changing the timeline. In this way, I will be able to release a more complete module.

1) What did I complete this week?

Checked the prototype version of the algorithm for cloud and shadow detection
Tested it
I better studied the documentation of the GRASS Python Scripting Library and started coding the GRASS python script following the workflow schema prepared during the bonding period
Coded the 'data preparation' phase
Coded the clouds detection procedure
Added this first part of python script to my GitHub repository
Discussed with my mentors about further additions
Discussed with the community about testing phase. I received some feedback and some developers stepped forward to help me in testing the algorithm
Shared progress with the community

Unfortunately, I wasn't able to code on Thursday because of a PhD meeting with my supervisor.