= GSoC 2018 Improve GRASS GIS integration in QGIS 3 = || Title: || '''Improve GRASS GIS integration in QGIS 3''' || ||Student Name: || Radek Novotný, [http://www.cvut.cz Czech Technical University in Prague] || ||Organization: || [http://www.osgeo.org OSGeo - Open Source Geospatial Foundation] || || Mentor Name: || Vaclav Petras, [https://wiki.osgeo.org/wiki/User:Martinl Marin Landa] || || !GitHub Repository: || [https://github.com/radeknovotny94/GRASSIntegrationInQGIS3 view repository] || || GSoC proposal: || [https://docs.google.com/document/d/1XlxRHvtaiebVIOvlq8G6Q5FuUkFigZ8DlQbaQ-y9keQ/edit?usp=sharing view proposal] || == Abstract == Currently, GRASS integration in QGIS is hard to maintain. There are two QGIS plugins - Processing and GRASS Plugin, which differs each other. This proposal is focused on Processing plugin. Processing plugin is at this moment maintained by QGIS developers in core Git repository. This situation has been discussed in the QGIS and GRASS GIS mailing lists several times. GUI dialogues of the Processing plugin are generated from manually maintained text UI description files. == Goal == My goal is to design a Python library (and possibly also a GRASS module based on this library) which allows creating this text UI files automatically from generated GRASS XML description. This library also has to simplify and divide parameters for QGIS. This approach would be similar to the function of SAGA (or OTB) Processing plugin integration QGIS. After this addition will be possible to generate text UI files for new GRASS modules (added to the core or even for add-ons) in an automated way. These generated UI files can be part of Git repository, but in future could be generated automatically during QGIS compilation, when a user installs GRASS Addon module using Processing plugin, or on the fly when the GUI is started. == Timeline == {{{#!th style="background: #ddd" rowspan=2 '''Time Period''' }}} {{{#!th style="background: #ddd" colspan=2 '''Milestones''' }}} |----------------------- {{{#!th style="background: #ddd" Tasks }}} {{{#!th style="background: #ddd" Status }}} |----------------------- {{{#!td April 23 - May 14 \\ ''Community Bonding'' }}} {{{#!td - Introduce myself in dev list, get in contact with my mentors and discuss the design of functions - Prepare the wiki page about the project - Set up the !GitHub repository of the project - Set up a developer environment }}} {{{#!td Ok \\ Ok \\ Ok \\ Ok }}} |----------------------- {{{#!td May 14 - May 20 \\ ''Week 1'' }}} {{{#!td - Implement an initial version of Python library for generating UI files used by Processing plugin based on GRASS modules XML description. }}} {{{#!td Done }}} |----------------------- {{{#!td May 21 - May 27 \\ ''Week 2'' }}} {{{#!td - Integration of generated description UI files from initial Python library to Processing plugin in order to generate GUI dialogue for GRASS modules. }}} {{{#!td Done }}} |----------------------- {{{#!td May 28 - June 3 \\ ''Week 3'' }}} {{{#!td - Split GRASS module parameters in Processing plugin to basic and advanced tabs. }}} {{{#!td Postponed }}} |----------------------- {{{#!td June 4 - June 10 \\ ''Week 4'' }}} {{{#!td - Customization of parameter descriptions in order to take care of differences in GRASS and QGIS terminology. }}} {{{#!td In progress }}} |----------------------- {{{#!td style="background: #ddd" July 11 - July 15 \\ ''Evaluation 1'' }}} {{{#!td style="background: #ddd" - Goal: Working Python library generating files which could be used in Processing plugin - Mentors evaluate me and I evaluate mentors of official coding period phase. }}} {{{#!td }}} |----------------------- {{{#!td June 11 - June 17 \\ ''Week 5'' }}} {{{#!td - Solve how to split GRASS modules for Processing plugin and minimize user confusion. }}} {{{#!td In progress }}} |----------------------- {{{#!td June 18 - June 24 \\ ''Week 6'' }}} {{{#!td - Bugfixing, fix problems which appear after integration of Python library in Processing plugin. }}} {{{#!td In progress }}} |----------------------- {{{#!td June 25 - July 1 \\ ''Week 7'' }}} {{{#!td - Implement simplified run or --exec method logic in GRASS. }}} {{{#!td }}} |----------------------- {{{#!td July 2 - July 8 \\ ''Week 8'' }}} {{{#!td - Research actual processing code and simplified it for using a run method. }}} {{{#!td }}} |----------------------- {{{#!td style="background: #ddd" July 9 - July 13 \\ ''Evaluation 2'' }}} {{{#!td style="background: #ddd" - Goal: Release version of the library, implemented run method in GRASS - Mentors evaluate me and I evaluate mentors of official coding period phase. }}} {{{#!td }}} |----------------------- {{{#!td July 9 - July 15 \\ ''Week 9'' }}} {{{#!td - Integrate GRASS run logic in Processing plugin }}} {{{#!td }}} |----------------------- {{{#!td July 16 - July 22 \\ ''Week 10'' }}} {{{#!td - Create documentation related to GRASS Processing plugin integration (generating UI files, GRASS caller, …). - Testing implementation. }}} {{{#!td }}} |----------------------- {{{#!td July 23 - July 29 \\ ''Week 11'' }}} {{{#!td - Fix bugs and documentation details. }}} {{{#!td }}} |----------------------- {{{#!td July 30 - August 5 \\ ''Week 12'' }}} {{{#!td - Prepare for final delivery. }}} {{{#!td }}} |----------------------- {{{#!td style="background: #ddd" August 6 - August 14 \\ ''Evaluation 3 (Final)'' }}} {{{#!td style="background: #ddd" - Wrap up my projects and submit the final evaluation of my mentors. }}} {{{#!td }}} \\ == Suggested timeline for final period of GSoC == |----------------------- {{{#!td July 16 - July 22 \\ ''Week 10'' }}} {{{#!td - Finalize parser library, fix bugs in XML description of moduls }}} {{{#!td }}} |----------------------- {{{#!td July 23 - July 29 \\ ''Week 11'' }}} {{{#!td - Simplify processing code for using a run method. }}} {{{#!td }}} |----------------------- {{{#!td July 30 - August 5 \\ ''Week 12'' }}} {{{#!td - Testing implementation, fix bugs }}} {{{#!td }}} |----------------------- {{{#!td style="background: #ddd" August 6 - August 14 \\ ''Evaluation 3 (Final)'' }}} {{{#!td style="background: #ddd" - Fix bugs, create documentation, prepare for final delivery. Wrap up my projects and submit the final evaluation of my mentors. }}} {{{#!td }}} == Evaluation 2 Status == Screenshots of current version of GUI from created description from parsing library. [[Image(https://ibb.co/fHHFHT)]] [[Image(https://ibb.co/iq4vHT)]] [[Image(https://ibb.co/it96P8)]] == Bonding period report == '''Introduce myself in dev list, get in contact with my mentors and discuss the design of functions'''\\ After being accepted as a student for GSoC 2018, I introduced my self in grass-dev lists on 24th of April (1). Also, I got in contact with my mentors - Marin Landa and Vaclav Petras. I discussed with them how to set-up my dev environment and ask how the functions should be designed. '''Prepare the wiki page about the project'''\\ I created my project wiki page (2) and added the link to the GSoC 2018 Accepted proposals page (3). The wiki page includes - General information about the project (title, mentors, links to the proposal and !GitHub repository, etc.), a brief description of the project, goal and timeline of tasks and deliverables. I will keep my wiki page constantly up to date and I’ll add weekly reports following the instructions in the GSoC Recommendations for Student page (4). '''Set up the !GitHub repository of the project'''\\ I chose !GitHub (5) as a public repository for the development of this project. I added the link to the GSoC 2018 Accepted proposals page (3), to my wiki page (2) and I shared it with my mentors. My repository is licensed under the GNU General Public License v3.0, according to the licence of GRASS GIS. '''Set up a developer environment'''\\ I checked and updated my developer environment so as to be ready to start coding after the bonding period. (1) https://lists.osgeo.org/pipermail/grass-dev/2018-April/088262.html \\ (2) https://trac.osgeo.org/grass/wiki/GSoC/2018/IntegrationInQGIS3 \\ (3) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_2018_Accepted \\ (4) https://wiki.osgeo.org/wiki/Google_Summer_of_Code_Recommendations_for_Students \\ (5) https://github.com/radeknovotny94/GRASSIntegrationInQGIS3 \\ == Weekly reports == == Week 1 == '''1) What did I complete this week?'''\\ I started work on implementing an initial version of Python library for generating UI files used by Processing plugin based on GRASS modules XML description. Unfortunately, I was not able to finish this task in time, because of I stuck in an automated generation of XML description from GRASS (1). So I have not enough sources for implementation and test of the parser (2). '''2) What am I going to achieve for next week?'''\\ I want to finish work from the Week 1 and also complete the goal of week 2 - Integration of generated description UI files from initial Python library to Processing plugin in order to generate GUI dialogue for GRASS modules. '''3) Is there any blocking issue?'''\\ I am blocked with generating XML descriptions from GRASS. (1) (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI/blob/master/generate_batch_for_xml.py \\ (2) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI/blob/master/GRASSDescribtionParser.py == Week 2 == '''What did you get done this week?'''\\ - Implementation of an initial version of the library for parsing description files for QGIS Processing plugin. I solved the problem with creating XML files from week 1. (1) - Create an independent version of Processing plugin for development. (2) - Try to create GUI Integration from generated description UI files from initial Python library to Processing plugin. '''What do you plan on doing next week?'''\\ - Final Python library for parsing descriptions. - Deep integration of UI descriptions to GUI Processing plugin. - Split GRASS module parameters in Processing plugin to basic and advanced tabs. '''Are you blocked on anything?'''\\ No, at the moment I’m not blocked. (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI\\ (2) https://github.com/radeknovotny94/GRASSIntegrationInQGIS3 == Week 3 == '''What did you get done this week?'''\\ - I was working on the improvement of the library for parsing description files for QGIS Processing plugin. (1) - I test creating of GUI from descriptions. '''What do you plan on doing next week?'''\\ - Final Python library for parsing descriptions. - Split GRASS module parameters in Processing plugin to basic and advanced tabs. '''Are you blocked on anything?'''\\ - I have some questions about different ways in QGIS and GRASS, I will discuss them with my mentors. (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI\\ == Week 4 == '''What did you get done this week?'''\\ - I finalized the library for parsing description files for QGIS Processing plugin. But of course I suppose, there will occur some bugs, after testing modules in QGIS Processing plugin. (1) - Add 'Controversy decision' in building parser to wiki. (2) '''What do you plan on doing next week?'''\\ - Split GRASS module parameters in Processing plugin to basic and advanced tabs - it should be required and optional tab. - Check GRASS X QGIS parameter descriptions in order to take care of differences in terminology. '''Are you blocked on anything?'''\\ - No, at the moment I’m not blocked. (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI\\ (2) https://trac.osgeo.org/grass/wiki/GSoC/2018/IntegrationInQGIS3 == Week 5 == '''What did you get done this week?'''\\ Unfortunately, I did not have a lot of time for the project this week, because I have exams at my university. - I improved the library for parsing description files for QGIS Processing plugin. (1) - Add all 'Controversy decision' in project and summary from hangouts to wiki. (2) '''What do you plan on doing next week?'''\\ - Split GRASS module parameters in Processing plugin to tabs - it should be on GUI Tabs. - Find all differences in terminology of QGIS and GRASS GIS. '''Are you blocked on anything?'''\\ - No, at the moment I’m not blocked. (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI\\ (2) https://trac.osgeo.org/grass/wiki/GSoC/2018/IntegrationInQGIS3 == Week 6 == '''What did you get done this week?'''\\ I had last week of my exams at my university, hence a progress in the project is not big. - I continued with bug fixing in parser library (1) - Split GRASS module parameters in Processing plugin to tabs was postponed. See (2) '''What do you plan on doing next week?'''\\ - Fix bugs in parsing library - Research how modules and parameters are divided at this moment and also check terminology. '''Are you blocked on anything?'''\\ - No, at the moment I’m not blocked. For more see wiki (3) (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI\\ (2) http://osgeo-org.1560.x6.nabble.com/Re-Improve-GRASS-GIS-integration-in-QGIS-3-Week-3-Report-tp5368107p5368128.html\\ (3) https://trac.osgeo.org/grass/wiki/GSoC/2018/IntegrationInQGIS3 == Week 7 == '''What did you get done this week?'''\\ - I continued with bug fixing in parser library (1) - I did research on how parameters are divided at this moment and also check terminology. '''What do you plan on doing next week?'''\\ - Fix bugs in parsing library - Resolve terminology differences '''Are you blocked on anything?'''\\ - No, at the moment I’m not blocked. For more see wiki (2) (1) https://github.com/radeknovotny94/GRASS_Parse_to_QGIS_UI/commit/0d9fc2f6d7d255e9e5d38a0b338c44b06942f852\\ (2) https://trac.osgeo.org/grass/wiki/GSoC/2018/IntegrationInQGIS3 == Week 8 == '''What did you get done this week?'''\\ - Unfortunately, I was all week on camp with my sports team, there was no electricity and the internet. So I was not able to work on the project, I hope I will be able to catch up with the delay next week. '''What do you plan on doing next week?'''\\ - Fix bugs in parsing library - Resolve terminology differences '''Are you blocked on anything?'''\\ - No, at the moment I’m not blocked. == Notes from discussions == === June 4: Hangout - Vaclav Petras, Radek Novotny === === How to parse questionable parameters? === ''' Raster 3D ''' If 'prompt' is 'raster_3d' - if this parameter is required - delete description file for QGIS Processing, because Processing does not have 3D Raster class. Could be resolve later. - if it is optional - pass this parameter - in both cases is give information about the parameter to log Examples: Not required parameter: r.colors Not recognized >>> raster_3d Required parameter: r.to.rast3 Need of 3D raster for module >>> r.to.rast3 ''' Group input/output ''' If 'prompt' is 'group': - only in module 'i.group' is parameter 'group' output - in all other modules should be 'group' input -if it is in 'r.' module, it is treated like raster multilayer -if it is in 'v.' module, it is treated like vector multilayer ''' What is treating as the string in QGIS Processing: ''' Separators - 'separator' and all names which have not equivalent class in Processing plugin - 'color', 'cats', 'dbname', 'dbtable'. ''' GRASS layer, sublayer: ''' GRASS GIS layer and sublayer is passed because it is no equivalent in QGIS. === June 11: Hangout - Vaclav Petras, Martin Landa, Radek Novotny === === How solve splitting modules in QGIS === r.slope.aspect in GRASS GIS vs. r.slope, r.aspect, ... in QGIS Processing plugin Reasons for splitting: * Users are used to modules/tools doing one single thing (slope only versus many other things) * Too many options Conclusion: More intuitive distribution of parameters in GRASS GIS (more in Future to do) === What is the best way how to divide parameters of modules in Processing plugin QGIS? === Discussed possibilities: - Required x Optional - This version was declined, because of a lot of important parameters are optional in GRASS modules. - Frequently used x Rarely used - This system was declined for now, because it is not clear what parameter is rarely or often used. But it could be a good point to think for future because this distribution has to be first done in GRASS GIS. - GUI Tabs - We decided for distribution according to the GUI sections in GRASS. It should be the least confusing system for users because it will be same in QGIS and GRASS. Also for future changes and maintenance, it appears like a good solution. === How solved differences in GRASS and QGIS terminology. === The biggest problem is that descriptions of parameters are different between GRASS and QGIS now and simultaneously parameter names are not shown in QGIS Processing plugin. So it is very difficult for a user to orientate in manuals. There are following variants: - what is possible to change in GRASS, so change it. E.g. 'raster map' in GRASS x 'raster layer' in QGIS - could be solved as 'raster input' or 'raster output' without the word map or layer. - what is not possible change in GRASS, then add a new tag with OGC (or QGIS) friendly description. This description could be shown in manuals too. Because that's a big deal and have to be thoroughly discussed in GRASS Mailing list. I will try to find all differences in QGIS and GRASS terminology and report them to ML as a source for a final decision. == Future to do: == - add keywords to Processing Plugin - for improvement of searching in modules - add similar keywords to ArcGIS tools to modules - for better orientation for ArcGIS users - show parameters in Processing Plugin - for better orientation in GRASS manual for QGIS users. - more intuitive distribution of parameters in GRASS - e.g. 'r.slope.aspect' could be divided into tabs slope and aspect, after this is not necessary to divide module in Processing Plugin to two modules - one for slope and second for aspect. Because it is intuitive to use just one tab if I want to compute slope.