Changes between Version 40 and Version 41 of GSoC/2018


Ignore:
Timestamp:
Mar 10, 2018, 2:07:56 PM (6 years ago)
Author:
wenzeslaus
Comment:

update with --exec and grass_session and more

Legend:

Unmodified
Added
Removed
Modified
  • GSoC/2018

    v40 v41  
    289289* Test of skills:
    290290
     291=== New easy-to-use CLI and API for GRASS GIS ===
     292
    291293=== New easy-to-use command line interface for GRASS GIS ===
    292294
    293  * GRASS GIS requires GRASS GIS Database, Location and Mapset to be set up to maintain data consistency, efficiency and security. Unfortunately, this is cumbersome when GRASS GIS is not the primary tools one is using. There are different workarounds for calling GRASS modules without starting GRASS explicitly, or running GRASS in a batch mode. However, none of these allows one to skip the database setup phase. This leads to the need for constant reimplementing of setup, import and export steps in various environments including user scripts (Bash, Python, R), QGIS Processing, gvSIG/SEXTANTE, uDig/JGrassTools and all the web/server/cloud tools which use GRASS GIS as a processing backend.
     295 * TL;DR: Make running of GRASS GIS modules as easy as it is to run GDAL commands.
     296  * No GRASS Database, Location, Mapset to deal with.
     297  * No import, export from user perspective.
     298  * Reasonable defaults for things like region.
     299  * CLI and API still allows user to specify any of the above.
     300 * GRASS GIS requires GRASS GIS Database, Location and Mapset to be set up to maintain data consistency, efficiency and security. Unfortunately, this is cumbersome when GRASS GIS is not the primary tools one is using.
     301 * There are different ways for calling GRASS modules without starting iterative GRASS session:
     302  * Modules executed with the `--exec` interface (see [https://grass.osgeo.org/grass74/manuals/grass7.html#batch-jobs-with-the-exec-interface the grass7 manual page] >=7.2)
     303  * `GRASS_BATCH_JOB`: same as newer `--exec` but through environmental variable and more limited
     304  * Use `grass.script.setup` package from GRASS GIS (requires boilerplate to add the packages on path first) 
     305  * Use the standalone `grass_session` package (new)
     306  * Set up environmental variables and "RC file" yourself (the classic method).
     307 * None of these allows one to skip the database setup phase. This leads to the need for constant reimplementing of setup, import and export steps in various environments including ''user scripts'' (in Bash, Python, R), QGIS Processing, gvSIG/SEXTANTE, uDig/JGrassTools, and all the web/server/cloud tools and applications which use GRASS GIS as a processing backend (e.g. PyWPS server).
    294308 * GRASS GIS itself can make it easier for the callers (at least in most cases) by implementing an interface which would allow to use GRASS GIS modules without explicit dealing with GRASS GIS database.
    295  * The command line call using the proposed interface would look like this:
     309 * The command line call using the proposed interface would look like these:
    296310
    297311{{{
     312grass run r.lake elevation=some/file.tiff water_level=10 lake=some/new/file.tiff coordinates=100,520
    298313grass run r.slope.aspect elevation=file://.../elevation.tiff aspect=file://...aspect.tiff
    299 grass run r.lake elevation=some/file.tiff water_level=10 lake=some/new/file.tiff coordinates=100,520
     314grass run r.slope.aspect elevation=https://.../elevation.tiff.zip aspect=file://...aspect.tiff
    300315}}}
    301316
    302317 * The `grass` command would have to parse the command line, find the files which should be maps (either using `file://` or perhaps anything would work) and import/export/link them, and then call the actual command.
    303  * The input maps could be linked (external) rather than imported (except for the cases when projection differs) which should be faster than import.
    304  * The output maps could be be also linked (external) with projection same as input which is should be faster then export.
    305  * Ideally this should work also with PostGIS and databases provided through GDAL/OGR.
    306  * The GRASS GIS Database, Location and Mapset should be created on the fly and deleted afterwards (the `.grassrc` wouldn't be used).
    307  * This project should also contain a [https://github.com/qgis/QGIS QGIS] Processing part.
    308   * Since QGIS Processing is one of the reasons for this interface, implementing is a crucial use case which should be tested from the very beginning a with QGIS Processing implementation, the interface would be used right away.
    309   * Alternatively, another project or more than one project can be in this part.
    310   * Title should change according to decisions made about this part.
    311  * Proposal should discuss how advanced things such raster algebra, multi-map inputs and outputs, temporal framework, cartography and visualization tools will work.
     318  * The input maps could be linked (external) rather than imported (except for the cases when projection differs) which should be faster than import.
     319  * Doing the work in GRASS rather than in the other software would allow GRASS to make the decision about the details, for example the data exchange (r.external vs r.import vs r.in.gdal - see [https://github.com/qgis/QGIS/pull/5426#issuecomment-345067457 comment from MarkusN for QGIS Processing issue] or [https://lists.osgeo.org/pipermail/grass-dev/2017-November/086598.html mailing list]).
     320 * GRASS Database would be created with an appropriate Location (projection based on input or additional CLI input).
     321  * The GRASS GIS Database, Location and Mapset should be created on the fly and deleted afterwards (the `.grassrc` wouldn't be used).
     322 * Computational region would be set based on input file(s) or additional CLI input.
     323 * The output maps could be be also linked (e.g. r.external.out) with projection same as input which is should be faster then export.
     324  * Ideally this should work also with PostGIS and databases provided through GDAL/OGR.
     325 * Proposal should discuss and address how advanced things such raster algebra, multi-map inputs and outputs, temporal framework, cartography and visualization tools will work.
    312326 * The system behind the interface will be inherently fragile, so it is necessary to write large amount of tests which would check different combinations of data types and projections.
    313  * Bonus task would be to find/implement a way to run Python code easily in case `grass` is called from Python (`grass python -c "print('...')"` is a good start).
    314  * It is expected that this would work for any `g.gui.*` modules too but implementing similar mechanism also for module dialogs is out of scope of this project (although some basic implementation might be quite straightforward).
    315  * Language requirements: Python (both GRASS GIS startup scripts and QGIS Processing are in Python)
     327 * All the underlying code is expected to be in Python, so the project should involve also creation of Python API on the way.
     328 * Bonus tasks:
     329  * Making this work for the GUI in the same way. It is expected that this would work for any `g.gui.*` modules too but implementing similar mechanism also for module dialogs is more work (but some basic implementation might be quite straightforward).
     330  * Making this connected to the standalone `grass_session` package.
     331  * Generalization of the API, so that it incorporates also the concept of remote sessions (see e.g. [https://github.com/wenzeslaus/g.remote g.remote on GitHub])
     332 * Current GRASS code involved:
     333  * source:grass/trunk/lib/python/script/setup.py (library function(s) for Location setup in Python)
     334  * source:grass/trunk/lib/init/grass.py (full GRASS GIS standard startup "script")
    316335 * See also:
    317336  * #2579 Specify command to be executed as parameter of grass command (patch) and other suggestion for CLI
     
    323342  * [https://github.com/qgis/QGIS/blob/0a1382a0be36d408aebd227fb0066f68c513e41e/python/plugins/processing/algs/grass7/Grass7Algorithm.py QGIS Processing Grass7Algorithm.py] source code
    324343  * [https://github.com/moovida/jgrasstools/blob/530c87f26d220f3eeff9d2fb9d21abd8821c00c3/grass/src/main/java/org/jgrasstools/grass/utils/ModuleSupporter.java uDig's JGrasstools] source code
    325   * [http://sextante.googlecode.com/svn/trunk/soft/sextante_lib/sextante_gui/src/es/unex/sextante/gui/grass/ SEXTANTE grass] source code
    326   * [https://github.com/geopython/PyWPS/blob/425f0eb160f60714a6705a24ba926e03690ab371/pywps/Grass.py PyWPS source code] and [http://pywps.wald.intevation.org/documentation/process.html PyWPS documentation]
    327   * [https://code.google.com/p/wps-grass-bridge/source/browse/trunk/?r=97 wps-grass-bridge] source code
     344  * [http://sextante.googlecode.com/svn/trunk/soft/sextante_lib/sextante_gui/src/es/unex/sextante/gui/grass/ SEXTANTE grass] source code (broken link)
     345  * [https://github.com/geopython/PyWPS/blob/425f0eb160f60714a6705a24ba926e03690ab371/pywps/Grass.py PyWPS source code] and [http://pywps.wald.intevation.org/documentation/process.html PyWPS documentation] (broken link)
     346  * [https://code.google.com/p/wps-grass-bridge/source/browse/trunk/?r=97 wps-grass-bridge] source code (broken link)
    328347  * Related discussions on mailing list:
    329348   * [http://lists.osgeo.org/pipermail/grass-dev/2015-February/073867.html On GSoC Proposal New easy-to-use...]
    330349   * [http://lists.osgeo.org/pipermail/grass-dev/2015-March/074433.html GSOC project proposal]
     350 * Language requirements: Python
    331351 * Mentor for the GRASS GIS part: Vaclav Petras
    332352 * Mentor for the QGIS part: ?