Changes between Initial Version and Version 1 of Toolboxes


Ignore:
Timestamp:
Apr 30, 2013, 6:58:14 AM (10 years ago)
Author:
wenzeslaus
Comment:

Moving the page Toolboxes from GRASS Wiki (http://grasswiki.osgeo.org/grass-wiki/index.php?title=Toolboxes&oldid=18005), small changes, merged with talk, long tables excluded, nothing wxGUI toolboxes

Legend:

Unmodified
Added
Removed
Modified
  • Toolboxes

    v1 v1  
     1[[TOC]]
     2
     3= Toolboxes =
     4
     5== wxGUI toolboxes ==
     6
     7''Main article: wiki:wxGUIDevelopment/Toolboxes''
     8
     9Toolboxes as GUI-only concept.
     10
     11=== Toolbox approach discussed at the GRASS Community Code Sprint 2012 ===
     12
     13At the [http://grasswiki.osgeo.org/wiki/GRASS_Community_Sprint_Prague_2012|community sprint in Prague 2012] the toolbox approach with different repositories as well as the approach using `"--with-toolbox/--without-toolbox"` configure statements at compile time have been discussed. The major drawback of these solutions is the their complexity and the problem of cross dependencies between modules.
     14
     15A new approach has been suggested using XML files to describe toolboxes and special directory structure to hide unneeded modules from the user.
     16
     17The idea is to have a binary directory containing only modules that are listed in a XML file that defines what core modules should be available and what toolboxes should be used. The XML file will be used by the GUI to create menu listings for the selected modules. All modules that are deactivated in the XML file will be held in a directory that is not available in the users binary path, but in the binary search path of modules. Hence unneeded modules are invisible at the command line and in the GUI, but still available for modules that might call other modules.
     18
     19New toolboxes and core modules can be simply added to the search path and the GUI by editing the XML definition and calling g.extension. The module g.extension will simply links/copy the hidden modules into the binary directory that is visible by the user or move deselected modules in the hidden directory. The GUI will always analyzes the XML toolbox configuration file for menu generation.
     20
     21The configuration XML file will include a list of core modules for general, raster, image, voxel and vector processing as well as the definition of toolboxes. Toolboxes are defined by their type (raster, vector, ...) a short description and a list of modules. Modules can be defined in several different toolboxes.
     22
     23
     24The GRASS development team should provide several pre-defined XML toolbox configuration files as for hydrological modeling, vector processing, terrain modeling, image analysis, ..., and WPS server.
     25Using this approach the repository structure must not be changed. The build system has to be modified to use the XML toolbox file to decide where to copy the binary modules and scripts. `g.extension` has to be extended to link/copy selected modules in the users binary search path and to remove modules from them. GRASS will be build as usual.
     26
     27=== Original approach ===
     28
     29
     30 * See [http://lists.osgeo.org/pipermail/grass-dev/2010-April/050210.html original post by Jarosław Jasiewicz]
     31
     32{{{
     33Hi all!
     34
     35That rather radical ideas I present here are rather for future, at least for GRASS 8,
     36but I'd like present it now for long-term reflection.
     37
     38Probably all notice that for over two years there is big increase in add-on repository
     39(including me). There are modules of different quality: from fully GRASS toolsets,
     40to shell or python scripts, from  actively developed tools to abandoned,
     41from all-purpose tools to very specialized etc. I also think that that activity
     42will be grown due to substitute shell script by python
     43
     44Similar situation is in main GRASS branch: there are modules for all like conversion tools,
     45interpolation methods, georeferencing etc, and very specialized modules for very limited
     46group of users (like wild fire), there are also some modules out of date.
     47
     48I'm not enthusiastic about moving new modules into main branch. Almost every module has
     49different coding style and it will lasting in future that GRASS would be difficult to maintain.
     50On the other hand some people complains that some interesting modules are only available as
     51add-ons (I assume for some reasons they cannot install it)
     52
     53So my suggestion is to rearrange future GRASS form two layers (main branch/add-on) into
     54three layers architecture:
     55
     561) GRASS core layer: much limited limited than now, only GIS environment and basic,
     57all-puropse tools, slow changes, great stability
     582) GRASS toolset layer: oficcial GRASS thematic tools and toolsets (like terrain analysis,
     59hydrological analysis, photo-interpretation, landscape analysis etc,) every toolset with its
     60maintainer, rapid development, new ready to use tools after quality control may appear here,
     61also some of current main branch tool shall be moved to that layer
     623) GRASS community layer:  everything else like experimental, actively development new tools,
     63that what do not pass quality control, simple scripts, etc....
     64
     65What benefits:
     66for developers and contributors: much clear situation and better publication path.
     67Toolset layer should be much more open for new tools than current GRASS main branch
     68
     69for users: faster access to new tools.
     70There is no doubt that new tools are faster developed (less risk) than GRASS core
     71Binaries with toolsets could be maintained as separate apt/urpmi/pacman/yum/exe etc packages,
     72so it may appear in linux repository separetly form GRASS core.
     73
     74There is only loose ideas. Most of them are of course taken from R (core/toolsets/rest of packages;
     75separate core and package development) but I think it is worth of some discuss ...
     76
     77regards
     78Jarek
     79}}}
     80
     81Criticism from the GRASS Wiki [http://grasswiki.osgeo.org/grass-wiki/index.php?title=Talk:Toolboxes&oldid=14131 Tooboxes talk page]:
     82 * One of our best selling points vs. the competition is that you don't have to buy expensive addon toolboxes to have it do what you want to do.
     83 * It makes it a lot harder for new users to get started with what they want to do. Even when done well it's a brittle system dependent on 100% uptime servers etc. which in practice do not exist.
     84  * Comment: I don't see any problem here. We can distribute several packages of GRASS. It will give us just more flexibility. One of the packages could contain all (or all fundamental) toolboxes already installed. Basically we could publish two packages - 'minimal' with only core modules installed to give user freedom to decide (install) which toolboxes/modules he/she wants and 'standard' with all major toolboxes included.
     85 * Non-"core" modules will be neglected by the core devs and die from bit rot. (outside of grep's reach)
     86 * Those "non-core" modules have personally led me into all new ideas and directions outside of my normal field of study, which has rather positively affected the direction of my career and let me solve problems in novel ways for my peers that only cross-discipline tools/perspective could introduce us to.
     87 * Our download size is only about 25ish megs. that's tiny. Docs are bigger than code. Windows deps "aren't our fault" and switching to a different distribution model won't help that much at all.
     88 * Rather than focus development I fear it will dilute it. Divided we fall..
     89 * Big change is big work which could more productively be funneled into more critical pursuits. (I am not against needed change, but very against change-for-change's-sake.)
     90
     91{{{
     92#!comment
     93
     94==== Proposal to change repository layout ====
     95
     96===== Current repository layout =====
     97
     98- grass
     99|- trunk
     100 |- raster/
     101 |- vector/
     102 |- ...
     103
     104- grass-addons
     105|- raster/
     106|- vector/
     107|- ...
     108
     109
     110===== Proposed repository layout =====
     111
     112- grass
     113|- trunk
     114 |- raster/
     115 |- vector/
     116 |- ...
     117
     118- grass-tools
     119|- trunk
     120 |- raster/
     121 |- vector/
     122 |- ...
     123
     124- grass-addons
     125|- raster/
     126|- vector/
     127|- ...
     128
     129}}}
     130
     131== Proposed toolboxes ==
     132
     133'''Toolbox''' is a topic-based set of modules used for solving common GIS analysis, like hydrology, networking, LRS, etc.
     134
     135=== Managing toolboxes ===
     136
     137''This section describes how toolboxes in addons currently works. This can be part of the documentation. However, it this might be changed with the wxGUI toolboxes.''
     138
     139`g.extension` is designed to manage locally installed GRASS [source:grass-addons/grass7 Addons]. It allows to install single addons modules and also toolboxes (flag `-t`). Note that toolbox support is '''experimental'''.
     140
     141 * list available toolboxes
     142
     143{{{
     144g.extension -tl
     145}}}
     146
     147{{{
     148List of available extensions (toolboxes):
     149Hydrology (HY)
     150 * r.stream.basins
     151 * r.stream.channel
     152 * r.stream.distance
     153 * r.stream.extract
     154 * r.stream.order
     155 * r.stream.segment
     156 * r.stream.slope
     157 * r.stream.snap
     158 * r.stream.stats
     159...
     160}}}
     161
     162 * list available modules from [source:grass-addons/grass7 Addons]
     163
     164{{{
     165g.extension -l
     166}}}
     167
     168{{{
     169List of available extensions (modules):
     170...
     171r.stream.slope
     172...
     173}}}
     174
     175 * install new toolbox ("Hydrology")
     176
     177{{{
     178g.extension -t extension=HY
     179}}}
     180
     181 * install single module from "Hydrology" toolbox
     182
     183{{{
     184g.extension extension=r.stream.slope
     185}}}
     186
     187Module `g.extension` also allows to list installed extensions
     188
     189 * toolboxes
     190
     191{{{
     192g.extension -ta
     193}}}
     194
     195{{{
     196List of installed extensions (toolboxes):
     197HY
     198}}}
     199
     200 * or modules
     201
     202{{{
     203g.extension -a
     204}}}
     205
     206{{{
     207List of installed extensions (modules):
     208r.stream.basins
     209r.stream.channel
     210r.stream.distance
     211r.stream.extract
     212r.stream.order
     213r.stream.segment
     214r.stream.slope
     215r.stream.snap
     216r.stream.stats
     217}}}
     218
     219 * remove locally installed toolbox
     220
     221{{{
     222g.extension -ft extension=HY operation=remove
     223}}}
     224
     225 * remove single module
     226
     227{{{
     228g.extension -f extension=r.stream.slope operation=remove
     229}}}
     230
     231
     232=== List of toolboxes ===
     233
     234|| '''Toolbox name''' || '''Toolbox code''' || '''Description''' || '''Correlate toolbox''' ||
     235|| 3D Raster ||  R3 ||  All `r3.*` modules ||  ||
     236|| 3D Visualization ||  V3 || nviz_cmd ||  ||
     237|| Atmospheric tools ||  AT ||  Subset of image processing tool ||  ||
     238|| Cartography ||  CA || ps.map, ps.output ||  ||
     239|| Database ||  DB ||  All `db.*` modules || Vector analysis ||
     240|| Geostatistcal ||  GS || v.krige maybe other ||  ||
     241|| GUI ||  GI || wxGUI ||  ||
     242|| Hydrology ||  HY || r.watershed, r.terraflow, `r.stream.*` modules in addons, maybe other, r.flow, r.sim.water, r.sim.sediment || Raster analysis ||
     243|| Image Processing ||  IM ||  All `i.*` modules, selected r.* modules ||  ||
     244|| Interoperability tools ||  IO ||  All the various import/export modules except for GDAL/OGR || Raster analysis, Vector analysis, Image Processing, Database ||
     245|| Landscape analysis ||  LA ||  All `r.li.*` modules, maybe other || Raster analysis ||
     246|| Lidar analysis ||  LI ||  All `v.lidar.*` modules, v.surf.bspline, r.in.xyz ||  ||
     247|| Linear Reference Systems ||  LS ||  All `v.lrs.*` modules || Vector analysis ||
     248|| Network analysis ||  NA ||  All `v.net.*` modules, maybe other || Vector analysis ||
     249|| Raster analysis ||  RA ||  A lot of basic `r.*` modules (like r.buffer, r.reclass, r.fillnull...) ||  ||
     250|| Terrain analysis ||  TA ||  Different kind of analysis: soil (r.sun,), terrain (r.slope.aspect, r.los), geomorphometry (r.param.scale) || Raster analysis ||
     251|| Vector analysis ||  VA ||  A lot of basic `v.*` modules (like v.buffer, v.centroid, v.overlay...) || Database ||
     252
     253
     254== How to implement ==
     255
     256=== Proposal 1 ===
     257 * add html tag <param> in the description.html with these options:
     258{{{
     259<param name = "core"> for core module
     260<param name = "tool" value="tool_code/codes"> for toolbox
     261<param name = "addons" [value="tool_code/codes"]> for addons, value is optional
     262}}}
     263
     264 * create a script to generate a xml file for each toolbox
     265 * add toolbox option in g.extension to read the xml and install the modules
     266
     267=== Proposal 2 ===
     268
     269 * modules (currently only [source:grass-addons/grass7 Addons]) are described (name, description, keywords, files) in metadata XML file called [http://grass.osgeo.org/addons/grass7/modules.xml modules.xml] - this file is auto-generated, see [http://trac.osgeo.org/grass/wiki/AddOnsManagement#AddOnsXMLfile trac page] for details
     270 * similarly toolboxes are described (name, code, correlate modules, list of modules) in [http://grass.osgeo.org/addons/grass7/toolboxes.xml toolboxes.xml] - this file is manually manageable via [https://svn.osgeo.org/grass/grass-addons/grass7/toolboxes.xml SVN]
     271
     272`g.extension` reads (from remote server http://grass.osgeo.org/addons/grass7) `modules.xml` when listing available modules (`-l`), and `toolboxes.xml` when listing available toolboxes (`-lt`). When installing toolbox using `g.extension` the module reads `toolboxes.xml` and installs in the loop given list of modules. When all modules are installed it also updates local metadata files - `GRASS_ADDON_BASE/modules.xml` and `GRASS_ADDON_BASE/toolboxes.xml`. These local metadata files are checked when listing locally installed toolboxes or single modules (`-a` or `-ta`).
     273
     274=== Proposal 3 ===
     275
     276Use a modified version of [http://pypi.python.org/pypi/pip pip] (a tool for installing and managing Python packages)
     277
     278With Pip you can add/change repository, compile C/Fortran module, manage dependencies, install from some version control system (use
     279URLs like hg+http://domain/repo -- or prefix git+, svn+ etc), and using [http://pypi.python.org/pypi/virtualenv virtualenv]