GSoC 2014 Metadata for GRASS GIS maps and datasets

Title:	Metadata for GRASS GIS maps and datasets
Student Name:	Matej Krejci, ​Czech Technical University in Prague

Organization:

​OSGeo - Open Source Geospatial Foundation

Mentor Name:

​Martin Landa, ​Margherita Di Leo

GSoC proposal:

​view proposal

Link to code:

​https://trac.osgeo.org/grass/browser/sandbox/krejcmat

Motivation

Current GRASS GIS offers just a poor and out-of-date administration of metadata. The goal of this project is to implement administration of metadata according to the standards of ISO 19115, ISO 19139 and to create a graphical user interface which would allow the user to operate with metadata, such as creating, updating, browsing, searching etc.

Timeline

timeline from proposal		newly proposed timeline (1)	status (1)	after mid-term proposed timeline (2)	status (2)
MAY 1- 7	"Early starts" Study ISO, INSPIRE and FGDC standards	+
MAY 7 - 14	Getting familiar with GRASS GIS modules code.	+
MAY 14 - 19	Design of metadata GRASS library	design GRASS library (in Python) to read / write metadata based on ISO as flat XML files	+
MAY 19	Starting GoSC
MAY 19- 25	Support of different metadata profiles	design GRASS library (in Python) to read / write metadata based on ISO as flat XML files including profies (XSD validation, Inspire as the show-case)	+
MAY 26 - JUN 16	"Break"(Prepare for bachelor final exam)	add support for conversion GRASS (current) metadata files <-> XML ISO metadata	+
JUN 16 - 24	Implement basic library model for command line	design cmd modules for reading and writing ISO-based metadata -> r.info.iso + v.info.iso (including support for conversion GRASS (current) metadata files <-> XML ISO metadata)	+
JUN 24 - 30	Modules functionality (searching, publishing metadata)	design GUI for reading and writing ISO-based metadata		design source of information from jinja	+
JULY 1 - 7	Design GUI	design GUI for reading and writing ISO-based metadata -> g.gui.metadata	g.gui.metadata version 1.0		+
JULY 7 - 14	Connect GUI library, basic functions	advanced metadata publishing/searching based on owslib + pycsw	g.gui.metadata version 1.1		+
JULY 14 - 21	Implementation of search function to GUI	advanced metadata publishing/searching based on owslib + pycsw	g.gui.metadata version 1.2
JULY 21 - 30	Improve metadata validation using designed GUI	advanced metadata publishing/searching based on owslib + pycsw -> testing prototype	g.gui.metadata 1.3&1.4., polishing and testing g.gui.metadata
AUGUST 1 - 9	Polishing code	polish code and advanced metadata publishing/searching based on owslib + pycsw	designing of metadata catalog browser
AUGUST 9 - 18	Testing and documentation
AUGUST 18	The end of GSoC

Development

Dependencies

Current development of metadata management has dependence on external libraries:

• OWSLib (no additional dependence) development version ()

  git clone git://github.com/geopython/OWSLib.git
  cd OWSLib && sudo python setup.py install
  

• Jinja templates (suitable to remove this dependence)

  git clone git://github.com/mitsuhiko/jinja2.git
  cd jinja2 && sudo python setup.py install
  

Architecture

The new designed metadata management in GRASS is based on external python modules without processing of the GRASS core.

• First step which is partly done is to create basic modules for exporting ISO based metadata for raster and vector maps. The modules are intended to implement class that is providing parsing metadata from r.info and v.info. These metadata are assigned to ISO attributes with using OWSLib. The OWSLib is offering creation instances of ISO elements. The instances are suitable to filled predefined of Jinja xml ISO profiles. From this part of project r.info.iso and v.info.iso are created.

r.info.iso and v.info.iso

• These modules are based on the mentioned implementation.
• Offers converting metadata from *.info to xml file based on ISO standard.
• Currently available metadata profile for GRASS:
  • INSPIRE profile,
  • GRASS BASIC profile - this profile is mainly including available metadata from GRASS.

g.gui.metadata

Current task is to design and the implementation of the GUI interface. Below is the first design provided. Main aggravating factor is, that some attributes in ISO can be defined multiple times. For the solution, there is necessary to create interface for getting information from some source. Currently the easiest way is to get information forom predefined XML Jinja template. Interface ensures independence between editor and source of information. This is useful for future development steps. The editor will support:

version	function
1.0	to implement main window and layout
	function for browsing maps with using treeCtrl
	to create xml ISO file (with using v/r.info.iso)
	interface to get information about ISO attributes
1.1	editing of created xml files with using Jinja templates
1.2	to add multiple values for specific attributes
1.3	validator of inspire xml
	options for update metadata with using r/v.support by changed values in editor.
1.4	multiple metadata editing ( choosing maps invokes process that changes selected attributes of all chosen maps.

scheme of designed implementation

draft of GUI for choosing ISO profile or options edit xml only

draft of GUI editor

How to test

cd ~/bin/
svn checkout http://svn.osgeo.org/grass/sandbox/krejcmat/src/ md-iso
echo "export GRASS_ADDON_PATH=$HOME/bin/md-iso" >> $HOME/.grass7/bashrc

grassXY

GRASS> r.info.iso.py elevation

GRASS> g.gui.metadata.py

Weekly reports

Week 1 (May 23)

studied metadata and tried to find suitable python lib during last three weeks:

• studied ISO 19115 and EN ISO 19119
  • INSPIRE
  • be advised to leave out implementation of FGDC - just ISO, INSPIRE
• get familiar with OWSLib and pycsw
  • set pycsw metadata server catalog
  • handled metadata by OWSLib
• spent time with design lib based on automated generation of XML from XSD - failed
• tested suitability of generateDS to generate python object from XML scheme (XSD)
• designed library for parse GRASS GIS metadata to XML based on md. ISO
  • parsed metadata from:
    • r.info
    • v.info
• designed essential metadata profile for raster and vector maps (based on ISO standard)
  • prepared lib for new update of OWSLib that will be ready during next week. (in contact with author)
• created new timeline - main changes:
  • will create basic metadata management based on creating, editing, validating ISO. (obligatory)
  • will create advanced management based on client-server with using pycsw. (optional)

​report email

Week 2 (May 30)

created cmd modules for writing ISO-based metadata -> r.info.iso + v.info.iso support:

• basic grass ISO profile
• inspire profile (for subsequent editing in future wxGUI editor)

​report email

Week 3 (June 06)

• Consulted design of metadata editor with mentors.
• Consulted current implementation of v.info.iso and r.info.iso
• Search for alternatives to Jinja templates (current implementation)
• Learned basics of wxPython

​report email

Week 4 (June 13)

​report email

Week 5 (June 22)

Redesigned GUI of editor with inspiration from ​INSPIRE Editor - The editor will include:

• on the top (fixedly) = toolbar: create new, save, validate, etc (new)
• panels based on MultiSplitterWindow
  • on the left side = panel with Grass map browser (proposed)
  • on the right side ='notebook' with two items: advanced browser/editor and html help (new)
  • in the middle = Main editor of metadata

Implemented part of GUI editor:

• part of the GUI (main frame and layout) based on MultiSplitterWindow
• GRASS maps browser in the left panel
• in right panel implemented the "notebook" with metadata browser and editor (necessary to tune up the view of metadata).
• class with icon based toolbox

​Report email

Week 6 (June 29)

Worked on GUI editor

• implemented generator of metadata items (control text, label, button) in main editor
• fixed some bugs in g.gui.metadata
• implemented interface for parse information from jinja template
• added additional information about metadata to jinja template.

​Report email

Week 7 (July 4)

• designed and implemented function that allows generating GUI editor from xml loaded by OWSLib with using objects and loops from Jinja template.

​Report email

Week 8 (July 11)

• re-implemented generator of GUI to more object oriented design
• implemented main part of exporting edited metadata to XML
• started with dynamically adding wx widgets to frame (multiplicity of metadata items)
• changed timeline

​Report email

Week 9 (July 18)

• implemented 'multiplicity button' for dynamically adding widgets and also removing them
• support of exporting metadata from dynamically generated GUI(multiplicity button) (need more working on allow exporting keywords to xml(multiple keywords))
• connected finished modules together(g.gui.metadata)
  • validator
  • tree editor
  • main editor
  • toolbox

Week 10 (July 25)

​Report email

Week 11 (August 1)

The yellow text inputs marking the values which are not possible to fill by information from r.info and v.info. On the picture below is the session of editor which allows to mark metadata items for predefining template and save it for next times. This function is well fit to editing multiple maps with batch exporting. Red colors marking the metadata which are obligatory for chosen jinja template but the loaded xml(ISO metadata) doesn't include them.

The editor allows editing maps from GRASS which GRASS BASIC profile and INSPIRE profile. Also allows to initialize editor with custom templates which will be filled as much as possible by intersection with metadata in selected maps (intersection of inspire template and own template). Of course editor can be initialize just with template and xml without any connection to GRASS mapset.

TODO&IDEAS

• Stored predefined metadata values in SQLite i.g.: keyword lists, points of contacts, and so on.​mailing list