Fast Index Update
Date | 2012/09/14 |
Contact(s) | Jesse Eichar |
Last edited | |
Status | Not proposed. Just brain storm |
Assigned to release | 2.9.x |
Resources | Swisstopo |
Code | https://github.com/jesseeichar/core-geonetwork/tree/improvement/fastupdate |
Overview
Currently in order to change any fields in the lucene index the metadata document has to be completely reindexed. This proposal outlines how certain field changes can be efficiently performed.
The types of actions this would make efficient are:
- Update popularity
- Update privileges
Proposal Type
- Type: Improvement
- App: Geonetwork
- Module: Index
Links
- Email discussions:
- IRC discussions:
- Related work:
Voting History
- None as yet
Proposal
Background
First of all lucene does not support updating a document directly. The normal way of updating a lucene document is to:
- Recreate the lucene document from the source data (in our case the metadata xml)
- delete the document from the index
- add new (recreated) document to the index
If the recreation of the lucene document is an expensive operation then performing this task is naturally expensive. There are work arounds for example:
- Load document from index
- Modify document
- delete document from index
- insert modified document to index
However this is a naive solution and fails in practice because there are both stored and non-stored fields. Using this strategy will lose the non-stored fields.
A potential solution comes to mind:
Lucene 4 has the concept of flexible indexing (http://java.dzone.com/videos/heavy-committing-flexible)
Backwards Compatibility Issues
Will require rebuild of index
Risks
Participants
- As above