Changes between Version 2 and Version 3 of PerformanceEnhancements

Timestamp:

Mar 5, 2010, 7:25:43 AM (14 years ago)

Author:

simonp

Comment:

--

PerformanceEnhancements

@@ -35 +35 @@
 == Proposal == 
 
-Addressing the search speed issue: Investigation found that the lucene component of searches in GeoNetwork is in fact very fast. The delay in returning search results came from processing of all results to gather the most frequently used keywords for the search summary. Since all hits were processed no matter how large the result set, large result sets could cause a big delay in the first page of search results getting back to the user. A simple fix is to limit the number of hits that are processed to build the keyword frequency info for the search summary. This parameter is specified on search services as maxRecordsInKeywordSummary and has been set to 1000, the LuceneSearcher code then limits the number of hits it examines to build the keyword frequency info. 1000 is an arbitrary number that sites can change according to the number of keywords used in their metadata records and the time delay that is considered acceptable for search results to be returned to the user.
+Addressing the search speed issue: Investigation found that the lucene component of searches in !GeoNetwork is in fact very fast. The delay in returning search results came from processing of all results to gather the most frequently used keywords for the search summary. Since all hits were processed no matter how large the result set, large result sets could cause a big delay in the first page of search results getting back to the user. A simple fix is to limit the number of hits that are processed to build the keyword frequency info for the search summary. This parameter is specified on search services as maxRecordsInKeywordSummary and has been set to 1000, the LuceneSearcher code then limits the number of hits it examines to build the keyword frequency info. 1000 is an arbitrary number that sites can change according to the number of keywords used in their metadata records and the time delay that is considered acceptable for search results to be returned to the user.
 
 Addressing the indexing/loading issue: Investigation of this problem led to two issues and two solutions:
 
- * Speeding up Lucene indexing: GeoNetwork was opening and closing the Lucene IndexWriter every time it wrote a document to the search index. This is a very safe way to handle the Lucene Index Writer as only one Index Writer can be open. However the IndexWriter class in Lucene is now much more sophisticated than it was when GeoNetwork was first written. In particular, it is thread safe and can buffer documents in RAM before writing them out to the index file on disk with its own thread. To use the IndexWriter in this way without forcing major changes in the GeoNetwork code, resulted in an IndexWriter facade that allows:
+ * Speeding up Lucene indexing: !GeoNetwork was opening and closing the Lucene IndexWriter every time it wrote a document to the search index. This is a very safe way to handle the Lucene Index Writer as only one Index Writer can be open. However the IndexWriter class in Lucene is now much more sophisticated than it was when GeoNetwork was first written. In particular, it is thread safe and can buffer documents in RAM before writing them out to the index file on disk with its own thread. To use the IndexWriter in this way without forcing major changes in the !GeoNetwork code, resulted in an IndexWriter facade that allows:
 - code that intends to write a number of documents to the Lucene Index to keep the IndexWriter open, and thus take advantage of the more sophisticated IndexWriter implementation
 - multiple threads to schedule documents that need to be written to the index without blocking