GeoNetwork 2.6.x exhausts open files resources over time

[simonp]

Reported by Sylvain Grellet:

...

We are monitoring files opened by Geonetwork on our server. It appears that the number of files opened by Geonetwork grows steadily over time (see the 2 images attached). On "Geonetwork_opened_files.png" you'll see we had to reload Geonetwork 3 times (lsof goes down to zero).

We see some peaks when we edit metadata files (yellow peaks on Geonetwork_opened_files_third_peak_detail.png), so it seems there is some kind of garbage collection. But still the trend is not really nice.

...

Cheers Sylvain

[simonp]

• GN uses reference counting IndexReader as advised by Lucene In Action (2nd edition)
• IndexReader (LuceneSearcher or CatalogSearcher) gets stored in session so that user can return to page through results (this should be the same for CSW or user interface)
• When changes are made to metadata records, IndexWriter is used and Lucene index is rewritten to include changes
• Lucene is a file based index, rewriting of index deletes some files and creates new ones
• it seems that the deleted files are never closed, so number of open files used by the servlet gradually gets larger

Files left open that cause the eventual problem (see via ls /proc/<process_id>/fd or use lsof) have a path something like the following:

/usr/local/gn264/web/geonetwork/WEB-INF/lucene/nonspatial/_nnz.cfs (deleted)

[simonp]

Problem apparently disappears with Lucene 3.6.1 when our code is upgraded to use the Lucene SearcherManager and SearcherLifetimeManager. A benefit of using these is that we no longer need to cache the IndexSearcher/Reader in the user session to capture the snapshot of the index at search time. Instead we can cache a token and ask Lucene to give the IndexSearcher/Reader back when we need it. Also, old search sessions are pruned (SearcherLifetimeManager) - user gets 'search expired' exception if they leave a search session idle for too long (and the index changes) - this also saves resources.

[simonp]

Initial commit

​https://github.com/geonetwork/core-geonetwork/commit/8a2f3b5ed7ab33236c766c3861255a121889ff60

Some more testing is needed before this ticket can be closed. Stay tuned.

[jesseeichar]

I have a change ready for trunk as well. It was a bit complicated because on trunk there is one index per language when dealing with multilingual metadata which I do frequently. The meant a single search manager couldn't work.

You can take a look at:

​https://github.com/jesseeichar/core-geonetwork/compare/improvement/lucene-searchmanager

[simonp]

The patch for geonetwork 2.6.x in ​https://github.com/geonetwork/core-geonetwork/commit/8a2f3b5ed7ab33236c766c3861255a121889ff60 fixes this problem - see the discussion on ​http://osgeo-org.1560.n6.nabble.com/GN-2-6-4-Java-1-7-Too-many-open-files-error-tt4982753.html. Still testing Jesse's patch for master.

[simonp]

Fixed in commit 8a2f3b5ed7ab33236c766c3861255a121889ff60