Opened 9 years ago
Closed 7 years ago
#1522 closed defect (wontfix)
GeoNetwork 3.0.3 fails to start in 32bit system
Reported by: | kalxas | Owned by: | |
---|---|---|---|
Priority: | critical | Milestone: | OSGeoLive11.0 |
Component: | OSGeoLive | Keywords: | geonetwork 3 |
Cc: | live-demo@…, simonp |
Description
While testing the latest nightly build, I found that GeoNetwork is consuming a lot of RAM and swaps out both my test VMs (both 32 and 64 bit, running the live system) even with 4GB RAM allocated.
This seems to be a Jetty/Java issue.
Attachments (2)
Change History (39)
comment:1 by , 9 years ago
comment:2 by , 9 years ago
VirtualBox VM: revB1 i686, 826GB of RAM, four cores no changes to -Xmx - "Unable to Connect" in Firefox, even after four minutes
comment:3 by , 9 years ago
VM: revB1 i686, 1024GB of RAM, four cores no changes to -Xmx
Start GeoNetwork; all cores active, RAM allocated past all available to 480GB swap used; before the launcher progress bar is done, activity ceases; "Unable to Connect" even after three minutes
comment:4 by , 9 years ago
VM: revB1 i686, 1824GB of RAM, four cores no changes to -Xmx
Start GeoNetwork?; all cores active, RAM allocated all available to some small swap used; before the launcher progress bar is done, activity ceases; "Unable to Connect"
ps aux | grep java
shows no java processes running
comment:5 by , 9 years ago
The java process dies when it runs out of RAM after 5 mins or so. I am using the live system on VirtualBox, not an installed system.
comment:6 by , 9 years ago
I have prepared a branch to revert GeoNetwork back to 2.10 in case no solution is found for 3.0.1
comment:7 by , 9 years ago
Priority: | blocker → critical |
---|
GeoNetwork 2.10 works fine in build 158 so this is not a blocker anymore...
comment:10 by , 9 years ago
New builds are up for testing:
Build 162 with GeoNetwork 3.0.1 (including fix from pull request):
http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-build162-amd64-53002ef.iso
Build 161 with GeoNetwork 2.10 (fallback):
http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-build161-amd64-b086b2a.iso
I tested a bit build 162 and I could not get GeoNetwork 3 to start on my VM.
Best, Angelos
comment:11 by , 9 years ago
Thanks. Build 162 starts for me without installing the VM (but it was also with previous build).
But then I've another error preventing part of the application to work properly:
2015-08-27 13:22:23.200:WARN:oejs.ServletHandler:Error for /geonetwork/images/logos/10e9ce99-b51f-47b0-a237-4dd749a2f4e4.png java.lang.NoClassDefFoundError: Could not initialize class org.fao.geonet.resources.Resources at org.fao.geonet.resources.ResourceFilter$Instance.<init>(ResourceFilter.java:65)
Could this be related to specific ImageIO setup on the osgeolive ?
comment:12 by , 9 years ago
So, I understand that providing the database binary now works fine (does not need to be created on the first startup anymore), right?
Regarding ImageIO, we have not changed anything from the default Java 7 on Ubuntu. Perhaps a permissions problem?
comment:13 by , 9 years ago
So, I understand that providing the database binary now works fine
After further testing of last build with different VM configuration, I would say:
- use GN3 on a virtual box with at least 3+Go RAM (as it looks like startup issues are mainly related to memory troubles) - 1Go does not start, 2Go sometimes starts.
- on first startup, it still create the database and load default data so it takes some times.
- sometimes (not sure exactly when and why) it refresh the JS cache and needs memory for that
- ImageIO, it looks related to JDK version and the imageio version used (first time this issue is observed AFAIK). Removing imageio from the lib folder fix the issue:
/usr/local/lib/geonetwork/bin$ rm ../web/geonetwork/WEB-INF/lib/jai_imageio-1.1.jar
Once you have an application running with a default db and a JS cache populated, startup and using the application is fast.
comment:14 by , 9 years ago
Milestone: | OSGeoLive9.0 → OSGeoLive9.5 |
---|
comment:15 by , 9 years ago
Priority: | critical → blocker |
---|
Hi,
I tested the GeoNetwork 3.0.1 setup again in latest 9.5 nightly build (live system under Virtual Box):
- I removed the jai_imageio-1.1.jar library manually as suggested.
- With 2GB RAM the VM goes to swap and finally dies...
- With 4GB RAM it takes ~6 mins to finally start up. In the meanwhile the Firefox window shows up with an "page not found" error.
- After the application starts it seems that there are no data loaded in the application. I guess this is not expected, right?
I am very concerned that a simple user will not be patient enough to wait for 6 mins for the application to start (even if we update the Quickstart to mention this). Is there any way we can create the database with a script during the installation process?
Best, Angelos
comment:16 by , 9 years ago
Update: After the first startup, restarting GeoNetwork takes only 2 mins. The sample data are easy to find and load from the UI. We need to revert the OSGeoLive-doc commits to the 3.0.1 version...
comment:18 by , 9 years ago
Keywords: | geonetwork 3 added |
---|---|
Priority: | blocker → critical |
comment:19 by , 9 years ago
Have built 3.0.4 with compile time js/css cache - a couple of issues remain but this should avoid the cpu/memory intensive problems with wro4j building the js/css cache at run time when the app is first used.
comment:20 by , 9 years ago
Excellent news. Waiting for a pull request so we can build an iso and test the result.
comment:22 by , 9 years ago
With this PR the launch time of GeoNetwork seems to be much less (CPU takes 3 mins to settle down) the service seems to be down after that.
ISO to test: http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-pr84-amd64-1b2aaf7.iso
comment:23 by , 9 years ago
Ok - thanks Angelos - working on why this is failing now.
comment:24 by , 9 years ago
The old sealing violation pain on the jai stuff was causing a misleading exception due to a static initialization in a GN class that required the jai imageio code. Seems the best fix to get this working reliably is to chuck away the jai jars installed in GeoNetwork lib after installation - so a fix to install_geonetwork.sh will do this. Without the jai jars it works reliably.
Could also extend the startup delay for the progress bar to take into account longer startup time?
comment:25 by , 9 years ago
Hi,
Just made a new build based on latest commits and I am still not able to see the UI...
http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-pr84-amd64-fa1852a.iso
http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-pr84-amd64-fa1852a-log.tar.gz
comment:26 by , 9 years ago
Cc: | added |
---|
The pull request has been merged, but the installation still has issues. I have extended the startup delay as requested.
There is a new nightly build:
http://aiolos.survey.ntua.gr/gisvm/dev/osgeo-live-nightly-build151-amd64-81cad69.iso
GeoNetwork fails to start the first time (503 error after the CPU has calmed). The memory allocation and time to startup is much better now. After killing jetty and restarting GeoNetwork, it works as expected.
At least now the status is better than before...
comment:27 by , 9 years ago
comment:28 by , 9 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
comment:29 by , 9 years ago
Priority: | critical → blocker |
---|---|
Resolution: | fixed |
Status: | closed → reopened |
Summary: | GeoNetwork 3.0.1 fails to start in a VM → GeoNetwork 3.0.3 fails to start in 32bit system |
While GeoNetwork 3.0.3 is working as expected in 64 bit system (RC1), unfortunately it fails to work on the 32 bit system.
Here is the Jetty output log from the live 32bit RC1 iso:
2016-03-14 19:20:50.516:INFO:oejs.Server:jetty-8.1.16.v20140903 2016-03-14 19:20:50.762:INFO:oejdp.ScanningAppProvider:Deployment monitor /usr/local/lib/geonetwork/web at interval 1 2016-03-14 19:20:50.768:INFO:oejd.DeploymentManager:Deployable added: /usr/local/lib/geonetwork/web/geonetwork 2016-03-14 19:21:52.707:WARN:oejuc.AbstractLifeCycle:FAILED o.e.j.w.WebAppContext{/geonetwork,[file:/usr/local/lib/geonetwork/web/geonetwork/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/coffee-script-1.8.0.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/emberjs-1.5.0.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/handlebars-1.3.0.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/jquery-1.10.2-1.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/jshint-2.4.3.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/jslint-c657984cd7.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/json2-20110223.jar!/META-INF/resources/, jar:file:/usr/local/lib/geonetwork/web/geonetwork/WEB-INF/lib/less-1.3.3.jar!/META-INF/resources/]},/usr/local/lib/geonetwork/web/geonetwork: java.lang.OutOfMemoryError java.lang.OutOfMemoryError at java.util.zip.Inflater.inflateBytes(Native Method) at java.util.zip.Inflater.inflate(Inflater.java:259) at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:152) at java.io.FilterInputStream.read(FilterInputStream.java:133) at java.io.FilterInputStream.read(FilterInputStream.java:133) at org.objectweb.asm.ClassReader.a(Unknown Source) at org.objectweb.asm.ClassReader.<init>(Unknown Source) at org.eclipse.jetty.annotations.AnnotationParser.scanClass(AnnotationParser.java:898) at org.eclipse.jetty.annotations.AnnotationParser$2.processEntry(AnnotationParser.java:857) at org.eclipse.jetty.webapp.JarScanner.matched(JarScanner.java:161) at org.eclipse.jetty.util.PatternMatcher.matchPatterns(PatternMatcher.java:100) at org.eclipse.jetty.util.PatternMatcher.match(PatternMatcher.java:82) at org.eclipse.jetty.webapp.JarScanner.scan(JarScanner.java:84) at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:869) at org.eclipse.jetty.annotations.AnnotationParser.parse(AnnotationParser.java:884) at org.eclipse.jetty.annotations.AnnotationConfiguration.parseWebInfLib(AnnotationConfiguration.java:422) at org.eclipse.jetty.annotations.AnnotationConfiguration.configure(AnnotationConfiguration.java:120) at org.eclipse.jetty.webapp.WebAppContext.configure(WebAppContext.java:468) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1237) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:717) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:494) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:39) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:186) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:494) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:141) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:145) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:56) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:615) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:540) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:403) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:337) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:121) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:555) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:230) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.util.component.AggregateLifeCycle.doStart(AggregateLifeCycle.java:81) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:58) at org.eclipse.jetty.server.handler.HandlerWrapper.doStart(HandlerWrapper.java:96) at org.eclipse.jetty.server.Server.doStart(Server.java:282) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1274) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1197) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.jetty.start.Main.invokeMain(Main.java:473) at org.eclipse.jetty.start.Main.start(Main.java:615) at org.eclipse.jetty.start.Main.main(Main.java:96)# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate # An error report file with more information is saved as: # /usr/local/lib/geonetwork/jetty/hs_err_pid3960.log
and /usr/local/lib/geonetwork/jetty/hs_err_pid3960.log
# # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (allocation.cpp:214), pid=3960, tid=3008150336 # # JRE version: OpenJDK Runtime Environment (7.0_95) (build 1.7.0_95-b00) # Java VM: OpenJDK Client VM (24.95-b01 mixed mode, sharing linux-x86 ) # Derivative: IcedTea 2.6.4 # Distribution: Ubuntu 14.04.3 LTS, package 7u95-2.6.4-0ubuntu0.14.04.1 # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # --------------- T H R E A D --------------- Current thread (0xb674dc00): JavaThread "C1 CompilerThread0" daemon [_thread_in_native, id=3970, stack(0xb344b000,0xb34cc000)] Stack: [0xb344b000,0xb34cc000] Current CompileTask: C1: 66070 812
comment:30 by , 9 years ago
comment:32 by , 9 years ago
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
comment:33 by , 8 years ago
Milestone: | OSGeoLive9.5 → OSGeoLive10.0 |
---|---|
Resolution: | fixed |
Status: | closed → reopened |
This issue is back on 32 bit iso
comment:34 by , 8 years ago
As can be seen from the attached logs (32 bit system), java runs out of memory on a 4GB VM.
comment:35 by , 8 years ago
Milestone: | OSGeoLive10.0 → OSGeoLive10.5 |
---|
Ticket retargeted after milestone closed
comment:36 by , 7 years ago
Milestone: | OSGeoLive10.5 → OSGeoLive11.0 |
---|
Ticket retargeted after milestone closed
comment:37 by , 7 years ago
Resolution: | → wontfix |
---|---|
Status: | reopened → closed |
32bit system will not be officially supported for 11.0 release
Testing beta1 64b, I don't have the issue on a 2Go RAM virtual machine (with -Xmx set to 1g for the app). It should consume quite some RAM on first startup as it builds the JS cache (minify deps for all apps - search/editor/admin - taking 2-3 min on the VM).
When do you have the error ? on startup ? on use ?