ERROR: could not load library "/usr/lib/postgresql/9.5/lib/rtpostgis-2.2.so": libnetcdf.so.11: cannot map zero-fill pages

[reecefowell]

I am using the PostGIS with Hibernate-Spatial for a Java Spring project, the version output from PostGreSQL is:

"PostgreSQL 9.5.4 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.2) 5.4.0 20160609, 64-bit POSTGIS="2.2.1 r14555" GEOS="3.5.0-CAPI-1.9.0 r4084" PROJ="Rel. 4.9.2, 08 September 2015" GDAL="GDAL 1.11.3, released 2015/09/16" LIBXML="2.9.3" (…)"

When running tests with Concordion, after the 6th example passes, all subsequent tests fail.

The error I get is:

2016-10-03 14:07:28.722  INFO 35396 --- [           main] o.f.c.i.metadatatable.MetaDataTableImpl  : Creating Metadata table: "public"."schema_version"
2016-10-03 14:07:28.738  INFO 35396 --- [           main] o.f.core.internal.command.DbMigrate      : Current version of schema "public": << Empty Schema >>
2016-10-03 14:07:28.738  INFO 35396 --- [           main] o.f.core.internal.command.DbMigrate      : Migrating schema "public" to version 1 - Initial
2016-10-03 14:07:29.703 ERROR 35396 --- [           main] o.f.core.internal.command.DbMigrate      : Migration of schema "public" to version 1 failed! Changes successfully rolled back.

org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException:
Migration V1__Initial.sql failed
--------------------------------
SQL State  : XX000
Error Code : 0
Message    : ERROR: could not load library "/usr/lib/postgresql/9.5/lib/rtpostgis-2.2.so": libnetcdf.so.11: cannot map zero-fill pages
Location   : db/migration/V1__Initial.sql (/Users/reecefowell/Projects/Spring/bvddy/matchmaking-service/build/resources/main/db/migration/V1__Initial.sql)
Line       : 1
Statement  : CREATE EXTENSION IF NOT EXISTS Postgis

    at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:117)
    at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:71)
    at org.flywaydb.core.internal.command.DbMigrate$5.doInTransaction(DbMigrate.java:284)
    at org.flywaydb.core.internal.command.DbMigrate$5.doInTransaction(DbMigrate.java:282)
    at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
    at org.flywaydb.core.internal.command.DbMigrate.applyMigration(DbMigrate.java:282)
    at org.flywaydb.core.internal.command.DbMigrate.access$800(DbMigrate.java:46)
    at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:207)
    at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:156)
    at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
    at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:156)
    at org.flywaydb.core.Flyway$1.execute(Flyway.java:1059)
    at org.flywaydb.core.Flyway$1.execute(Flyway.java:1006)
    at org.flywaydb.core.Flyway.execute(Flyway.java:1418)
    at org.flywaydb.core.Flyway.migrate(Flyway.java:1006)
    at com.bvddy.util.DatabaseUtil.migrate(DatabaseUtil.java:44)
    at com.bvddy.util.DatabaseUtil.cleanAndMigrate(DatabaseUtil.java:19)
    at com.bvddy.test.spec.AbstractSpec.setUp(AbstractSpec.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.RunPrepareTestInstanceCallbacks.evaluate(RunPrepareTestInstanceCallbacks.java:64)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.springframework.test.context.junit4.statements.SpringFailOnTimeout.evaluate(SpringFailOnTimeout.java:87)
    at org.springframework.test.context.junit4.statements.ProfileValueChecker.evaluate(ProfileValueChecker.java:101)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
    at org.concordion.integration.junit4.ConcordionRunner.runChild(ConcordionRunner.java:187)
    at org.concordion.integration.junit4.ConcordionRunner.runChild(ConcordionRunner.java:23)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.springframework.test.context.junit4.statements.ProfileValueChecker.evaluate(ProfileValueChecker.java:101)
    at org.springframework.test.context.junit4.rules.SpringClassRule$TestContextManagerCacheEvictor.evaluate(SpringClassRule.java:242)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.concordion.integration.junit4.ConcordionRunner.run(ConcordionRunner.java:146)
    at org.junit.runners.Suite.runChild(Suite.java:128)
    at org.junit.runners.Suite.runChild(Suite.java:27)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:117)
    at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:42)
    at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:262)
    at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:84)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)
Caused by: org.postgresql.util.PSQLException: ERROR: could not load library "/usr/lib/postgresql/9.5/lib/rtpostgis-2.2.so": libnetcdf.so.11: cannot map zero-fill pages
    at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2310)
    at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2023)
    at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:217)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:421)
    at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:318)
    at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:310)
    at org.flywaydb.core.internal.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:238)
    at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:114)
    ... 65 more

After this is output on first test that fails, all other tests show that it could not connect to the database.

My Schema looks looks like this:

CREATE EXTENSION IF NOT EXISTS Postgis;

CREATE TABLE "user" (
    id               bigint  NOT NULL,
    first_name       varchar NOT NULL,
    search_radius    int,
    latitude         double precision,
    longitude        double precision,
    location         geography(Point, 4326),

    PRIMARY KEY (id)
);

Each test runs the following to clean the database state:

@Component
public class DatabaseUtil {
    @Autowired
    private DataSource dataSource;
    @Autowired
    private Flyway flyway;

    public void cleanAndMigrate() {
        clean();
        migrate();
    }

    private void clean() {
        try {
            dataSource.getConnection();
        } catch (SQLException e) {
        }
        JdbcTemplate jdbcTemplate = new JdbcTemplate(dataSource);
        jdbcTemplate.execute("DROP SCHEMA IF EXISTS public CASCADE");
        jdbcTemplate.execute("CREATE SCHEMA public");
    }

    private void migrate() {
        flyway.migrate();
    }
}

From what I can tell, it looks like maybe the process of generating the table spatial_ref_sys could be eating up all memory.

Whatever it is, it seems to kill the postgresql server and cause all tests to fail. Could this be a memory leak?

My spring setup. concordion:2.0.2 flyway-core:3.2.1 hibernate-core:5.0.9.Final hibernate-spatial:5.0.9.Final

Whats strange is that if I paste the following into pgAdmin3 and run arbitrary SQL over and over by slamming the run button, it doesn't seem to fail. What could be causing this disparity?

DROP SCHEMA IF EXISTS public CASCADE;
CREATE SCHEMA public;

CREATE EXTENSION IF NOT EXISTS Postgis;

CREATE TABLE "user" (
    id               bigint  NOT NULL,
    first_name       varchar NOT NULL,
    search_radius    int,
    latitude         double precision,
    longitude        double precision,
    location         geography(Point, 4326),

    PRIMARY KEY (id)
);

As per the ticket process, I tried to use an earlier version but in Ubuntu there is no version of PostGIS above or below 2.2. I install the following:

• postgresql-9.5
• postgresql-9.5-postgis-2.2
• pgadmin3
• postgresql-contrib-9.5
• python-psycopg2

[robe]

oh boy, this is a tough one to troubleshoot especially since I don't build with netcdf and error seems to be triggered by that.

Caused by: org.postgresql.util.PSQLException: ERROR: could not load library "/usr/lib/postgresql/9.5/lib/rtpostgis-2.2.so": libnetcdf.so.11: cannot map zero-fill pages

which is kinda bizzare because I wouldn't expect that to be loaded until needed.

Anyrate the error suggests you are just out of memory. Is your JDBC project running on same box as your PostgreSQL instance?

My guess would be that your JDBC Hibernate, is just hogging up resources enough so PostGIS extension can't load. You don't see it in pgAdmin perhaps because you are running pgAdmin from your desktop or it's not as much of a resource hog.

[robe]

I forgot to ask, how much motherboard RAM do you have on that box?

[reecefowell]

The Java is running on my Mac when running the tests, and the Postgresql instance in a VirtualBox instance via Vagrant.

The vbox instance has only 1 GB of RAM as my project requires lots of separate instances due to being a microservices based setup.

I changed it to 2GB and now all tests pass.

Thanks.