Opened 4 years ago

Closed 3 years ago

#4779 closed defect (invalid)

server process (PID 49666) was terminated by signal 6: Aborted, munmap_chunk(): invalid pointer

Reported by: Lars Aksel Opsahl Owned by: robe
Priority: medium Milestone: PostGIS 3.1.0
Component: postgis Version: 3.0.x
Keywords: Cc:

Description

Hi

Postgres crashes right after this command.

pg_restore  --section=pre-data --format=d -d sl data/sl/schema_sl_rest_ofd/ 2> log/sl/restore_sl_rest_ofdata_pre_data.log

There are no error in the restore_sl_rest_ofdata_pre_data.log

The pre-data input is little bit more than 200.000 lines with more than 6000 CREATE statements, if I test with the first 100 lines it does not crash. It seems like no data are lost here because the server does recovery at startup.

This happens on with Postgres 12 and Postgres 13 for Os I use CentOS Linux release 8 on all the severs. It seems to work on Postgis 2.5 but it seems to fail on any Postgis 3 version.

I have tried to find out what commands that are causeing this problem , but have not been able to find it, but I will get some help on Friday to see if we can find out more and make a simple SQL that fails.

The test 1 with Postgis 3.0 :

POSTGIS="3.0.2 2fb2a18" [EXTENSION] PGSQL="120" GEOS="3.8.1-CAPI-1.13.3" SFCGAL="1.3.7" PROJ="7.1.1" GDAL="GDAL 3.1.3, released 2020/09/01" LIBXML="2.9.7" LIBJSON="0.13.1" LIBPROTOBUF="1.3.0" WAGYU="0.4.3 (Internal)" TOPOLOGY RASTER

In the postgres log I see this :

munmap_chunk(): invalid pointer
<  , , 2020-10-28 06:37:37.215 CET >LOG:  server process (PID 49666) was terminated by signal 6: Aborted
<  , , 2020-10-28 06:37:37.215 CET >LOG:  terminating any other active server processes
<  , , 2020-10-28 06:37:37.215 CET >WARNING:  terminating connection because of crash of another server process
<  , , 2020-10-28 06:37:37.215 CET >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
<  , , 2020-10-28 06:37:37.215 CET >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
<  , , 2020-10-28 06:37:37.216 CET >LOG:  all server processes terminated; reinitializing
<  , , 2020-10-28 06:37:37.618 CET >LOG:  database system was interrupted; last known up at 2020-10-28 06:37:34 CET
<  , , 2020-10-28 06:37:37.628 CET >LOG:  database system was not properly shut down; automatic recovery in progress
<  , , 2020-10-28 06:37:37.628 CET >LOG:  redo starts at 0/164B250
<  , , 2020-10-28 06:37:37.662 CET >LOG:  invalid record length at 0/21C9FE8: wanted 24, got 0
<  , , 2020-10-28 06:37:37.662 CET >LOG:  redo done at 0/21C6478
<  , , 2020-10-28 06:37:37.768 CET >LOG:  database system is ready to accept connections

Here is the stacktrace :

gdb /usr/pgsql-12/bin/postmaster /var/lib/systemd/coredump/core.postmaster.26.3d2c9cba8e9e4eefb0831fc9d85d7a4f.50446.1603863575000000

Program terminated with signal SIGABRT, Aborted.
#0  0x00007f0929bf970f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install SuperLU-5.2.0-7.el8.x86_64 atlas-3.10.3-7.el8.x86_64 audit-libs-3.0-0.17.20191104git1c2f876.el8.x86_64 blas-3.8.0-8.el8.x86_64 boost-atomic-1.66.0-7.el8.x86_64 boost-chrono-1.66.0-7.el8.x86_64 boost-date-time-1.66.0-7.el8.x86_64 boost-serialization-1.66.0-7.el8.x86_64 boost-system-1.66.0-7.el8.x86_64 boost-thread-1.66.0-7.el8.x86_64 brotli-1.0.6-1.el8.x86_64 bzip2-libs-1.0.6-26.el8.x86_64 cyrus-sasl-lib-2.1.27-1.el8.x86_64 expat-2.2.5-3.el8.x86_64 fontconfig-2.13.1-3.el8.x86_64 freetype-2.9.1-4.el8.x86_64 gdal31-libs-3.1.3-2.rhel8.x86_64 geos38-3.8.1-2.rhel8.x86_64 giflib-5.1.4-3.el8.x86_64 glibc-2.28-101.el8.x86_64 gmp-6.1.2-10.el8.x86_64 jasper-libs-2.0.14-4.el8.x86_64 json-c-0.13.1-0.2.el8.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.17-18.el8.x86_64 lapack-3.8.0-8.el8.x86_64 libblkid-2.32.1-22.el8.x86_64 libcap-2.26-3.el8.x86_64 libcap-ng-0.7.9-5.el8.x86_64 libcom_err-1.45.4-3.el8.x86_64 libcurl-7.61.1-12.el8.x86_64 libgcc-8.3.1-5.el8.0.2.x86_64 libgcrypt-1.8.3-4.el8.x86_64 libgfortran-8.3.1-5.el8.0.2.x86_64 libgomp-8.3.1-5.el8.0.2.x86_64 libgpg-error-1.31-1.el8.x86_64 libicu-60.3-2.el8_1.x86_64 libidn2-2.2.0-1.el8.x86_64 libjpeg-turbo-1.5.3-10.el8.x86_64 libmount-2.32.1-22.el8.x86_64 libnghttp2-1.33.0-3.el8_2.1.x86_64 libpng-1.6.34-5.el8.x86_64 libpsl-0.20.2-5.el8.x86_64 libquadmath-8.3.1-5.el8.0.2.x86_64 libselinux-2.9-3.el8.x86_64 libssh-0.9.0-4.el8.x86_64 libstdc++-8.3.1-5.el8.0.2.x86_64 libtiff-4.0.9-17.el8.x86_64 libuuid-2.32.1-22.el8.x86_64 libwebp-1.0.0-1.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 libxml2-2.9.7-7.el8.x86_64 libzstd-1.4.2-2.el8.x86_64 lz4-libs-1.8.1.2-4.el8.x86_64 mpfr-3.1.6-1.el8.x86_64 nspr-4.25.0-2.el8_2.x86_64 ogdi-4.1.0-1.el8.x86_64 openblas-openmp-0.3.3-5.el8.x86_64 openblas-threads-0.3.3-5.el8.x86_64 openldap-2.4.46-11.el8_1.x86_64 openssl-libs-1.1.1c-15.el8.x86_64 pam-1.3.1-8.el8.x86_64 pcre-8.42-4.el8.x86_64 poppler-0.66.0-26.el8.x86_64 postgis30_12-3.0.2-2.rhel8.x86_64 proj-6.3.2-4.el8.x86_64 proj71-7.1.1-1.rhel8.x86_64 protobuf-c-1.3.0-4.el8.x86_64 sqlite-libs-3.26.0-6.el8.x86_64 sssd-client-2.2.3-20.el8.x86_64 systemd-libs-239-31.el8_2.2.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64
(gdb) bt
#0  0x00007f0929bf970f in raise () from /lib64/libc.so.6
#1  0x00007f0929be3b25 in abort () from /lib64/libc.so.6
#2  0x00007f0929c3c897 in __libc_message () from /lib64/libc.so.6
#3  0x00007f0929c42fdc in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f0929c448dc in _int_free () from /lib64/libc.so.6
#5  0x00007f09224fe285 in osgeo::proj::common::UnitOfMeasure::~UnitOfMeasure() () from /usr/proj71/lib/libproj.so.19
#6  0x00007f0929bfbe9c in __run_exit_handlers () from /lib64/libc.so.6
#7  0x00007f0929bfbfd0 in exit () from /lib64/libc.so.6
#8  0x000000000075beb0 in proc_exit (code=code@entry=0) at ipc.c:152
#9  0x000000000077fd21 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x111c218, dbname=<optimized out>, username=<optimized out>) at postgres.c:4455
#10 0x00000000007085a1 in BackendRun (port=0x1119df0, port=0x1119df0) at postmaster.c:4448
#11 BackendStartup (port=0x1119df0) at postmaster.c:4139
#12 ServerLoop () at postmaster.c:1704
#13 0x00000000007094d0 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x10e87f0) at postmaster.c:1377
#14 0x0000000000482c7e in main (argc=3, argv=0x10e87f0) at main.c:228

Test 2 with Postgis 3.1 :

POSTGIS="3.1.0alpha2 b6f415d" [EXTENSION] PGSQL="120" GEOS="3.8.1-CAPI-1.13.3" SFCGAL="1.3.7" PROJ="7.1.1" GDAL="GDAL 3.1.3, released 2020/09/01" LIBXML="2.9.7" LIBJSON="0.13.1" LIBPROTOBUF="1.3.0" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER

In the postgres log I see this :

free(): invalid pointer
<  , , 2020-10-29 09:27:51.215 CET >LOG:  server process (PID 352306) was terminated by signal 6: Aborted
<  , , 2020-10-29 09:27:51.215 CET >LOG:  terminating any other active server processes
<  , , 2020-10-29 09:27:51.215 CET >WARNING:  terminating connection because of crash of another server process
<  , , 2020-10-29 09:27:51.215 CET >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
<  , , 2020-10-29 09:27:51.215 CET >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
<  , , 2020-10-29 09:27:51.216 CET >LOG:  all server processes terminated; reinitializing
<  , , 2020-10-29 09:27:51.618 CET >LOG:  database system was interrupted; last known up at 2020-10-29 09:27:48 CET
<  , , 2020-10-29 09:27:51.628 CET >LOG:  database system was not properly shut down; automatic recovery in progress
<  , , 2020-10-29 09:27:51.629 CET >LOG:  redo starts at 0/7BD37E0
<  , , 2020-10-29 09:27:51.758 CET >LOG:  invalid record length at 0/8736410: wanted 24, got 0
<  , , 2020-10-29 09:27:51.758 CET >LOG:  redo done at 0/8736368
<  , , 2020-10-29 09:27:51.867 CET >LOG:  database system is ready to accept connections

Here is the stacktrace :

gdb /usr/pgsql-12/bin/postmaster /var/lib/systemd/coredump/core.postmaster.26.3d2c9cba8e9e4eefb0831fc9d85d7a4f.352306.1603960069000000

Program terminated with signal SIGABRT, Aborted.
#0  0x00007fcea664a70f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install SuperLU-5.2.0-7.el8.x86_64 atlas-3.10.3-7.el8.x86_64 audit-libs-3.0-0.17.20191104git1c2f876.el8.x86_64 blas-3.8.0-8.el8.x86_64 boost-atomic-1.66.0-7.el8.x86_64 boost-chrono-1.66.0-7.el8.x86_64 boost-date-time-1.66.0-7.el8.x86_64 boost-serialization-1.66.0-7.el8.x86_64 boost-system-1.66.0-7.el8.x86_64 boost-thread-1.66.0-7.el8.x86_64 brotli-1.0.6-1.el8.x86_64 bzip2-libs-1.0.6-26.el8.x86_64 cyrus-sasl-lib-2.1.27-1.el8.x86_64 expat-2.2.5-3.el8.x86_64 fontconfig-2.13.1-3.el8.x86_64 freetype-2.9.1-4.el8.x86_64 gdal31-libs-3.1.3-2.rhel8.x86_64 geos38-3.8.1-2.rhel8.x86_64 giflib-5.1.4-3.el8.x86_64 glibc-2.28-101.el8.x86_64 gmp-6.1.2-10.el8.x86_64 jasper-libs-2.0.14-4.el8.x86_64 json-c-0.13.1-0.2.el8.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.17-18.el8.x86_64 lapack-3.8.0-8.el8.x86_64 libblkid-2.32.1-22.el8.x86_64 libcap-2.26-3.el8.x86_64 libcap-ng-0.7.9-5.el8.x86_64 libcom_err-1.45.4-3.el8.x86_64 libcurl-7.61.1-12.el8.x86_64 libgcc-8.3.1-5.el8.0.2.x86_64 libgcrypt-1.8.3-4.el8.x86_64 libgfortran-8.3.1-5.el8.0.2.x86_64 libgomp-8.3.1-5.el8.0.2.x86_64 libgpg-error-1.31-1.el8.x86_64 libicu-60.3-2.el8_1.x86_64 libidn2-2.2.0-1.el8.x86_64 libjpeg-turbo-1.5.3-10.el8.x86_64 libmount-2.32.1-22.el8.x86_64 libnghttp2-1.33.0-3.el8_2.1.x86_64 libpng-1.6.34-5.el8.x86_64 libpsl-0.20.2-5.el8.x86_64 libquadmath-8.3.1-5.el8.0.2.x86_64 libselinux-2.9-3.el8.x86_64 libssh-0.9.0-4.el8.x86_64 libstdc++-8.3.1-5.el8.0.2.x86_64 libtiff-4.0.9-17.el8.x86_64 libuuid-2.32.1-22.el8.x86_64 libwebp-1.0.0-1.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 libxml2-2.9.7-7.el8.x86_64 libzstd-1.4.2-2.el8.x86_64 lz4-libs-1.8.1.2-4.el8.x86_64 mpfr-3.1.6-1.el8.x86_64 nspr-4.25.0-2.el8_2.x86_64 ogdi-4.1.0-1.el8.x86_64 openblas-openmp-0.3.3-5.el8.x86_64 openblas-threads-0.3.3-5.el8.x86_64 openldap-2.4.46-11.el8_1.x86_64 openssl-libs-1.1.1c-15.el8.x86_64 pam-1.3.1-8.el8.x86_64 pcre-8.42-4.el8.x86_64 poppler-0.66.0-26.el8.x86_64 postgis31_12-3.1.0-alpha2_2.rhel8.x86_64 proj-6.3.2-4.el8.x86_64 proj71-7.1.1-1.rhel8.x86_64 protobuf-c-1.3.0-4.el8.x86_64 sqlite-libs-3.26.0-6.el8.x86_64 sssd-client-2.2.3-20.el8.x86_64 systemd-libs-239-31.el8_2.2.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64
(gdb) bt
#0  0x00007fcea664a70f in raise () from /lib64/libc.so.6
#1  0x00007fcea6634b25 in abort () from /lib64/libc.so.6
#2  0x00007fcea668d897 in __libc_message () from /lib64/libc.so.6
#3  0x00007fcea6693fdc in malloc_printerr () from /lib64/libc.so.6
#4  0x00007fcea66958dc in _int_free () from /lib64/libc.so.6
#5  0x00007fbe13f8e285 in osgeo::proj::common::UnitOfMeasure::~UnitOfMeasure() () from /usr/proj71/lib/libproj.so.19
#6  0x00007fcea664ce9c in __run_exit_handlers () from /lib64/libc.so.6
#7  0x00007fcea664cfd0 in exit () from /lib64/libc.so.6
#8  0x000000000075beb0 in proc_exit (code=code@entry=0) at ipc.c:152
#9  0x000000000077fd21 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x1820138, dbname=<optimized out>, username=<optimized out>) at postgres.c:4455
#10 0x00000000007085a1 in BackendRun (port=0x181dc80, port=0x181dc80) at postmaster.c:4448
#11 BackendStartup (port=0x181dc80) at postmaster.c:4139
#12 ServerLoop () at postmaster.c:1704
#13 0x00000000007094d0 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x17ed800) at postmaster.c:1377
#14 0x0000000000482c7e in main (argc=3, argv=0x17ed800) at main.c:228



The test 3 with Postgis Compiled code :

POSTGIS="3.1.0dev 3.1.0alpha2-100-g50b1e70b1" [EXTENSION] PGSQL="120" GEOS="3.8.1-CAPI-1.13.3" SFCGAL="1.3.7" PROJ="7.1.1" GDAL="GDAL 3.1.3, released 2020/09/01" LIBXML="2.9.7" LIBJSON="0.13.1" LIBPROTOBUF="1.3.0" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER

In the postgres log I see this :

free(): invalid pointer
<  , , 2020-10-29 13:34:29.525 CET >LOG:  server process (PID 418670) was terminated by signal 6: Aborted
<  , , 2020-10-29 13:34:29.525 CET >LOG:  terminating any other active server processes
<  , , 2020-10-29 13:34:29.525 CET >WARNING:  terminating connection because of crash of another server process
<  , , 2020-10-29 13:34:29.525 CET >DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
<  , , 2020-10-29 13:34:29.525 CET >HINT:  In a moment you should be able to reconnect to the database and repeat your command.
<  , , 2020-10-29 13:34:29.526 CET >LOG:  all server processes terminated; reinitializing
<  , , 2020-10-29 13:34:29.933 CET >LOG:  database system was interrupted; last known up at 2020-10-29 13:34:06 CET
<  , , 2020-10-29 13:34:30.063 CET >LOG:  database system was not properly shut down; automatic recovery in progress
<  , , 2020-10-29 13:34:30.091 CET >LOG:  redo starts at 0/14F4F038
<  , , 2020-10-29 13:34:30.624 CET >LOG:  invalid record length at 0/1A56E4A0: wanted 24, got 0
<  , , 2020-10-29 13:34:30.624 CET >LOG:  redo done at 0/1A56E428
<  , , 2020-10-29 13:34:30.893 CET >LOG:  database system is ready to accept connections

Here is the stacktrace :

gdb /usr/pgsql-12/bin/postmaster /var/lib/systemd/coredump/core.postmaster.26.3d2c9cba8e9e4eefb0831fc9d85d7a4f.418670.1603974867000000

Program terminated with signal SIGABRT, Aborted.

#0  0x00007f5b7f80270f in raise () from /lib64/libc.so.6
Missing separate debuginfos, use: yum debuginfo-install SuperLU-5.2.0-7.el8.x86_64 atlas-3.10.3-7.el8.x86_64 audit-libs-3.0-0.17.20191104git1c2f876.el8.x86_64 blas-3.8.0-8.el8.x86_64 boost-atomic-1.66.0-7.el8.x86_64 boost-chrono-1.66.0-7.el8.x86_64 boost-date-time-1.66.0-7.el8.x86_64 boost-serialization-1.66.0-7.el8.x86_64 boost-system-1.66.0-7.el8.x86_64 boost-thread-1.66.0-7.el8.x86_64 brotli-1.0.6-1.el8.x86_64 bzip2-libs-1.0.6-26.el8.x86_64 cyrus-sasl-lib-2.1.27-1.el8.x86_64 expat-2.2.5-3.el8.x86_64 fontconfig-2.13.1-3.el8.x86_64 freetype-2.9.1-4.el8.x86_64 giflib-5.1.4-3.el8.x86_64 glibc-2.28-101.el8.x86_64 gmp-6.1.2-10.el8.x86_64 jasper-libs-2.0.14-4.el8.x86_64 jbigkit-libs-2.1-14.el8.x86_64 json-c-0.13.1-0.2.el8.x86_64 keyutils-libs-1.5.10-6.el8.x86_64 krb5-libs-1.17-18.el8.x86_64 lapack-3.8.0-8.el8.x86_64 libaec-1.0.2-3.el8.x86_64 libblkid-2.32.1-22.el8.x86_64 libcap-2.26-3.el8.x86_64 libcap-ng-0.7.9-5.el8.x86_64 libcom_err-1.45.4-3.el8.x86_64 libcurl-7.61.1-12.el8.x86_64 libgcc-8.3.1-5.el8.0.2.x86_64 libgcrypt-1.8.3-4.el8.x86_64 libgfortran-8.3.1-5.el8.0.2.x86_64 libgomp-8.3.1-5.el8.0.2.x86_64 libgpg-error-1.31-1.el8.x86_64 libicu-60.3-2.el8_1.x86_64 libidn2-2.2.0-1.el8.x86_64 libjpeg-turbo-1.5.3-10.el8.x86_64 libmount-2.32.1-22.el8.x86_64 libnghttp2-1.33.0-3.el8_2.1.x86_64 libpng-1.6.34-5.el8.x86_64 libpsl-0.20.2-5.el8.x86_64 libquadmath-8.3.1-5.el8.0.2.x86_64 libselinux-2.9-3.el8.x86_64 libssh-0.9.0-4.el8.x86_64 libstdc++-8.3.1-5.el8.0.2.x86_64 libtiff-4.0.9-17.el8.x86_64 libuuid-2.32.1-22.el8.x86_64 libwebp-1.0.0-1.el8.x86_64 libxcrypt-4.1.1-4.el8.x86_64 libxml2-2.9.7-7.el8.x86_64 libzstd-1.4.2-2.el8.x86_64 lz4-libs-1.8.1.2-4.el8.x86_64 mpfr-3.1.6-1.el8.x86_64 nspr-4.25.0-2.el8_2.x86_64 ogdi-4.1.0-1.el8.x86_64 openblas-openmp-0.3.3-5.el8.x86_64 openblas-threads-0.3.3-5.el8.x86_64 openldap-2.4.46-11.el8_1.x86_64 openssl-libs-1.1.1c-15.el8.x86_64 pam-1.3.1-8.el8.x86_64 pcre-8.42-4.el8.x86_64 pgrouting_12-3.1.0-2.rhel8.x86_64 poppler-0.66.0-26.el8.x86_64 proj-6.3.2-4.el8.x86_64 protobuf-c-1.3.0-4.el8.x86_64 sqlite-libs-3.26.0-6.el8.x86_64 sssd-client-2.2.3-20.el8.x86_64 systemd-libs-239-31.el8_2.2.x86_64 xz-libs-5.2.4-3.el8.x86_64 zlib-1.2.11-16.el8_2.x86_64
(gdb) bt
#0  0x00007f5b7f80270f in raise () from /lib64/libc.so.6
#1  0x00007f5b7f7ecb25 in abort () from /lib64/libc.so.6
#2  0x00007f5b7f845897 in __libc_message () from /lib64/libc.so.6
#3  0x00007f5b7f84bfdc in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f5b7f84d8dc in _int_free () from /lib64/libc.so.6
#5  0x00007f5b77adf285 in __gnu_cxx::new_allocator<char>::deallocate (this=0x16a6840, __p=<optimized out>) at /usr/include/c++/8/ext/new_allocator.h:116
#6  std::allocator_traits<std::allocator<char> >::deallocate (__a=..., __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/8/bits/alloc_traits.h:462
#7  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_destroy (__size=<optimized out>, this=0x16a6840) at /usr/include/c++/8/bits/basic_string.h:226
#8  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_dispose (this=0x16a6840) at /usr/include/c++/8/bits/basic_string.h:221
#9  std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string (this=0x16a6840, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/basic_string.h:657
#10 osgeo::proj::common::UnitOfMeasure::Private::~Private (this=0x16a6840, __in_chrg=<optimized out>) at iso19111/common.cpp:72
#11 std::default_delete<osgeo::proj::common::UnitOfMeasure::Private>::operator() (this=0x7f5b77f2dfc0 <osgeo::proj::common::UnitOfMeasure::PPM_PER_YEAR+16>, __ptr=0x16a6840)
    at /usr/include/c++/8/bits/unique_ptr.h:81
#12 std::unique_ptr<osgeo::proj::common::UnitOfMeasure::Private, std::default_delete<osgeo::proj::common::UnitOfMeasure::Private> >::~unique_ptr (
    this=0x7f5b77f2dfc0 <osgeo::proj::common::UnitOfMeasure::PPM_PER_YEAR+16>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/unique_ptr.h:269
#13 osgeo::proj::common::UnitOfMeasure::~UnitOfMeasure (this=0x7f5b77f2dfb0 <osgeo::proj::common::UnitOfMeasure::PPM_PER_YEAR>, __in_chrg=<optimized out>) at ../include/proj/common.hpp:60
#14 0x00007f5b7f804e9c in __run_exit_handlers () from /lib64/libc.so.6
#15 0x00007f5b7f804fd0 in exit () from /lib64/libc.so.6
#16 0x000000000075beb0 in proc_exit (code=code@entry=0) at ipc.c:152
#17 0x000000000077fd21 in PostgresMain (argc=<optimized out>, argv=argv@entry=0x123c4b8, dbname=<optimized out>, username=<optimized out>) at postgres.c:4455
#18 0x00000000007085a1 in BackendRun (port=0x122bd50, port=0x122bd50) at postmaster.c:4448
#19 BackendStartup (port=0x122bd50) at postmaster.c:4139
#20 ServerLoop () at postmaster.c:1704
#21 0x00000000007094d0 in PostmasterMain (argc=argc@entry=3, argv=argv@entry=0x11f8800) at postmaster.c:1377
#22 0x0000000000482c7e in main (argc=3, argv=0x11f8800) at main.c:228

Change History (9)

comment:1 by robe, 4 years ago

Version: 2.5.x3.0.x

comment:2 by Lars Aksel Opsahl, 4 years ago

I got help from Magnus Hagender he found out that the problem was related this https://stackoverflow.com/questions/42912038/what-is-the-difference-between-cxa-atexit-and-atexit. He also that this does not happen all systems. Hen had run it on CentOs before i failed.

Then I managed to make a simple test cases that fails.

Run this command

psql  postgres -c "drop DATABASE test1"
psql  postgres -c "CREATE DATABASE test1 template=template0;"
psql  test1 -c "create extension pg_stat_statements; create extension postgis; create extension postgis_topology; create extension postgis_raster ;"


and I find this in my postgres log

free(): invalid pointer
2020-10-31 10:50:50.857 CET [25765] LOG:  server process (PID 28736) was terminated by signal 6: Aborted
2020-10-31 10:50:50.857 CET [25765] LOG:  terminating any other active server processes
2020-10-31 10:50:50.857 CET [28731] WARNING:  terminating connection because of crash of another server process
2020-10-31 10:50:50.857 CET [28731] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-10-31 10:50:50.857 CET [28731] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2020-10-31 10:50:50.858 CET [25765] LOG:  all server processes terminated; reinitializing
2020-10-31 10:50:50.906 CET [28740] LOG:  database system was interrupted; last known up at 2020-10-31 10:50:49 CET
2020-10-31 10:50:51.404 CET [28740] LOG:  database system was not properly shut down; automatic recovery in progress
2020-10-31 10:50:51.427 CET [28740] LOG:  redo starts at 0/EF5E570
2020-10-31 10:50:51.470 CET [28740] LOG:  invalid record length at 0/FA38A08: wanted 24, got 0
2020-10-31 10:50:51.470 CET [28740] LOG:  redo done at 0/FA207A0
2020-10-31 10:50:52.021 CET [25765] LOG:  database system is ready to accept connections

Then run this

psql  test1 -c "CREATE SCHEMA test_topo_ar50;
CREATE UNLOGGED TABLE test_topo_ar50.ar50_utvikling_flate_border_line_many_points (
    id integer NOT NULL,
    log_time timestamp without time zone DEFAULT now(),
    geo public.geometry(LineString,25833)
);
CREATE SEQUENCE test_topo_ar50.ar50_utvikling_flate_border_line_many_points_id_seq
    AS integer
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;
CREATE SCHEMA tmp_jm_raster;
CREATE TABLE tmp_jm_raster.dem25_norge_raster (
    rid integer NOT NULL,
    rast public.raster,
    filename text,
    CONSTRAINT enforce_height_rast CHECK ((public.st_height(rast) = 100)),
    CONSTRAINT enforce_max_extent_rast CHECK (public.st_coveredby(public.st_convexhull(rast), '0103000020E96400000100000005000000000000001803FCC0000000A01C6C5841000000001803FCC0000000A06E645E4100000080DA483141000000A06E645E4100000080DA483141000000A01C6C5841000000001803FCC0000000A01C6C5841'::public.geometry)),
    CONSTRAINT enforce_nodata_values_rast CHECK (((public._raster_constraint_nodata_values(rast))::numeric(16,10)[] = '{32767}'::numeric(16,10)[])),
    CONSTRAINT enforce_num_bands_rast CHECK ((public.st_numbands(rast) = 1)),
    CONSTRAINT enforce_out_db_rast CHECK ((public._raster_constraint_out_db(rast) = '{f}'::boolean[])),
    CONSTRAINT enforce_pixel_types_rast CHECK ((public._raster_constraint_pixel_types(rast) = '{16BSI}'::text[])),
    CONSTRAINT enforce_same_alignment_rast CHECK (public.st_samealignment(rast, '0100000000000000000000394000000000000039C0000000001803FCC0000000A06E645E4100000000000000000000000000000000E964000001000100'::public.raster)),
    CONSTRAINT enforce_scalex_rast CHECK (((public.st_scalex(rast))::numeric(16,10) = (25)::numeric(16,10))),
    CONSTRAINT enforce_scaley_rast CHECK (((public.st_scaley(rast))::numeric(16,10) = ('-25'::integer)::numeric(16,10))),
    CONSTRAINT enforce_srid_rast CHECK ((public.st_srid(rast) = 25833)),
    CONSTRAINT enforce_width_rast CHECK ((public.st_width(rast) = 100))
);
CREATE SCHEMA ok_grl;
CREATE TABLE ok_grl.ok_kommkyst_flate (
    gid integer NOT NULL,
    ftema integer,
    komid integer,
    kartblad_5 character varying(9),
    kartblad_10 character varying(9),
    kartblad_20 character varying(9),
    kartblad_dmk character varying(9),
    dmk_kode integer,
    areal numeric,
    objectid integer,
    area numeric,
    len numeric,
    geo public.geometry,
    CONSTRAINT enforce_dims_geo CHECK ((public.st_ndims(geo) = 2)),
    CONSTRAINT enforce_geotype_geo CHECK (((public.geometrytype(geo) = 'POLYGON'::text) OR (geo IS NULL))),
    CONSTRAINT enforce_srid_geo CHECK ((public.st_srid(geo) = 4258))
);
CREATE SEQUENCE ok_grl.ok_kommkyst_flate_gid_seq
    START WITH 1
    INCREMENT BY 1
    NO MINVALUE
    NO MAXVALUE
    CACHE 1;"

and you see this again

free(): invalid pointer
2020-10-31 10:51:20.530 CET [25765] LOG:  server process (PID 28753) was terminated by signal 6: Aborted
2020-10-31 10:51:20.530 CET [25765] LOG:  terminating any other active server processes
2020-10-31 10:51:20.530 CET [28750] WARNING:  terminating connection because of crash of another server process
2020-10-31 10:51:20.530 CET [28750] DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
2020-10-31 10:51:20.530 CET [28750] HINT:  In a moment you should be able to reconnect to the database and repeat your command.
2020-10-31 10:51:20.531 CET [25765] LOG:  all server processes terminated; reinitializing
2020-10-31 10:51:20.624 CET [28756] LOG:  database system was interrupted; last known up at 2020-10-31 10:50:51 CET
2020-10-31 10:51:20.894 CET [28756] LOG:  database system was not properly shut down; automatic recovery in progress
2020-10-31 10:51:20.920 CET [28756] LOG:  redo starts at 0/FA38A80
2020-10-31 10:51:20.921 CET [28756] LOG:  invalid record length at 0/FA79D40: wanted 24, got 0
2020-10-31 10:51:20.921 CET [28756] LOG:  redo done at 0/FA78988
2020-10-31 10:51:21.322 CET [25765] LOG:  database system is ready to accept connections


(sorry wrong time on my test server)

comment:3 by Lars Aksel Opsahl, 3 years ago

Magnus told that postgis depend different versions of proj so I tested this and thats correct.

proj-datumgrid-1.8-6.3.2.4.el8.noarch
proj71-7.1.1-1.rhel8.x86_64
proj71-devel-7.1.1-1.rhel8.x86_64
proj71-static-7.1.1-1.rhel8.x86_64
maven-project-2.2.1-47.el7.noarch
proj71-debugsource-7.1.1-1.rhel8.x86_64
proj-6.3.2-4.el8.x86_64
proj71-debuginfo-7.1.1-1.rhel8.x86_64

I am not able to remove proj-6.3.2-4 without removing postgis

 yum remove proj-6*
Dependencies resolved.
==================================================================================================================================================================================================
 Package                                        Architecture                      Version                                                           Repository                               Size
==================================================================================================================================================================================================
Removing:
 proj                                           x86_64                            6.3.2-4.el8                                                       @epel                                   9.5 M
Removing dependent packages:
 gdal31                                         x86_64                            3.1.3-2.rhel8                                                     @pgdg-common                            830 k
 gdal31-devel                                   x86_64                            3.1.3-2.rhel8                                                     @pgdg-common                            1.1 M
 libgeotiff                                     x86_64                            1.5.1-1.el8                                                       @epel                                   338 k
 pgrouting_12                                   x86_64                            3.1.0-2.rhel8                                                     @pgdg12                                 5.4 M
 postgis31_12-client                            x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                 1.6 M
 postgis31_12-devel                             x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                   0  
 postgis31_12-utils                             x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                 214 k
Removing unused dependencies:
 armadillo                                      x86_64                            9.700.2-1.el8                                                     @epel                                    83 k
 arpack                                         x86_64                            3.7.0-1.el8                                                       @epel                                   729 k
 gdal31-libs                                    x86_64                            3.1.3-2.rhel8                                                     @pgdg-common                             27 M
 gpsbabel                                       x86_64                            1.6.0-3.el8                                                       @epel                                   3.1 M
 libspatialite                                  x86_64                            5.0.0-beta0_1.el8                                                 @epel                                    13 M
 libusb                                         x86_64                            1:0.1.5-12.el8                                                    @BaseOS                                  68 k
 mariadb-connector-c                            x86_64                            3.0.7-1.el8                                                       @AppStream                              424 k
 ogdi                                           x86_64                            4.1.0-1.el8                                                       @epel                                   865 k
 openblas                                       x86_64                            0.3.3-5.el8                                                       @AppStream                               27 M
 openblas-threads                               x86_64                            0.3.3-5.el8                                                       @AppStream                               27 M
 openblas-threads64_                            x86_64                            0.3.3-5.el8                                                       @PowerTools                              27 M
 perl-DBD-Pg                                    x86_64                            3.7.4-4.module_el8.1.0+268+81255909                               @AppStream                              585 k
 postgis31_12                                   x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                  33 M
 proj-datumgrid                                 noarch                            1.8-6.3.2.4.el8                                                   @epel                                   7.5 M
 shapelib                                       x86_64                            1.5.0-1.el8                                                       @epel                                   185 k

Transaction Summary
==================================================================================================================================================================================================
Remove  23 Packages


And with proj7

 yum remove proj71*
Dependencies resolved.
==================================================================================================================================================================================================
 Package                                        Architecture                      Version                                                           Repository                               Size
==================================================================================================================================================================================================
Removing:
 proj71                                         x86_64                            7.1.1-1.rhel8                                                     @pgdg-common                             10 M
 proj71-debuginfo                               x86_64                            7.1.1-1.rhel8                                                     @pgdg-common                             58 M
 proj71-debugsource                             x86_64                            7.1.1-1.rhel8                                                     @pgdg-common                            5.3 M
 proj71-devel                                   x86_64                            7.1.1-1.rhel8                                                     @pgdg-common                            564 k
 proj71-static                                  x86_64                            7.1.1-1.rhel8                                                     @pgdg-common                            7.7 M
Removing dependent packages:
 gdal31                                         x86_64                            3.1.3-2.rhel8                                                     @pgdg-common                            830 k
 libgeotiff16                                   x86_64                            1.6.0-1.rhel8                                                     @pgdg-common                            340 k
 libgeotiff16-devel                             x86_64                            1.6.0-1.rhel8                                                     @pgdg-common                            132 k
 pgrouting_12                                   x86_64                            3.1.0-2.rhel8                                                     @pgdg12                                 5.4 M
 postgis31_12-client                            x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                 1.6 M
 postgis31_12-devel                             x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                   0  
 postgis31_12-utils                             x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                 214 k
Removing unused dependencies:
 gpsbabel                                       x86_64                            1.6.0-3.el8                                                       @epel                                   3.1 M
 libtiff-devel                                  x86_64                            4.0.9-17.el8                                                      @AppStream                              1.5 M
 libusb                                         x86_64                            1:0.1.5-12.el8                                                    @BaseOS                                  68 k
 perl-DBD-Pg                                    x86_64                            3.7.4-4.module_el8.1.0+268+81255909                               @AppStream                              585 k
 postgis31_12                                   x86_64                            3.1.0-alpha2_2.rhel8                                              @pgdg12                                  33 M
 shapelib                                       x86_64                            1.5.0-1.el8                                                       @epel                                   185 k

Transaction Summary
==================================================================================================================================================================================================
Remove  18 Packages

comment:4 by robe, 3 years ago

Are you using yum.postgresql.org package? This seems like a packaging issue to me. So better to bring up on the yum mailing list - https://www.postgresql.org/list/pgsql-pkg-yum

As far as PostGIS goes since GDAL 3.0, GDAL now requires Proj to be linked in. This was not the case with prior GDAL.

That said if you have both postgis and postgis_raster loaded, it is possible for postgis to be using one proj version and GDAL (which postgis_raster relies on) to be using a different version of proj. That would cause all sorts of chaos. Same issue with ogr_fdw extension.

I think it's best for gdal and postgis to be compiled with the same version of proj.

comment:5 by robe, 3 years ago

Ah I see you are using raster so that is probably the issue. As mentioned bring it up on pgsql-pkg-yum. I think yum ships both proj and gdal. So it seems it was compiled with different versions which is bad.

Last edited 3 years ago by robe (previous) (diff)

comment:6 by Lars Aksel Opsahl, 3 years ago

Yes I am using yum .

Her is a update from magnus.hagander@… while he is waiting for OSGeo userid

[root@dd4500a87f1b /]# ldd /usr/pgsql-12/lib/postgis-3.so |grep proj
        libproj.so.15 => /lib64/libproj.so.15 (0x00007fd55f797000)
[root@dd4500a87f1b /]# ldd /usr/pgsql-12/lib/postgis_raster-3.so |grep proj
        libproj.so.15 => /lib64/libproj.so.15 (0x00007fe056242000)
        libproj.so.19 => /usr/proj71/lib/libproj.so.19 (0x00007fe04713c000)

And from gdal

# ldd /usr/gdal31/lib/libgdal.so.27|grep proj
        libproj.so.15 => /lib64/libproj.so.15 (0x00007f91d168c000)
        libproj.so.19 => /usr/proj71/lib/libproj.so.19 (0x00007f91c7a88000)


gdal are linked against libspataialite:

# ldd /usr/gdal31/lib/libgdal.so.27|grep spatialite
        libspatialite.so.7 => /lib64/libspatialite.so.7 (0x00007f6228a8f000)

# ldd /lib64/libspatialite.so.7|grep proj
        libproj.so.15 => /lib64/libproj.so.15 (0x00007ff5399e5000)

gdal → proj19

So here is the problem : gdal → spatialite → proj 17

postgis_raster linkes against gdal…

Magnus has sent message to Devrim.

comment:7 by pramsey, 3 years ago

Owner: changed from pramsey to robe

Should consider closing this as a packaging issue. Building against one version and shipping another, alas.

comment:9 by Lars Aksel Opsahl, 3 years ago

Resolution: invalid
Status: newclosed
Note: See TracTickets for help on using tickets.