Opened 3 weeks ago

Closed 2 weeks ago

#5779 closed defect (fixed)

create_upgrade.pl fails for 3.5.0-beta1

Reported by: sebastic Owned by: strk
Priority: medium Milestone: PostGIS 3.5.0
Component: build Version: master
Keywords: Cc:

Description

The Debian package for 3.5.0-beta1 failed to build on some architectures due to parallel build issue:

---- Making all in utils
make[3]: Entering directory '/build/postgis/utils'
sed -e '1,/^__END__/d' postgis_restore.pl.in \
        | sort -u \
        > postgis_restore_data.hardcoded
/usr/bin/make -C ../raster/rt_pg rtpostgis_upgrade.sql
make[4]: Entering directory '/build/postgis/raster/rt_pg'
/usr/bin/make -C ../raster/rt_pg uninstall_rtpostgis.sql
make[4]: Entering directory '/build/postgis/raster/rt_pg'
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp
grep -v '^#' rtpostgis.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp
rm -f rtpostgis.sql.tmp
/usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql
grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql
grep -v '^#' rtpostgis.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql
make[4]: Leaving directory '/build/postgis/raster/rt_pg'
grep -v '^#' rtpostgis_drop.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql
rm -f rtpostgis_upgrade_cleanup.sql.tmp
grep: rtpostgis.sql.tmp: No such file or directory
rm -f rtpostgis.sql.tmp
rm -f rtpostgis_drop.sql.tmp
/usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in
Unable to locate target new version number in rtpostgis.sql
make[4]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255
make[4]: *** Deleting file 'rtpostgis_upgrade.sql.in'
make[4]: Leaving directory '/build/postgis/raster/rt_pg'
make[3]: *** [Makefile:71: ../raster/rt_pg/rtpostgis_upgrade.sql] Error 2
make[3]: Leaving directory '/build/postgis/utils'
make[2]: *** [GNUmakefile:28: all] Error 1
make[2]: Leaving directory '/build/postgis'
make[1]: *** [debian/rules:129: override_dh_auto_build] Error 2
make[1]: Leaving directory '/build/postgis'

The build failed on amd64 CI (parallel=2), s390x buildd (parallel=2). It did not locally at first with parallel=11 but did the second time, but again not with parallel=1. This suggests a race condition between the Makefile dependencies.

Chaining the commands to create the SQL file seems to help, but that might also just be the race condition:

--- postgis.orig/raster/rt_pg/Makefile.in
+++ postgis/raster/rt_pg/Makefile.in
@@ -129,9 +129,9 @@ endif
 #       in the extension/ folder
 #
 %.sql: %.sql.in
-       $(SQLPP) -I@top_builddir@/postgis/ $< > $@.tmp
+       $(SQLPP) -I@top_builddir@/postgis/ $< > $@.tmp && \
        grep -v '^#' $@.tmp | \
-       $(PERL) -lpe "s'MODULE_PATHNAME'\$(MODULEPATH)'g;s'@extschema@\.''g" > $@
+       $(PERL) -lpe "s'MODULE_PATHNAME'\$(MODULEPATH)'g;s'@extschema@\.''g" > $@ && \
        rm -f $@.tmp
 
 ../../liblwgeom/.libs/liblwgeom.a:

Change History (11)

comment:1 by strk, 3 weeks ago

Can you still reproduce reverting [5aba84713102a23d8755d1df3f56c277e1dff6b1/git] ?

I saw utils/Makefile has targets to build thing under raster/ which I think was meant to be used when raster is disabled. Those utils/ rules should do nothing if everything under raster/ is already built, and the commit I'm referencing above is building raster/ *after* utils/

comment:2 by robe, 3 weeks ago

As noted by Bas on postgis-dev mailing list


Reverting 5aba84713102a23d8755d1df3f56c277e1dff6b1 seems to resolve the issue, the Salsa CI and buildd builds that failed (more than once) before succeeded now.

Kind Regards,

Bas


comment:3 by sebastic, 3 weeks ago

strk wrote on postgis-dev:

We've upgraded it, Regina updated the ticket with this information and I've further pushed a temptative fix so if you can test that'd be useful ( commit [4b786d9992479041f4bc0cf559cc10e3feca4e2c/git] )

That commit cannot be found on https://git.osgeo.org/gitea/postgis/postgis.git (yet).

Last edited 3 weeks ago by strk (previous) (diff)

comment:4 by strk, 3 weeks ago

Now it is, please check (Gitea failed to contact trac due to database connection limit being reached so the ticket wasn't updated)

comment:5 by sebastic, 3 weeks ago

With the changes from 4b786d9992479041f4bc0cf559cc10e3feca4e2c the build fails again:

---- Making all in postgis
make[4]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/extensions/postgis'
mkdir -p sql
/usr/bin/make -C ../../raster/rt_pg sql_objs
make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
/usr/bin/make -C ../../raster/rt_pg sql_objs
make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
/usr/bin/make -C ../../raster/rt_pg sql_objs
make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I./../../postgis ../../postgis/postgis.sql.in > sql/postgis_for_extension.sql.tmp
/usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' ../../spatial_ref_sys.sql > sql/spatial_ref_sys.sql
/usr/bin/perl ../../utils/create_spatial_ref_sys_config_dump.pl ../../spatial_ref_sys.sql > sql/spatial_ref_sys_config_dump.sql
grep -v '^#' sql/postgis_for_extension.sql.tmp | \
/usr/bin/perl -lpe \
        "s'MODULE_PATHNAME'\$libdir/postgis-3'g" \
        | /usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' > sql/postgis_for_extension.sql
echo '-- Just tag extension postgis version as "ANY"' > sql/postgis--TEMPLATED--TO--ANY.sql
echo '-- Installed by postgis 3.5.0beta1' >> sql/postgis--TEMPLATED--TO--ANY.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp
cat postgis.control.in \
        | sed -e 's|@EXTVERSION@|3.5.0beta1|g' \
        | sed -e 's|@EXTENSION@|postgis|g' \
        | sed -e 's|@MODULEPATH@|$libdir/postgis-3|g' \
        > postgis.control
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp
echo '-- Built on 2024-09-18 14:44:20' >> sql/postgis--TEMPLATED--TO--ANY.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp
rm -f sql/postgis_for_extension.sql.tmp
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp
echo "-- Nothing to do here" > sql/postgis--unpackaged.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp
printf '\\echo Use "CREATE EXTENSION postgis" to load this file. \\quit\n' > sql/postgis--3.5.0beta1.sql
/usr/bin/perl ../../utils/create_upgrade.pl sql/postgis_for_extension.sql > sql/postgis_upgrade_for_extension.sql.in
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp
cat sql/postgis_for_extension.sql sql/spatial_ref_sys_config_dump.sql sql/spatial_ref_sys.sql >> sql/postgis--3.5.0beta1.sql
grep -v '^#' rtpostgis.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp
grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql
grep -v '^#' rtpostgis_drop.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql
grep -v '^#' rtpostgis_legacy.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql
grep -v '^#' rtpostgis.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp
rm -f rtpostgis_legacy.sql.tmp
rm -f rtpostgis_upgrade_cleanup.sql.tmp
rm -f rtpostgis.sql.tmp
grep -v '^#' rtpostgis.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql
#strip BEGIN/COMMIT since these are not allowed in extensions
grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql
grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql
rm -f rtpostgis_drop.sql.tmp
grep: rtpostgis.sql.tmp: No such file or directory
rm -f rtpostgis.sql.tmp
grep -v '^#' rtpostgis_drop.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql
grep: /usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in
# WARNING: we should make sure these are not internal to function bodies
rtpostgis_upgrade_cleanup.sql.tmp: No such file or directory
grep -v '^#' rtpostgis_legacy.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql
grep: rtpostgis_upgrade_cleanup.sql.tmp: No such file or directory
/usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp
rm -f rtpostgis.sql.tmp
rm -f rtpostgis_upgrade_cleanup.sql.tmp
grep: rtpostgis_drop.sql.tmp: No such file or directory
Unable to locate target new version number in rtpostgis.sql
cat ../../postgis/common_before_upgrade.sql ../../postgis/postgis_before_upgrade.sql sql/postgis_upgrade_for_extension.sql.in ../../postgis/postgis_after_upgrade.sql ../../postgis/common_after_upgrade.sql | /usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' > sql/postgis_upgrade_for_extension.sql
make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255
make[5]: *** Deleting file 'rtpostgis_upgrade.sql.in'
make[5]: *** Waiting for unfinished jobs....
/usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in
rm -f rtpostgis_upgrade_cleanup.sql.tmp
rm -f rtpostgis_drop.sql.tmp
/usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql
/usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp
rm -f rtpostgis_legacy.sql.tmp
/usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in
/usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql
/usr/bin/perl -pe "s/BEGIN\;//g ; s/COMMIT\;//g; s/^(DROP .*)\;/SELECT postgis_extension_drop_if_exists('postgis', '\1');\n\1\;/" sql/postgis_upgrade_for_extension.sql > sql/postgis_upgrade.sql
Unable to locate target new version number in rtpostgis.sql
Unable to locate target new version number in rtpostgis.sql
make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255
make[5]: *** Deleting file 'rtpostgis_upgrade.sql.in'
make[5]: *** Waiting for unfinished jobs....
make[4]: *** [Makefile:76: ../../raster/rt_pg/rtpostgis_upgrade_cleanup.sql] Error 2
make[4]: *** Deleting file '../../raster/rt_pg/rtpostgis_upgrade_cleanup.sql'
make[4]: *** Waiting for unfinished jobs....
make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255
make[5]: *** Waiting for unfinished jobs....
grep -v '^#' rtpostgis_drop.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql
grep: rtpostgis_drop.sql.tmp: No such file or directory
grep -v '^#' rtpostgis_legacy.sql.tmp | \
/usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql
make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
make[4]: *** [Makefile:76: ../../raster/rt_pg/uninstall_rtpostgis.sql] Error 2
make[4]: *** Deleting file '../../raster/rt_pg/uninstall_rtpostgis.sql'
rm -f rtpostgis_drop.sql.tmp
rm -f rtpostgis_legacy.sql.tmp
make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg'
make[4]: *** [Makefile:76: ../../raster/rt_pg/rtpostgis_drop.sql] Error 2
make[4]: *** Deleting file '../../raster/rt_pg/rtpostgis_drop.sql'
make[4]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/extensions/postgis'
make[3]: *** [Makefile:36: all] Error 1
make[3]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/extensions'
make[2]: *** [GNUmakefile:34: all] Error 1
make[2]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg'
make[1]: *** [debian/rules:129: override_dh_auto_build] Error 2
make[1]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg'

comment:6 by ImreSamu, 3 weeks ago

I tested commit: 4b786d9 ( POSTGIS="3.5.0dev 4b786d9" ) with "make -j2" + CircleCI Arm64 (aarch64) + Bookworm + testing in Docker-PostGIS environment - and Looks OK, no problems detected

#26 50.05 + make -j2
#26 50.06 /usr/bin/perl ./utils/repo_revision.pl
#26 50.07 Wrote rev file at 4b786d9

...

#30 0.512  PostgreSQL 17rc1 (Debian 17~rc1-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
#30 0.512  POSTGIS="3.5.0dev 4b786d9" [EXTENSION] PGSQL="170" GEOS="3.14.0dev-CAPI-1.20.0" SFCGAL="SFCGAL 2.0.0, CGAL 6.0, BOOST 1.74.0" PROJ="9.6.0 NETWORK_ENABLED=OFF URL_ENDPOINT=https://cdn.proj.org USER_WRITABLE_DIRECTORY=/var/lib/postgresql/.local/share/proj DATABASE_PATH=/usr/local/share/proj/proj.db" (compiled against PROJ 9.14.0) GDAL="GDAL 3.10.0dev-9d7f03ea36, released 2024/09/18" LIBXML="2.9.14" LIBJSON="0.16" LIBPROTOBUF="1.4.1" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER

I re-ran 3 Docker images in CircleCI-ARM64, and now all three completed successfully (they failed yesterday):

Of course, this could just be a coincidence, and need more test.

By the way, I also noticed the 3.5.0beta1 issue on Arm64 with (Alpine 3.19, Alpine 3.20, Bookworm).


my build Dockerfile

relevant part/command:

    && git log -1 > /_pgis_last_commit.txt \
    && ldconfig \
    && ./autogen.sh \
# configure options taken from:
# https://anonscm.debian.org/cgit/pkg-grass/postgis.git/tree/debian/rules?h=jessie
    && ./configure \
        --enable-lto \
    && make -j$(nproc) \
    && make install \

comment:7 by sebastic, 3 weeks ago

You can probably make it fail with more parallel processes, on my system I use 11 of the 12 cores.

comment:8 by robe, 3 weeks ago

#5780 might be a duplicate but started seeing it recently so had flagged as a separate.

in reply to:  7 comment:9 by robe, 2 weeks ago

Replying to Bas Couwenberg:

You can probably make it fail with more parallel processes, on my system I use 11 of the 12 cores.

Can you retry with @strk patch he committed at Temptative fix in [be1c2273cdfe0490a43b0148bb8f18ce57a63916/git]

comment:10 by sebastic, 2 weeks ago

Replacing the patch which reverted 5aba8471 with the changes from both 4b786d99 and be1c2273 also seems to work.

Salsa CI builds succeeded (1) as did the buildd jobs (2) on the architectures where they failed before.

comment:11 by robe, 2 weeks ago

Resolution: fixed
Status: newclosed

great thanks for confirming.

Note: See TracTickets for help on using tickets.