Opened 3 months ago
Closed 3 months ago
#5779 closed defect (fixed)
create_upgrade.pl fails for 3.5.0-beta1
Reported by: | sebastic | Owned by: | strk |
---|---|---|---|
Priority: | medium | Milestone: | PostGIS 3.5.0 |
Component: | build | Version: | master |
Keywords: | Cc: |
Description
The Debian package for 3.5.0-beta1 failed to build on some architectures due to parallel build issue:
---- Making all in utils make[3]: Entering directory '/build/postgis/utils' sed -e '1,/^__END__/d' postgis_restore.pl.in \ | sort -u \ > postgis_restore_data.hardcoded /usr/bin/make -C ../raster/rt_pg rtpostgis_upgrade.sql make[4]: Entering directory '/build/postgis/raster/rt_pg' /usr/bin/make -C ../raster/rt_pg uninstall_rtpostgis.sql make[4]: Entering directory '/build/postgis/raster/rt_pg' /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp grep -v '^#' rtpostgis.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp rm -f rtpostgis.sql.tmp /usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql grep -v '^#' rtpostgis.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql make[4]: Leaving directory '/build/postgis/raster/rt_pg' grep -v '^#' rtpostgis_drop.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql rm -f rtpostgis_upgrade_cleanup.sql.tmp grep: rtpostgis.sql.tmp: No such file or directory rm -f rtpostgis.sql.tmp rm -f rtpostgis_drop.sql.tmp /usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in Unable to locate target new version number in rtpostgis.sql make[4]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255 make[4]: *** Deleting file 'rtpostgis_upgrade.sql.in' make[4]: Leaving directory '/build/postgis/raster/rt_pg' make[3]: *** [Makefile:71: ../raster/rt_pg/rtpostgis_upgrade.sql] Error 2 make[3]: Leaving directory '/build/postgis/utils' make[2]: *** [GNUmakefile:28: all] Error 1 make[2]: Leaving directory '/build/postgis' make[1]: *** [debian/rules:129: override_dh_auto_build] Error 2 make[1]: Leaving directory '/build/postgis'
The build failed on amd64 CI (parallel=2), s390x buildd (parallel=2). It did not locally at first with parallel=11 but did the second time, but again not with parallel=1. This suggests a race condition between the Makefile dependencies.
Chaining the commands to create the SQL file seems to help, but that might also just be the race condition:
--- postgis.orig/raster/rt_pg/Makefile.in +++ postgis/raster/rt_pg/Makefile.in @@ -129,9 +129,9 @@ endif # in the extension/ folder # %.sql: %.sql.in - $(SQLPP) -I@top_builddir@/postgis/ $< > $@.tmp + $(SQLPP) -I@top_builddir@/postgis/ $< > $@.tmp && \ grep -v '^#' $@.tmp | \ - $(PERL) -lpe "s'MODULE_PATHNAME'\$(MODULEPATH)'g;s'@extschema@\.''g" > $@ + $(PERL) -lpe "s'MODULE_PATHNAME'\$(MODULEPATH)'g;s'@extschema@\.''g" > $@ && \ rm -f $@.tmp ../../liblwgeom/.libs/liblwgeom.a:
Change History (11)
comment:1 by , 3 months ago
comment:2 by , 3 months ago
As noted by Bas on postgis-dev mailing list
Reverting 5aba84713102a23d8755d1df3f56c277e1dff6b1 seems to resolve the issue, the Salsa CI and buildd builds that failed (more than once) before succeeded now.
Kind Regards,
Bas
comment:3 by , 3 months ago
strk wrote on postgis-dev:
We've upgraded it, Regina updated the ticket with this information and I've further pushed a temptative fix so if you can test that'd be useful ( commit [4b786d9992479041f4bc0cf559cc10e3feca4e2c/git] )
That commit cannot be found on https://git.osgeo.org/gitea/postgis/postgis.git (yet).
comment:4 by , 3 months ago
Now it is, please check (Gitea failed to contact trac due to database connection limit being reached so the ticket wasn't updated)
comment:5 by , 3 months ago
With the changes from 4b786d9992479041f4bc0cf559cc10e3feca4e2c the build fails again:
---- Making all in postgis make[4]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/extensions/postgis' mkdir -p sql /usr/bin/make -C ../../raster/rt_pg sql_objs make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' /usr/bin/make -C ../../raster/rt_pg sql_objs make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' /usr/bin/make -C ../../raster/rt_pg sql_objs make[5]: Entering directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I./../../postgis ../../postgis/postgis.sql.in > sql/postgis_for_extension.sql.tmp /usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' ../../spatial_ref_sys.sql > sql/spatial_ref_sys.sql /usr/bin/perl ../../utils/create_spatial_ref_sys_config_dump.pl ../../spatial_ref_sys.sql > sql/spatial_ref_sys_config_dump.sql grep -v '^#' sql/postgis_for_extension.sql.tmp | \ /usr/bin/perl -lpe \ "s'MODULE_PATHNAME'\$libdir/postgis-3'g" \ | /usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' > sql/postgis_for_extension.sql echo '-- Just tag extension postgis version as "ANY"' > sql/postgis--TEMPLATED--TO--ANY.sql echo '-- Installed by postgis 3.5.0beta1' >> sql/postgis--TEMPLATED--TO--ANY.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp cat postgis.control.in \ | sed -e 's|@EXTVERSION@|3.5.0beta1|g' \ | sed -e 's|@EXTENSION@|postgis|g' \ | sed -e 's|@MODULEPATH@|$libdir/postgis-3|g' \ > postgis.control /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp echo '-- Built on 2024-09-18 14:44:20' >> sql/postgis--TEMPLATED--TO--ANY.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp rm -f sql/postgis_for_extension.sql.tmp /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis.sql.in > rtpostgis.sql.tmp echo "-- Nothing to do here" > sql/postgis--unpackaged.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp printf '\\echo Use "CREATE EXTENSION postgis" to load this file. \\quit\n' > sql/postgis--3.5.0beta1.sql /usr/bin/perl ../../utils/create_upgrade.pl sql/postgis_for_extension.sql > sql/postgis_upgrade_for_extension.sql.in /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp cat sql/postgis_for_extension.sql sql/spatial_ref_sys_config_dump.sql sql/spatial_ref_sys.sql >> sql/postgis--3.5.0beta1.sql grep -v '^#' rtpostgis.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_upgrade_cleanup.sql.in > rtpostgis_upgrade_cleanup.sql.tmp grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql grep -v '^#' rtpostgis_drop.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql grep -v '^#' rtpostgis_legacy.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql grep -v '^#' rtpostgis.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp rm -f rtpostgis_legacy.sql.tmp rm -f rtpostgis_upgrade_cleanup.sql.tmp rm -f rtpostgis.sql.tmp grep -v '^#' rtpostgis.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis.sql #strip BEGIN/COMMIT since these are not allowed in extensions grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql grep -v '^#' rtpostgis_upgrade_cleanup.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_upgrade_cleanup.sql rm -f rtpostgis_drop.sql.tmp grep: rtpostgis.sql.tmp: No such file or directory rm -f rtpostgis.sql.tmp grep -v '^#' rtpostgis_drop.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql grep: /usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in # WARNING: we should make sure these are not internal to function bodies rtpostgis_upgrade_cleanup.sql.tmp: No such file or directory grep -v '^#' rtpostgis_legacy.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql grep: rtpostgis_upgrade_cleanup.sql.tmp: No such file or directory /usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_drop.sql.in > rtpostgis_drop.sql.tmp rm -f rtpostgis.sql.tmp rm -f rtpostgis_upgrade_cleanup.sql.tmp grep: rtpostgis_drop.sql.tmp: No such file or directory Unable to locate target new version number in rtpostgis.sql cat ../../postgis/common_before_upgrade.sql ../../postgis/postgis_before_upgrade.sql sql/postgis_upgrade_for_extension.sql.in ../../postgis/postgis_after_upgrade.sql ../../postgis/common_after_upgrade.sql | /usr/bin/perl -pe 's/BEGIN\;//g ; s/COMMIT\;//g' > sql/postgis_upgrade_for_extension.sql make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255 make[5]: *** Deleting file 'rtpostgis_upgrade.sql.in' make[5]: *** Waiting for unfinished jobs.... /usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in rm -f rtpostgis_upgrade_cleanup.sql.tmp rm -f rtpostgis_drop.sql.tmp /usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql /usr/bin/cpp -traditional-cpp -w -P -Upixel -Ubool -I../../postgis/ rtpostgis_legacy.sql.in > rtpostgis_legacy.sql.tmp rm -f rtpostgis_legacy.sql.tmp /usr/bin/perl ../../utils/create_upgrade.pl rtpostgis.sql > rtpostgis_upgrade.sql.in /usr/bin/perl ../../utils/create_uninstall.pl rtpostgis.sql 170 > uninstall_rtpostgis.sql /usr/bin/perl -pe "s/BEGIN\;//g ; s/COMMIT\;//g; s/^(DROP .*)\;/SELECT postgis_extension_drop_if_exists('postgis', '\1');\n\1\;/" sql/postgis_upgrade_for_extension.sql > sql/postgis_upgrade.sql Unable to locate target new version number in rtpostgis.sql Unable to locate target new version number in rtpostgis.sql make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255 make[5]: *** Deleting file 'rtpostgis_upgrade.sql.in' make[5]: *** Waiting for unfinished jobs.... make[4]: *** [Makefile:76: ../../raster/rt_pg/rtpostgis_upgrade_cleanup.sql] Error 2 make[4]: *** Deleting file '../../raster/rt_pg/rtpostgis_upgrade_cleanup.sql' make[4]: *** Waiting for unfinished jobs.... make[5]: *** [Makefile:156: rtpostgis_upgrade.sql.in] Error 255 make[5]: *** Waiting for unfinished jobs.... grep -v '^#' rtpostgis_drop.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_drop.sql grep: rtpostgis_drop.sql.tmp: No such file or directory grep -v '^#' rtpostgis_legacy.sql.tmp | \ /usr/bin/perl -lpe "s'MODULE_PATHNAME'\$libdir/postgis_raster-3'g;s'@extschema@\.''g" > rtpostgis_legacy.sql make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' make[4]: *** [Makefile:76: ../../raster/rt_pg/uninstall_rtpostgis.sql] Error 2 make[4]: *** Deleting file '../../raster/rt_pg/uninstall_rtpostgis.sql' rm -f rtpostgis_drop.sql.tmp rm -f rtpostgis_legacy.sql.tmp make[5]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/raster/rt_pg' make[4]: *** [Makefile:76: ../../raster/rt_pg/rtpostgis_drop.sql] Error 2 make[4]: *** Deleting file '../../raster/rt_pg/rtpostgis_drop.sql' make[4]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/extensions/postgis' make[3]: *** [Makefile:36: all] Error 1 make[3]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg/extensions' make[2]: *** [GNUmakefile:34: all] Error 1 make[2]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg' make[1]: *** [debian/rules:129: override_dh_auto_build] Error 2 make[1]: Leaving directory '/build/postgis-3.5.0~beta1+dfsg'
comment:6 by , 3 months ago
I tested commit: 4b786d9 ( POSTGIS="3.5.0dev 4b786d9"
) with "make -j2" + CircleCI Arm64 (aarch64) + Bookworm + testing in Docker-PostGIS environment - and Looks OK, no problems detected
#26 50.05 + make -j2 #26 50.06 /usr/bin/perl ./utils/repo_revision.pl #26 50.07 Wrote rev file at 4b786d9 ... #30 0.512 PostgreSQL 17rc1 (Debian 17~rc1-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit #30 0.512 POSTGIS="3.5.0dev 4b786d9" [EXTENSION] PGSQL="170" GEOS="3.14.0dev-CAPI-1.20.0" SFCGAL="SFCGAL 2.0.0, CGAL 6.0, BOOST 1.74.0" PROJ="9.6.0 NETWORK_ENABLED=OFF URL_ENDPOINT=https://cdn.proj.org USER_WRITABLE_DIRECTORY=/var/lib/postgresql/.local/share/proj DATABASE_PATH=/usr/local/share/proj/proj.db" (compiled against PROJ 9.14.0) GDAL="GDAL 3.10.0dev-9d7f03ea36, released 2024/09/18" LIBXML="2.9.14" LIBJSON="0.16" LIBPROTOBUF="1.4.1" WAGYU="0.5.0 (Internal)" TOPOLOGY RASTER
I re-ran 3 Docker images in CircleCI-ARM64, and now all three completed successfully (they failed yesterday):
- build-arm64-17rc1-master-bookworm
- build-arm64-16-master-bookworm
- build-arm64-15-master-bookworm
Of course, this could just be a coincidence, and need more test.
By the way, I also noticed the 3.5.0beta1 issue on Arm64 with (Alpine 3.19, Alpine 3.20, Bookworm).
my build Dockerfile
relevant part/command:
&& git log -1 > /_pgis_last_commit.txt \ && ldconfig \ && ./autogen.sh \ # configure options taken from: # https://anonscm.debian.org/cgit/pkg-grass/postgis.git/tree/debian/rules?h=jessie && ./configure \ --enable-lto \ && make -j$(nproc) \ && make install \
follow-up: 9 comment:7 by , 3 months ago
You can probably make it fail with more parallel processes, on my system I use 11 of the 12 cores.
comment:8 by , 3 months ago
#5780 might be a duplicate but started seeing it recently so had flagged as a separate.
comment:9 by , 3 months ago
Replying to Bas Couwenberg:
You can probably make it fail with more parallel processes, on my system I use 11 of the 12 cores.
Can you retry with @strk patch he committed at Temptative fix in [be1c2273cdfe0490a43b0148bb8f18ce57a63916/git]
Can you still reproduce reverting [5aba84713102a23d8755d1df3f56c277e1dff6b1/git] ?
I saw utils/Makefile has targets to build thing under raster/ which I think was meant to be used when raster is disabled. Those utils/ rules should do nothing if everything under raster/ is already built, and the commit I'm referencing above is building raster/ *after* utils/