Regress failure with in-develop proj

[robe]

Our latest GHA which tests not yet released PROJ has a regression failur

 ./regress/core/regress_proj_cache_overflow .. failed (diff expected obtained: /tmp/pgis_reg/test_60_diff)
-----------------------------------------------------------------------------
--- ./regress/core/regress_proj_cache_overflow_expected	2023-01-19 05:30:28.000000000 +0000
+++ /tmp/pgis_reg/test_60_out	2023-01-19 05:31:57.217181967 +0000
@@ -1 +1 @@
-13|147
+ERROR:  transform: File not found or invalid (1029)
-----------------------------------------------------------------------------
 ./regress/core/regress_proj_4890 .. failed (diff expected obtained: /tmp/pgis_reg/test_61_diff)
-----------------------------------------------------------------------------
--- ./regress/core/regress_proj_4890_expected	2023-01-19 05:30:28.000000000 +0000
+++ /tmp/pgis_reg/test_61_out	2023-01-19 05:31:57.293182578 +0000
@@ -1 +1 @@
-4890|POINT(-122.4859 49.4531)
+ERROR:  transform: File not found or invalid (1029)

[robe]

Forgot to add the details of what latest/latest gha is running:

PostgreSQL 16devel on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
  Postgis 3.4.0dev - (d03aa1b) - 2023-01-19 05:38:04
  scripts 3.4.0dev d03aa1b
  raster scripts 3.4.0dev d03aa1b
  GEOS: 3.12.0dev-CAPI-1.18.0
  PROJ: 9.2.0
  SFCGAL: 1.4.1
  GDAL: GDAL 3.7.0dev-5ce7ba068f, released 2023/01/17

I'm going to check the job that builds this. Perhaps we are missing something during install of grid files.

[ImreSamu]

I have a similar problem with the ​https://github.com/postgis/docker-postgis project.

and an extra line is solving the test.

projsync --system-directory --all

but then the postgis/postgis:15-master image size

increasing from 555MB ——> ~ 1.28GB

( so I need a less space-consuming solution / workaround . )

[ImreSamu]

current workaround

    && projsync --system-directory --source-id us_noaa \
    && projsync --system-directory --source-id ch_swisstopo \

​https://github.com/postgis/docker-postgis/pull/337/commits/a7eec62d33b26f6ddd0627056b3518811b67210f

still not perfect : 555 MB —> 815MB

[robe]

Having same issue on buildenv. I mirrored the whole cdn.proj.org like I saw in the proj script, but that still didn't work.

I'll switch to your approach. Thanks, Regina

[robe]

Okay I switched our buildenv docker to use projsync and that fixed the issue but as you said there is still the issue of the extra size.

@pramsey suggested we just get rid of those tests. I'm willing to do that, but I guess I'll need to compile 9.2 proj branch now that our bots are all fixed to make sure I remove the right amount.

I'd hate to completely remove the regress_proj_cache_overflow

The other failing one regress_proj_4890 was designed to test #4890

I think that wone we can maybe replace with a different srid. I assume there is nothing special about the 3005 that was picked for that test.

[ImreSamu]

issue of the extra size.

My latest solution only adds +1 MB to the docker image size. I think this is an acceptable compromise. And it might avoid the removal of these tests. I think it would also make sense to include it in the postgis installation steps ? ( or the regression testing steps ?? ) And then I don't have to issue these commands separately in the Dockerfile.

# refresh proj data - workarounds: https://trac.osgeo.org/postgis/ticket/5316
    && projsync --system-directory --file ch_swisstopo_CHENyx06_ETRS \
    && projsync --system-directory --file us_noaa_eshpgn \
    && projsync --system-directory --file us_noaa_prvi \
    && projsync --system-directory --file us_noaa_wmhpgn \

​https://github.com/postgis/docker-postgis/pull/337/commits/0309004b39514b5ba9e8ff116998248782de0fd6

And the test is OK : ​https://github.com/postgis/docker-postgis/pull/337

[ImreSamu]

The latest Alpine Linux 3.18 includes PROJ 9.2. Therefore, I ( as a ​https://github.com/postgis/docker-postgis contributor ) need to apply a minimal (+1MB ) patch to the Docker postgis/postgis:1x-alpine images to pass the regression test.

• ​https://github.com/postgis/docker-postgis/pull/343

Alpine Linux 3.18 - released 2023-05-09

• ​https://www.alpinelinux.org/posts/Alpine-3.18.0-released.html

[Bas Couwenberg]

The grids are not available in Debian where postgis (3.3.3+dfsg-1) fails to build with proj (9.2.1-1), the package build environment also doesn't have network which prevents downloading the grids from the CDN.

Ideally there is a subset of tests which require network that can easily be skipped in environments where the network is not available.

[Paul Ramsey <pramsey@…>]

In 123a3dc/git:

Only test projections that use a WGS84 GEOGCS to avoid
exercising the grid shift system in this test.
References #5316

[Paul Ramsey <pramsey@…>]

In 8587a62/git:

Change test to harmonize GEOGCS entries and hopefully
avoid exercising the Proj gridshift system.
References #5316

[Paul Ramsey <pramsey@…>]

In f371d92/git:

Try to only use boring srid numbers for cache busting test, references #5316

[Regina Obe <lr@…>]

In cf7df4ca/git:

remove not boring enough srid. References #5316 for PostGIS 3.4.0

[robe]

That last change fixed the proj 8 failures, but sadly proj 9.0 and higher are still failing. So I guess there is at least one more of those. I'll compile against proj 9 to see if I can root out that last one.

[robe]

Oh wait proj8 is still failing on winnie. Not sure why it isn't failing on my setup when it should be identical to winnie's.

[Regina Obe <lr@…>]

In b038892/git:

Comment out srids that cause
"transform: Point outside of projection domain (2050)"
References #5316

[Regina Obe <lr@…>]

In b620d70/git:

Fix for proj failures
References #5316 for PostGIS 3.3.10

[robe]

Sadly changes did not fix 3.3.10, because PROJ 5.2 still errors out.

​https://github.com/postgis/postgis/actions/runs/5312510774/jobs/9617110243

 ./regress/core/regress_proj_cache_overflow .. failed (diff expected obtained: /tmp/pgis_reg/test_61_diff)
-----------------------------------------------------------------------------
--- ./regress/core/regress_proj_cache_overflow_expected	2023-06-19 13:30:42.000000000 +0000
+++ /tmp/pgis_reg/test_61_out	2023-06-19 13:32:56.056377261 +0000
@@ -1 +1 @@
-13|112
+ERROR:  transform: latitude or longitude exceeded limits (-14)
-----------------------------------------------------------------------------

[robe]

same failure on proj 4.9 ​https://dronie.osgeo.org/postgis/postgis/4308/1/3

PostgreSQL 13.8 (Ubuntu 13.8-1.pgdg18.04+1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit

  Postgis 3.3.4dev - (0) - 2023-06-19 13:33:09

  scripts 3.3.4dev 0

  raster scripts 3.3.4dev 0

  GEOS: 3.6.5-CAPI-1.10.5 0

  PROJ: Rel. 4.9.3, 15 August 2016

  SFCGAL: 1.3.5

  GDAL: GDAL 2.4.4, released 2020/01/08

[strk]

Environment variables used by libproj are here: ​https://proj.org/en/9.2/usage/environmentvars.html

There seem to be a variable to disable network and one to set proj data. For the sake of tests predictability I think we should set network off and point PROJ_DATA to an empty directory or similar

[Regina Obe <lr@…>]

In 2d77093/git:

Remove entries that cause
errors in Proj 5.2 and 4.9
References #5316 for PostGIS 3.3.4

Namely: transform: latitude or longitude exceeded limits (-14)

transform the point because either no grid shift files were found

failed to load datum shift file (-38)

[laurencekobold]

I'm still sometimes seeing this error when running the official Alpine postgis docker image. For example, the following statement will produce the error:

SELECT ST_Transform(ST_SetSRID(ST_MakePoint(355023.6737367718, 5401790.6769562615),26916), 4326);

Oddly, the statement will succeed if run a second time in the same connection. This error is not present when moving to the Debian postgis docker image.

docker image version: postgis/​postgis:16-3.4-alpine postgres version: PostgreSQL 16.0 on x86_64-pc-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r10) 12.2.1 20220924, 64-bit postgis version: POSTGIS="3.4.0 0" [EXTENSION] PGSQL="160" GEOS="3.11.2-CAPI-1.17.2" PROJ="9.2.1 NETWORK_ENABLED=OFF URL_ENDPOINT=​https://cdn.proj.org USER_WRITABLE_DIRECTORY=/var/lib/postgresql/.local/share/proj DATABASE_PATH=/usr/share/proj/proj.db" LIBXML="2.11.4" LIBJSON="0.16" LIBPROTOBUF="1.4.1" WAGYU="0.5.0 (Internal)"