Opened 21 months ago

Closed 18 months ago

Last modified 18 months ago

#5370 closed defect (fixed)

bessie regress failures on raster after upgrade to GDAL 3.6.1 and Proj 9.2.0

Reported by: robe Owned by: robe
Priority: blocker Milestone: PostGIS 3.4.0
Component: QA/buildbots Version: 3.3.x
Keywords: Cc:

Description

To get around #5368, I took off the cirrus FreeBSD 12.3 testing and upgraded bessie so she's testing more or less same versions of things. Though cirrus was never testing raster. I think we had disabled raster testing on cirrus a while ago cause it was giving mysterious errors.

But anyway after the upgrade surgery, bessie is in tears https://debbie.postgis.net/view/PostGIS/job/PostGIS_Worker_Run/4437/label=bessie/console

with these errors which might just be a postgres account issue that somehow the context changed and the account running under doesn't have rights to ENABLE_ALL / DISABLE_ALL anymore

20:32:28  raster/test/regress/check_gdal .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_230_diff)
20:32:28 -----------------------------------------------------------------------------
20:32:28 --- ./raster/test/regress/check_gdal_expected	2023-04-14 20:25:42.515822000 -0400
20:32:28 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_230_out	2023-04-14 20:32:28.456935000 -0400
20:32:28 @@ -3,7 +3,9 @@
20:32:28  DISABLE_ALL
20:32:28  ENABLE_ALL
20:32:28  t
20:32:28 +WARNING:  Unknown GDAL driver: PNG
20:32:28 +WARNING:  Unknown GDAL driver: JPEG
20:32:28  GTiff PNG JPEG
20:32:28 -t
20:32:28 +f
20:32:28  DISABLE_ALL
20:32:28  t
20:32:28 -----------------------------------------------------------------------------

20:32:49  raster/test/regress/rt_fromgdalraster .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_275_diff)
20:32:49 -----------------------------------------------------------------------------
20:32:49 --- ./raster/test/regress/rt_fromgdalraster_expected	2023-04-14 20:25:42.529874000 -0400
20:32:49 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_275_out	2023-04-14 20:32:49.627946000 -0400
20:32:49 @@ -1,3 +1,4 @@
20:32:49 -(0,0,90,90,1,-1,0,0,0,3)|(8100,1479000,182.592592592593,114.982851945091,0,255)|(8100,1479000,182.592592592593,114.982851945091,0,255)|(8100,1453500,179.444444444444,116.438931167192,0,255)
20:32:49 +WARNING:  Unknown GDAL driver: PNG
20:32:49 +ERROR:  RASTER_fromGDALRaster: Could not open bytea with GDAL. Check that the bytea is of a GDAL supported format
20:32:49  1|(-168,85,10,10,0.083,-0.083,0,0,4269,3)|(100,0,0,0,0,0)|(100,0,0,0,0,0)|(100,0,0,0,0,0)
20:32:49  2|(-168,85,10,10,0.083,-0.083,0,0,4326,3)|(100,0,0,0,0,0)|(100,0,0,0,0,0)|(100,0,0,0,0,0)
20:32:49 -----------------------------------------------------------------------------
20:32:49  raster/test/regress/rt_asgdalraster .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_276_diff)
20:32:50 -----------------------------------------------------------------------------
20:32:50 --- ./raster/test/regress/rt_asgdalraster_expected	2023-04-14 20:25:42.525932000 -0400
20:32:50 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_276_out	2023-04-14 20:32:49.988208000 -0400
20:32:50 @@ -1,16 +1,18 @@
20:32:50 +WARNING:  Unknown GDAL driver: PNG at character 49
20:32:50 +WARNING:  Unknown GDAL driver: JPEG at character 49
20:32:50  0
20:32:50  1
20:32:50  1
20:32:50  1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 -1
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 -----------------------------------------------------------------------------
20:32:50  raster/test/regress/rt_astiff .. ok in 384 ms
20:32:50  raster/test/regress/rt_asjpeg .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_278_diff)
20:32:50 -----------------------------------------------------------------------------
20:32:50 --- ./raster/test/regress/rt_asjpeg_expected	2023-04-14 20:25:42.526029000 -0400
20:32:50 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_278_out	2023-04-14 20:32:50.753779000 -0400
20:32:50 @@ -1,10 +1,11 @@
20:32:50 +WARNING:  Unknown GDAL driver: JPEG at character 18
20:32:50  t
20:32:50  ERROR:  The pixel type of band 1 in the raster is not 8BUI.  The JPEG format can only be used with the 8BUI pixel type.
20:32:50 -1
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50  ERROR:  The pixel type of band 1 in the raster is not 8BUI.  The JPEG format can only be used with the 8BUI pixel type.
20:32:50 -1
20:32:50 -1
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50  ERROR:  The pixel type of band 1 in the raster is not 8BUI.  The JPEG format can only be used with the 8BUI pixel type.
20:32:50  NOTICE:  The JPEG format only permits one or three bands.  The first band will be used.
20:32:50  ERROR:  The pixel type of band 1 in the raster is not 8BUI.  The JPEG format can only be used with the 8BUI pixel type.
20:32:50 -1
20:32:50 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:50 -----------------------------------------------------------------------------
20:32:50  raster/test/regress/rt_aspng .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_279_diff)
20:32:51 -----------------------------------------------------------------------------
20:32:51 --- ./raster/test/regress/rt_aspng_expected	2023-04-14 20:25:42.526326000 -0400
20:32:51 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_279_out	2023-04-14 20:32:51.109555000 -0400
20:32:51 @@ -1,11 +1,12 @@
20:32:51 +WARNING:  Unknown GDAL driver: PNG at character 17
20:32:51  t
20:32:51  ERROR:  The pixel type of band 1 in the raster is not 8BUI or 16BUI.  The PNG format can only be used with 8BUI and 16BUI pixel types.
20:32:51 -1
20:32:51 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:51  ERROR:  The pixel type of band 1 in the raster is not 8BUI or 16BUI.  The PNG format can only be used with 8BUI and 16BUI pixel types.
20:32:51 -1
20:32:51 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:51  ERROR:  The pixel type of band 1 in the raster is not 8BUI or 16BUI.  The PNG format can only be used with 8BUI and 16BUI pixel types.
20:32:51  ERROR:  The pixel type of band 1 in the raster is not 8BUI or 16BUI.  The PNG format can only be used with 8BUI and 16BUI pixel types.
20:32:51  ERROR:  The pixel type of band 1 in the raster is not 8BUI or 16BUI.  The PNG format can only be used with 8BUI and 16BUI pixel types.
20:32:51 -1
20:32:51 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:51  NOTICE:  The PNG format only permits one, three or four bands.  The first band will be used.
20:32:51 -1
20:32:51 +ERROR:  rt_raster_to_gdal: Could not load the output GDAL driver
20:32:51 -----------------------------------------------------------------------------

20:33:21  raster/test/regress/permitted_gdal_drivers .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_311_diff)
20:33:22 -----------------------------------------------------------------------------
20:33:22 --- ./raster/test/regress/permitted_gdal_drivers_expected	2023-04-14 20:25:42.525312000 -0400
20:33:22 +++ /home/jenkins/tmp/pgis_reg_5da0ec6e80a4a1e414433e79289482c35ef32474/test_311_out	2023-04-14 20:33:21.981975000 -0400
20:33:22 @@ -1,3 +1,5 @@
20:33:22 +WARNING:  Unknown GDAL driver: PNG
20:33:22 +WARNING:  Unknown GDAL driver: JPEG
20:33:22  4326
20:33:22  ERROR:  RASTER_fromGDALRaster: Could not open bytea with GDAL. Check that the bytea is of a GDAL supported format
20:33:22  ERROR:  rt_band_load_offline_data: Cannot open offline raster: <VRTDataset rasterXSize="1000" rasterYSize="1"><VRTRasterBand band="1" subClass="VRTRawRasterBand"><SourceFilename>/etc/passwd</SourceFilename></VRTRasterBand></VRTDataset>
20:33:22 -----------------------------------------------------------------------------

Before she was running with:

 -------------- Compiler Info ------------- 
23:11:05   C compiler:           cc -std=gnu99 -g -O2 -fno-math-errno -fno-signed-zeros -Wall
23:11:05   C++ compiler (Wagyu): cc -std=c++11 -x c++ 
23:11:05   C++ compiler (FlatGeobuf): cc -std=c++11 -x c++ 
23:11:05   CPPFLAGS:              -I/usr/local/include -I/usr/local/include -I/usr/local/include  -I/usr/local/include/libxml2 -I/usr/local/include -I/usr/local/include -I/usr/local/include/json-c  -I/usr/local/include  -DNDEBUG 
23:11:05   LDFLAGS:               -lm
23:11:05   SQL preprocessor:     /usr/bin/cpp -traditional-cpp -w -P
23:11:05   Archiver:             llvm-ar rs
23:11:05 
23:11:05  -------------- Additional Info ------------- 
23:11:05   POSTGIS lib file includes minor:   ENABLED
23:11:05   Interrupt Tests:   DISABLED use: --with-interrupt-tests to enable
23:11:05 
23:11:05  -------------- Dependencies -------------- 
23:11:05   GEOS config:          /usr/local/bin/geos-config
23:11:05   GEOS version:         3.10.3
23:11:05   GDAL config:          /usr/local/bin/gdal-config
23:11:05   GDAL version:         3.5.0
23:11:05   SFCGAL config:        /usr/local/bin/sfcgal-config
23:11:05   SFCGAL version:       1.4.1
23:11:05   PostgreSQL config:    /usr/local/bin/pg_config
23:11:05   PostgreSQL version:   PostgreSQL 13.7
23:11:05   PROJ4 version:        82
23:11:05   Libxml2 config:       /usr/local/bin/xml2-config
23:11:05   Libxml2 version:      2.9.13
23:11:05   JSON-C support:       yes
23:11:05   protobuf support:     yes
23:11:05   protobuf-c version:   1004000
23:11:05   PCRE support:         Version 2
23:11:05   Perl:                 /usr/local/bin/perl
23:11:05 
23:11:05  --------------- Extensions --------------- 
23:11:05   PostgreSQL EXTENSION support:       enabled
23:11:05   PostGIS Raster:                     enabled
23:11:05   PostGIS Topology:                   enabled
23:11:05   SFCGAL support:                     enabled
23:11:05   Address Standardizer support:       enabled
23:11:05 
23:11:05  -------- Documentation Generation -------- 
23:11:05   xsltproc:             /usr/local/bin/xsltproc
23:11:05   xsl style sheets:     
23:11:05   dblatex:              
23:11:05   convert:              /usr/local/bin/convert
23:11:05   mathml2.dtd:          http://www.w3.org/Math/DTD/mathml2/mathml2.dtd
23:11:05 

and now she is running:

20:26:02  -------------- Dependencies -------------- 
20:26:02   GEOS config:          /usr/local/bin/geos-config
20:26:02   GEOS version:         3.11.1
20:26:02   GDAL config:          /usr/local/bin/gdal-config
20:26:02   GDAL version:         3.6.2
20:26:02   SFCGAL config:        /usr/local/bin/sfcgal-config
20:26:02   SFCGAL version:       1.4.1
20:26:02   PostgreSQL config:    /usr/local/bin/pg_config
20:26:02   PostgreSQL version:   PostgreSQL 13.10
20:26:02   PROJ4 version:        92
20:26:02   Libxml2 config:       /usr/local/bin/xml2-config
20:26:02   Libxml2 version:      2.10.3
20:26:02   JSON-C support:       yes
20:26:02   protobuf support:     yes
20:26:02   protobuf-c version:   1004001
20:26:02   PCRE support:         Version 2
20:26:02   Perl:                 /usr/local/bin/perl

Change History (7)

comment:1 by lbartoletti, 21 months ago

"unknown driver" is caused by missing dependency for GDAL. It's fixed now, cf: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269708

comment:2 by robe, 20 months ago

Thanks. At a glance it looks like that patch might also fix the issue I am seeing with most of the drivers missing.

The regression failures I traced back to missing drivers and not permission issue as I had originally though.

For example doing this:

SET postgis.gdal_enabled_drivers = 'ENABLE_ALL';
SELECT short_name, long_name FROM ST_GDALDrivers();

Just lists:

 short_name |                 long_name
------------+--------------------------------------------
 VRT        | Virtual Raster
 DERIVED    | Derived datasets using VRT pixel functions
 GTiff      | GeoTIFF
 COG        | Cloud optimized GeoTIFF generator
 HFA        | Erdas Imagine Images (.img)
 MEM        | In Memory Raster
(6 rows)

I did a

pkg update && pkg upgrade

and only thing that got upgraded was sqlite. So I'm assuming that patch hasn't been pushed out yet.

comment:3 by Regina Obe <lr@…>, 20 months ago

In 1fc0c62f/git:

Turn of raster testing on bessie
Need to wait till
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=269708
is available via pkg
References #5370 for PostGIS 3.4.0

comment:4 by lbartoletti, 19 months ago

Did you use quarterly or latest for pkg? Cf /etc/pkg/FreeBSD.conf

To enable latest:

mkdir -p /usr/local/etc/pkg/repos
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
sed -e "s/quarterly/latest/" /etc/pkg/FreeBSD.conf > /usr/local/etc/pkg/repos/FreeBSD_latest.conf

comment:5 by robe, 18 months ago

@lbartoletti

Sorry didn't see your note until I did some more things. But I think it's all resolved now.

My /usr/local/etc/pkg/repos/FreeBSD.conf

looks like:

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
  mirror_type: "srv",
  signature_type: "fingerprints",
  fingerprints: "/usr/share/keys/pkg",
  enabled: yes
}

I used ports instead

portsnap fetch extract #first time 
# there after
portsnap fetch update
export ALLOW_UNSUPPORTED_SYSTEM=yes
cd /usr/ports/graphics/gdal/ && make install clean
make config # to change options

And unchecked some of the things like MySQL, ODBC etc, just keeping the basics.

That seemed to work fine and gave me GDAL 3.7.1

Last run of 3.2 branch looked good and had the below. I'm going to renable raster testing on bessie now on the 3.3 and master that I had disabled it on.

llvmorg-13.0.0-0-gd7b669b3a303), 64-bit
12:53:15   Postgis 3.2.6dev - (59a8f9d) - 2023-06-20 16:49:51
12:53:15   scripts 3.2.6dev 59a8f9d
12:53:15   raster scripts 3.2.6dev 59a8f9d
12:53:15   GEOS: 3.11.1-CAPI-1.17.1
12:53:15   PROJ: 9.2.0
12:53:15   SFCGAL: 1.4.1
12:53:15   GDAL: GDAL 3.7.0, released 2023/05/02

comment:6 by robe, 18 months ago

Resolution: fixed
Status: newclosed

comment:7 by Regina Obe <lr@…>, 18 months ago

In ae1b629/git:

enable raster testing on bessie. References #5370

Note: See TracTickets for help on using tickets.