Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#957 closed defect (fixed)

[raster] rt_asgdalraster regress test fails

Reported by: strk Owned by: Bborie Park
Priority: medium Milestone: PostGIS 2.0.0
Component: raster Version: master
Keywords: Cc:

Description (last modified by strk)

--- rt_asgdalraster_expected    2011-05-17 09:50:39.000000000 +0200
+++ /tmp/pgis_reg_9436/test_10_out      2011-05-17 17:05:26.000000000 +0200
@@ -11,5 +11,5 @@
 f0de16a21de438249ec0bbd28eafe63c
 ed4e4d60d2ccc82c4597e80c1f97272f
 83b6012757444ff7786b5e8473e6cea3
-b699be9f7045656edbb14ecec1c75b5a
-96a96f9a6d31544dfc5ba5cd25f40934
+f6fa7f8ae3c35c137a75acc855255083
+7031666f7a668577b9de9cc6a8799f75

The above is with PostgreSQL 8.4, PostGIS r7171, GDAL 1.9.0 revision 21597 on a 64bit system.

Attachments (2)

asgdalraster-dump.tar.bz2 (21.0 KB) - added by Bborie Park 9 years ago.
ST_AsGDALRaster outputs on 32-bit Linux system with variety of different GDAL versions
asgdalraster-dump-64.tar.bz2 (19.3 KB) - added by Bborie Park 9 years ago.
64-bit GDAL raster outputs from regression queries for ST_AsGDALRaster

Download all attachments as: .zip

Change History (30)

comment:1 Changed 9 years ago by Bborie Park

Summary: raster regress errors, SELECT and some gdal[raster] raster regress errors, SELECT and some gdal

The regression for rt_asgdalraster_expection is what concerns me. I'll have to test with GDAL 1.9.0 to see if that changed the output. My dev environment is PostgreSQL 9.0, PostGIS r7171, GDAL 1.8.0 on a 64-bit system.

And it would be nice if we could strip the "SELECT" lines out.

comment:2 Changed 9 years ago by strk

Summary: [raster] raster regress errors, SELECT and some gdal[raster] rt_asgdalraster regress test fails

With r7173 only the GDAL one remains ("SELECT" stripped, only raster regression was having selects in output...)

comment:3 Changed 9 years ago by Bborie Park

Which revision of GDAL 1.9.0 are you using?

comment:4 Changed 9 years ago by strk

It's probably revision 21597 (checked from source tree, not installed binary, so not 100% sure).

comment:5 Changed 9 years ago by strk

Description: modified (diff)

comment:6 Changed 9 years ago by Bborie Park

This is very strange. After testing the regress script on PostgreSQL 9.0.3, PostGIS r7192 and GDAL 1.9.0 (using the daily snapshot http://www.gdal.org/daily/gdal-svn-trunk-2011.05.01.tar.gz), the test fails on exactly those two queries. I may redo those two queries as the raster generated from them should be invalid as the raster used has bands of 8BSI rather than 8BUI.

comment:7 Changed 9 years ago by Bborie Park

So, even with the queries corrected to be 8BUI for output using JPEG compression, the regression test fails with the same hashes. So, the pixel type isn't the problem.

Is there any way you can try the regression test with GDAL 1.8.0? I'd like to test the idea that the issue may be with changes in GDAL between version, which is what I'm seeing.

comment:8 Changed 9 years ago by strk

I don't have GDAL 1.8.0 around, sorry. But if you want I can test 1.7.3 on a 32bit syste.

comment:9 Changed 9 years ago by Bborie Park

Can you test it on 1.7.3? I'll need to set myself up a 32-bit test system. I may need to end up testing all GDAL versions from 1.6.0 and see what happens.

comment:10 Changed 9 years ago by strk

ouch, the 32bit system suffer from the segfault reported in ticket #958. Better focus on that one first, it is much worst anyway...

comment:11 Changed 9 years ago by strk

Ok, just running the single test on 32bit with GDAL 1.7.3:

--- raster/test/regress/rt_asgdalraster_expected        2011-05-17 20:39:28.000000000 +0200
+++ /tmp/pgis_reg_11760/test_1_out      2011-05-18 16:24:07.000000000 +0200
@@ -5,8 +5,8 @@
 55279950e29968bcf36b2c11ce8bf88b
 4ae84e77caa1a35aa4da233e83feb746
 24188762b5745acda4aa7b92776e7280
-da3c4e827b617f9d7cd83d5759f1d781
-0d6d88030359474c2e86280a65e5e90a
+b0fb3339eb96331e7ba55a379c077f38
+b0fb3339eb96331e7ba55a379c077f38
 f0de16a21de438249ec0bbd28eafe63c
 f0de16a21de438249ec0bbd28eafe63c
 ed4e4d60d2ccc82c4597e80c1f97272f

comment:12 Changed 9 years ago by Bborie Park

I don't know what I can do about the crashing in #958 unless Jorge doesn't mind someone else taking a look. I hesitate to dig in as r7106 is the big raster memory management refactoring...

comment:13 Changed 9 years ago by strk

Oops, sorry then, dunno why I tought you were involved in it. Luckly you can invoke run_test passing a single test to it ...

comment:14 Changed 9 years ago by jorgearevalo

Please, update to r7202 before trying it again. Anyway, as Sandro said, you can execute any test individually.

comment:15 Changed 9 years ago by Bborie Park

Updated to r7202. ST_AsGDALRaster fails for me. I get the exact same "new" hashes as in your test 10 diff in 32-bit linux with GDAL 1.8.0. I'm going to have to dump out the rasters themselves and double-check them with GDAL to make sure that the outputs are correct.

comment:16 Changed 9 years ago by strk

Alright, now that the segfault is fixed I get another two hash failures: RT_AsPNG and RT_AsJPEG. Do you want another ticket for each ?

comment:17 Changed 9 years ago by Bborie Park

Probably not as both hook into the same function underneath as ST_AsGDALRaster.

Changed 9 years ago by Bborie Park

Attachment: asgdalraster-dump.tar.bz2 added

ST_AsGDALRaster outputs on 32-bit Linux system with variety of different GDAL versions

comment:18 Changed 9 years ago by Bborie Park

I've attached the raster outputs from the ST_AsGDALRaster regression test using four different versions of GDAL (1.6.3, 1.7.3, 1.8.0, 1.9.0 r22405). Based upon the outputs, their md5 checksums and running "gdalinfo -stats -checksum RASTERFILE", I new another way to verify that the GDAL raster outputs are valid. As you can see in the checksum.md5, the same query can generate different checksums across different versions.

The ideal solution is to validate the output raster using gdalinfo as that showed that though the rasters of one query had different md5 checksums, the actual stats and per-band checksums are correct. I will be sending the postgis-devel list a question to see if the regression tests for ST_AsGDALRaster, ST_AsTIFF, ST_AsPNG and ST_AsJPEG can be done in the same manner as regress/test_index_concurrency, a bash shell script.

If there are any suggestions, let me know.

comment:19 Changed 9 years ago by Bborie Park

Status: newassigned

Between 32 and 64 bit systems, the regression raster outputs from GDAL have no differences. I've attached the output from a pure 64-bit linux system with different GDAL versions.

Changed 9 years ago by Bborie Park

64-bit GDAL raster outputs from regression queries for ST_AsGDALRaster

comment:20 Changed 9 years ago by robe

Just to add to this. With a bit of fussing and doing a second make check by ccding itno raster folder (and copying some missing dlls in my mingw compile), I wwas able to get my pg 8.4.4 windows VC++ build to pass most of the tests.

Failing 4 though -- this is with libgdal 1.8.0 rtm version.

rt_asgdalraster, rt_aspng, rt_asjpeg, rt_spatial_relationship

Are these the 4 others are having issue with? This is on a 32-bit windows 7 box.

comment:21 Changed 9 years ago by Bborie Park

Please try r7235, which should resolve the rt_asgdalraster, rt_aspng and rt_asjpeg issues. I've refactored the existing regression tests from attempting to validate the contents returned from the respective functions to checking that the functionality is correct.

A new script testgdalraster ("make testgdalraster" in raster/test/regress) has been added to test the output of rt_asgdalraster and related functions. As the testgdalraster depends upon python (specifically raster/scripts/python/rtgdalraster.py), you'll need to have psycopg2 installed. The testgdalraster scripts uses the gdalinfo utility to validate the output rasters band checksums. In testing on both 32-bit and 64-bit environments with four versions of GDAL (1.6.3, 1.7.3, 1.8.0 and 1.9.0 r22409), the script testgdalraster runs correctly.

As for rt_spatial_relationship, I have no idea as to that function.

comment:22 Changed 9 years ago by strk

Resolution: fixed
Status: assignedclosed

Works for me.

comment:23 Changed 9 years ago by pracine

Run tests: 53 Failed: 0

Great work Bborie! Thanks.

comment:24 Changed 9 years ago by pracine

About rt_spatial_relationship: robe could you post the diff file for this test? Might be an approximation problem (VC++)...

comment:25 Changed 9 years ago by jorgearevalo

It works for me too. Ubuntu 9.10 32bits box.

That's amazing Bborie! Great functionality.

comment:26 Changed 9 years ago by robe

works for me on my 8.4 windows install (all tests pass), but still getting a ton of errors on my 9.0 and 9.1. I think it might be a dumb configuration thing though because all the raster casts just aren't being installed, though I don't see any errors.

comment:27 Changed 9 years ago by Bborie Park

I don't know if the problems I had are the same but I was experiencing a similar scenario with all the raster regression errors. In looking at the outputted regress_log, I found that the loading of rtpostgis.sql failed on the last ST_MapAlgebra function (~ line 1645). Everything worked fine once I commented out that line.

comment:28 Changed 9 years ago by strk

please don't hijack this ticket, open a new one for unrelated bugs.

Note: See TracTickets for help on using tickets.