[raster] Debbie failing on GDAL_Trunk PostGIS regress

[robe]

Debbie seems to be failing now on some regress tests after GDAL trunk update

http://debbie.postgis.net:8080/job/GDAL_Trunk/1115/
GDAL SVN Revision 26554
    added DST_METHOD and support for GCP and TPS dest
    added DST_METHOD testing for GCP_POLYNOMIAL and GCP_TPS

which changed /trunk/gdal/alg/gdaltransformer.cpp

The tests failing:

Fulll ong here with errors ​http://debbie.postgis.net:8080/job/GDAL_PostGIS_Regress/1051/console

abbreviated and a crasher at rt_setvalues_geomval_expected +psql: FATAL: the database system is in recovery mode

PostgreSQL 9.2.4 on x86_64-unknown-linux-gnu, compiled by gcc (Debian 4.4.5-8) 4.4.5, 64-bit
  Postgis 2.2.0dev - r12054 - 2013-10-26 11:22:42
  GEOS: 3.5.0dev-CAPI-1.9.0 r3883
  PROJ: Rel. 4.7.1, 23 September 2009
  GDAL: GDAL 1.11dev, released 2013/04/13

Running tests

 check_gdal .. ok 
 load_outdb ... ok 
 check_raster_columns .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_3_diff)
 check_raster_overviews .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_4_diff)
 rt_io .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_5_diff)
 rt_bytea .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_6_diff)
 box3d .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_7_diff)
 rt_addband .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_8_diff)
 rt_band .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_9_diff)
 rt_tile .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_10_diff)
 rt_dimensions .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_11_diff)
 rt_scale .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_12_diff)
 rt_pixelsize .. ok 
 rt_upperleft .. ok 
 rt_rotation .. ok 
 rt_georeference .. ok 
 rt_set_properties .. ok 
 rt_isempty .. ok 
 rt_hasnoband .. ok 
 rt_metadata .. ok 
 rt_rastertoworldcoord .. ok 
 rt_worldtorastercoord .. ok 
 rt_convexhull .. ok 
 rt_band_properties .. ok 
 rt_set_band_properties .. ok 
 rt_pixelaspolygons .. ok 
 rt_pixelaspoints .. ok 
 rt_pixelascentroids .. ok 
 rt_setvalues_array .. ok 
 rt_summarystats .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_30_diff)
 rt_count .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_31_diff)
 rt_histogram .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_32_diff)
 rt_quantile .. ok 
 rt_valuecount .. ok 
 rt_valuepercent .. ok 
 rt_bandmetadata .. ok 
 rt_pixelvalue .. ok 
 rt_neighborhood .. ok 
 rt_nearestvalue .. ok 
 rt_pixelofvalue .. ok 
 rt_polygon .. ok 
 rt_utility .. ok 
 rt_fromgdalraster .. ok 
 rt_asgdalraster .. ok 
 rt_astiff .. ok 
 rt_asjpeg .. ok 
 rt_aspng .. ok 
 rt_reclass .. ok 
 rt_gdalwarp .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_49_diff)
 rt_asraster .. ok 
 rt_dumpvalues .. ok 
 rt_mapalgebraexpr .. ok 
 rt_mapalgebrafct .. ok 
 rt_mapalgebraexpr_2raster .. ok 
 rt_mapalgebrafct_2raster .. ok 
 rt_mapalgebrafctngb .. ok 
 rt_mapalgebrafctngb_userfunc .. ok 
 rt_intersection .. ok 
 rt_clip .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_59_diff)
 rt_mapalgebra .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_60_diff)
 rt_mapalgebra_expr .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_61_diff)
 rt_union .. ok 
 rt_invdistweight4ma .. ok 
 rt_4ma .. ok 
 rt_setvalues_geomval .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_65_diff)
 rt_elevation_functions .. failed (diff expected obtained: /var/lib/jenkins/workspace/postgis/gdal_regress/tmp/2_2_pg9.2w64/test_66_diff)
 rt_colormap .. ok 
 rt_gist_relationships .. ok 
 rt_intersects .. ok 
 rt_samealignment .. ok 
 rt_geos_relationships .. ok 
 rt_iscoveragetile .. ok 
 bug_test_car5 .. ok 
 tickets .. ok 
 loader/Basic .... ok 
 loader/BasicCopy .... ok 
 loader/BasicFilename .... ok 
 loader/BasicOutDB .... ok 
 loader/Tiled10x10 .... ok 
 loader/Tiled10x10Copy .... ok 
 loader/Tiled8x8 .... ok 
 clean .. ok 
 uninstall .

[robe]

Dusty or Pierre,

When you get a chance, can you try build PostGIS with latest GDAL. I'll try to do that locally but not setup to at moment, to rule this out as just a bot issue.

[robe]

Here is the GDAL commit which I think triggered the issue

​http://trac.osgeo.org/gdal/changeset/26554

[dustymugs]

Confirmed. Thanks for doing the leg-work… I'll see how fast I can fix this.

[dustymugs]

There has been lots of work on GDALCreateGenImgProjTransformer2() in the last day or two. I'll give it a few days in case the new GDAL behavior is to be expected.

[robe]

Bborie,

For the record I've been chatting with EvenR on IRC. He told me to keep him abreast of the situation. It doesn't sound like they intentionally meant to break anything in PostGIS land. Maybe we can ask Frank about it.

Can we at least confirm this is a GDAL change that broke our code? Then maybe we can put in a separate ticket on GDAL to find out the intention.

[dustymugs]

Yes, the GDAL change resulted in different behavior. I'm going to see what changed exactly.

[rouault]

I have reviewed the differences in GDALCreateGenImgProjTransformer2() and the use made of it in PostGIS Raster code (2 occurrences in in rt_warp.c). I believe that there should be no impact on the one at line 358, but there might be an influence on the one at line 840 if the geotransform on the dest ds is set to [ 0, 1, 0, 0, 0, ± 1 ]. If that's the case, I believe that you get a "Unable to compute a transformation between pixel/line and georeferenced coordinates for …" error message from GDAL, and a NULL result, that causes rt_raster_gdal_warp() to return in error.

Is it the case ?

[dustymugs]

Yes, that geotransform can be present in PostGIS. I was looking at the problem through GDALRasterizeGeometries() in rt_raster.c. The test geometry and raster I am using makes use of that geotransform [0, 1, 0, 0, 0, -1].

[warmerdam]

I'll remove the check for the null transform. I only added it for similarity to the rules on the "source" side of the transformation.

[warmerdam]

OK, that change is now in gdal (r26578).

[dustymugs]

Thanks warmerdam.

I plan on testing what happens if non-spatially referenced input rasters and geometries are made to be spatially referenced (faked). If the tests work, you should be able to undo r26578.

[dustymugs]

Well, I couldn't get non-spatial geometries and rasters with the default geotransform [0, 1, 0, 0, 0, -1] to work with faked spatial references.

Thanks warmerdam!

[dustymugs]

I'm closing this as "wontfix" since there isn't an option "cantfix". The real resolution is for postgis to have a full-fledged rasterizer as per #1168.

[rouault]

Replying to dustymugs:

I'm closing this as "wontfix" since there isn't an option "cantfix". The real resolution is for postgis to have a full-fledged rasterizer as per #1168.

Hum, I don't understand the link between rasterization and the issue raised in that ticket… Is there anything left on GDAL side related to gdaltransformer that would be needed for PostGIS raster ?

[dustymugs]

No, there is nothing left of the GDAL side related to gdaltransformer that needs to be done. I am concerned of GDAL being blocked (warmerdam having to undo some of his work) because of PostGIS, which is probably not the best thing for either project in the long term.

[rouault]

Well, in the instance, it was a test that wasn't strictly necessary and doesn't remove functionnality in GDAL. Feedback (and contributions) from projects using GDAL are mutually valuable.

[robe]

Just to maintain accuracy of this ticket to repflect the issue was INDEED resolved and did not require a change in PostGIS, I've flipped this.

[robe]

Was only an issue in latest GDAL trunk which Frank has resolved. Thank you Frank and Even for the quick response and resolution.

[strk]

The problem of identity matrix was found also on the usage with no destination, see #2914