gdal_retile - test failing, pyramids maybe wrong?

[warmerdam]

I am getting failures of the checksum for test_gdal_retile.py - test 2 - on 32bit linux (Debug build).

Examining the results the expected result (which I get on win32) is as follows (band 1 translated to AAIGRID format for convenience):

ncols        5
nrows        5
xllcorner    500000.000000000000
yllcorner    4499800.000000000000
cellsize     40.000000000000
 0 0 0 0 0
 0 255 255 0 0
 0 255 255 0 0
 0 0 0 0 0
 0 0 0 0 0

While on linux I am getting:

ncols        5
nrows        5
xllcorner    500000.000000000000
yllcorner    4499800.000000000000
cellsize     40.000000000000
 0 0 0 0 0
 0 0 0 0 0
 0 255 255 0 0
 0 255 255 0 0
 0 0 0 0 0

On further examination it appears that the full resolution data has 255 for *all* pixels in band 1. It is not clear to me why the reduced resolution pyramid levels (like tmp/outretile2/2/rgba_1_1.tif) have any zeros in band 1. It seems they also ought to be all 255.

I'm not too keen to dig deeply into this today, but I am now generally dubious about the overview generation in gdal_retile.py.

[Even Rouault]

It is expected that some pixels are 0, because the source raster has a non-fully opaque alpha band. The issue was more that with some compilers, padfY[iDstX]=0.9999999999750173, which got truncated into 0. Adding a small epsilon 1e-10 helps working around that.

r22887 /trunk/gdal/alg/gdalwarpkernel.cpp: Warp kernel: workaround floating-point precision loss that make test_gdal_retile_2 fail with some MSVC versions or gcc -ffast-math -mfpmath=both -O2 (#4174)