NODATA missing in blank area when warping vrt file with expanded RGB into vrt

[klokan]

I am warping a VRT file which is expanded into RGB from a tiff with color palette. The result is different when warping into tiff and vrt.

The newly added blank areas created because of the change of the map shape during reprojection are always black (0,0,0) in vrt. They should be filled with correct NODATA value. Even when the nodata value is explicitly set when calling gdalwarp command the result is not correct:

Example:

gdal_translate -of vrt -a_nodata 255 -expand rgb palette.tif rgb.vrt
gdalwarp -of vrt -t_srs epsg:900913 -srcnodata 255 -dstnodata 255 rgb.vrt rgbwarped.vrt

If I use in the second step gtiff driver instead of vrt, like:

gdalwarp -of gtiff -t_srs epsg:900913 -srcnodata 255 -dstnodata 255 rgb.vrt rgbwarped.tif

then the result is correct - new blank areas are filled with NODATA value.

Difference is visible in the top left corner of the file. OpenEV shows:

rgbwarped.vrt: 0r 0g 0b
rgbwarped.tif: 255r 255g 255g [NODATA]

It seems like a bug in the warping core related to vrt with expanded rgb.

[warmerdam]

Some notes from an IRC discussion:

<EvenR> the 'memset( pabyDstBuffer, 0, nDstBufferSize )' in VRTWarpedDataset::ProcessBlock() looks like a good place to try things

<FrankW> I see, because we use WarpRegionToBuffer() we have skipped all the stuff about initializating the destination file normally done at a higher level.

<FrankW> I believe we need to duplicate logic somewhat like around line 1110 in gdalwarpoperation.cpp in the ProcessBlock() in vrtwarped.cpp

[warmerdam]

Petr,

I have applied two fixes in trunk.

The first (r15915) ensures that the INIT_DEST warp option is honoured in the vrt warped driver.

The second (r15914) ensures that the warped vrt reading properly reads back the nodata (it was overwriting the real component with the imaginary value).

With these applied trunk should handle your situation properly. If you can confirm this with trunk then I'd be willing to backport the changes to the 1.6 branch. Don't forget to mention to your client the fix is free due to the immense positive karma you have on the project!

[klokan]

Hi Frank,

thanks a lot for the time you have spent on this bug patching.

I did the compilation of latest trunk, and some testing as well. For the first look it runs great, but then I tried several real files and I have found a small bug which appeared. To reproduce it try this:

gdal_translate -of vrt -a_nodata 255 -expand rgb palette.tif rgb.vrt
gdalwarp -of vrt -t_srs epsg:900913 rgb.vrt rgbwarped.vrt
gdal_translate rgbwarped.vrt rgbwarped.vrt.tif

The result has wrong colors and sometimes even some artefacts appears: ​http://klokan.mzk.cz/~klokan/newzealand/rgbwarped.vrt.tif ​http://klokan.mzk.cz/~klokan/newzealand/wrong/googlemaps.html

I had a look at your changesets and the second is just a comment in the source code and the first is changing only one function, is that correct?

Thank you for the compliment. BTW I passed the note to the client already ;-).

[warmerdam]

My mistake in recording the revisions. It was actually r15916 and r15914.

I'll try and reproduce the problem you are still having.

[Even Rouault]

For rgbwarped.vrt.tif, the wrong colors can probably be explained because you didn't add -dstnodata 255 to the gdalwarp command line.

I've had a look at the color table of your 8bit image, and it makes me think that you'll run into issues when warping an RGB image with nodata set on the 3 bands (I mentionned it quickly on IRC yesterday). The warping algorithm works one band at a time, completely ignoring the others. You probably wanted to make index 255 of your 8bit image (which maps to 255,255,255) the only nodata value. However when expanded to 24 bit, there are many colors now whose one of the components is 255 on one band, but not the other ones. To be clear, you'd like to consider white as nodata, but not the red component of the red color (255,0,0), the green component of the green color (0,255,0), etc etc

I'm pretty sure there's a ticket about that, where someone shows that it leads to strange color effects. Specifying -dstnodata 255 probably reduces most of the wrong colors, but not the fundamental problem.

As for the googlemaps artefacts, it's difficult to say what is going wrong. You should probably try to isolate things a bit more. Maybe try without VRTs as they may carry their own set of bugs...

You could try the following workaround if you want to output a 24 bit PNG :

# This works since it will put 0 in the areas that have are not
# in the source image, and 0 is 255,255,255. But don't use
# anything other than the default nearest resampling.
gdalwarp -of vrt -t_srs epsg:900913 palette.tif palettewarped.vrt

# This works since the PNG driver understands properly the NODATA_VALUES metadata
gdal_translate -of PNG -expand rgb palettewarped.vrt rgbwarped.png -mo NODATA_VALUES="255 255 255"

[warmerdam]

I never got around to trying to reproduce the outstanding issue(s).

I will note that there is a warp option called UNIFIED_SRC_NODATA which causes the warper to treat the nodata mask per-pixel across all bands if set to YES. I can't find a corresponding thing on the destination side. Possibly it would help?

[klokan]

The second issue I was describing seams to be related to the UNIFIED_SRC_NODATA as Frank mentioned.

When gdalwarp is called without -srcnodata, then this option is off and then the warping core can produce strange colors as I demonstrated.

If I add this line into the rgbwarped.vrt generated in the second step everything is OK:

<Option name="UNIFIED_SRC_NODATA">YES</Option>

C++ applications can set this option directly. Applications using AutoCreateWarpedVRT from SWIG binding must add this line in to the text of the produced VRT manually.

On the command line with gdalwarp it is recommended to specify -srcnodata parameter.

I reallized gdalwarp has some kind of autodetection of the nodata in the input file. It prints: "Using internal nodata values (eg. 255) for image rgb.vrt.". Maybe it would make sense to turn on the variable UNIFIED_SRC_NODATA in case nodata presence is detected. But I am not sure what are exactly the consequences of such step...

From my point of view this issue is closed. I hope the modification will soon appear in the stable release of GDAL.

Big thanks to both FrankW and EvenR for working on this. This feature is going to be used very often in MapTiler application...

[warmerdam]

I was wrong again above about which changesets were applied. They were r15916 and r15915.

I have also applied the same changes in 1.6 branch (r15926, r15927) and 1.5 branch (r15928, r15929).

Closing.