gdalwarp with pixel interleave, resample, and compress makes huge files

[matt.wilkie@…]

Using gdalwarp with -co interleave=pixel produces larger files than gdal_translate does with the same option. Combining pixel interleaving with resampling and compression makes even larger files than when no compression at all is used. When the 3 are used together I get file sizes up to 200% larger than no compression at all.

Original:
	328,390 source.tif

Various combinations of gdalwarp -co [options]:
	# lzw	-co compress=lzw
	# pix	-co pixel=interleave
	# rb	-rb

	165,284 gw_lzw.tif
	263,640 gw_lzw-rb.tif
	304,716 gw_pix-lzw.tif
	340,604 gw_pix-rb.tif
	340,604 gw_pix.tif
	524,786 gw_lzw-pix-rb.tif ***

The above pushed through gdal_translate -co compress=lzw -co interleave=pixel:
	123,292 tr_gw_lzw.tif
	123,292 tr_gw_pix-lzw.tif
	123,292 tr_gw_pix.tif
	123,408 tr_source.tif

...and with gdal_translate -co compress=none
	328,274 tr_nocomp_gw_lzw-rb.tif
	...etc.

...and with gdal_translate -co compress=none -co interleave=pixel:
	328,274 tr_nocomp-pix_gw_lzw-rb.tif
	...etc.

Command lines to produce these files:

gdalwarp -co interleave=pixel source.tif gw_pix.tif
gdalwarp -co compress=lzw source.tif gw_lzw.tif
gdalwarp -co compress=lzw -co interleave=pixel -rb source.tif gw_lzw-pix-rb.tif

[matt.wilkie@…]

source image

[matt.wilkie@…]

rb, pix interleave, and lzw combined, 1.6 times larger than original

[matt.wilkie@…]

Workaround for when this bug causes failures when output is over 4gb:

   gdalwarp -of HFA [list of input] xxx.img
   gdal_translate -co compress=lzw xxx.img xxx_lzw.tif
 
If you need to reassign nodata lost above:

   gdalwarp --co compress=lzw -srcnodata 255 -dstnodata 255 xxx_lzw.tif xxx-lzw-nodata.tif

And now properly compress and interleave final output:

   gdal_translate -co compress=lzw -co interleave=pixel xxx_lzw-nodata.tif final.tif

[warmerdam]

I don't see any problems with compress=lzw on it's own, but with interleave=pixel the output file is indeed much too large. It appears this is because on closing and flushing out the file the buffers are passed to GTiffRasterBand::IWriteBlock() one band at a time. So when band 1 is written, bands 2 and 3 will be set to zero. When band 2 is written the compressed result will be bigger, so the block is moved to the end of the file, and the old space is lost. When band 3 is written the same happens again. The result is an output file roughly twice the size it should be.

The solution would appear to be smarts in geotiff.cpp's GTiffRasterBand::IWriteBlock() to collect results from all bands in one pass. I'll look into this, but I am dubious this is something I could push into the GDAL 1.4. branch, so I'll likely just pursue it in trunk.

[warmerdam]

I have applied the fix, completely contained in IWriteBlock(). On reflection the fix is contained enough that I have retrofit it to 1.4 branch as well. r11081 and r11082.

[maphew]

thank you frank!

[warmerdam]

Thanks for the good bug report. I just wish i'd addressed it sooner. I've been negligent addressing bugs for quite a long time. Hopefully with Mateusz' help some of the backlog can be worked through. Thanks for reviewing your own bug submissions.