Opened 13 years ago

Closed 13 years ago

Last modified 13 years ago

#1470 closed defect (fixed)

gdalwarp with pixel interleave, resample, and compress makes huge files

Reported by: matt.wilkie@… Owned by: warmerdam
Priority: normal Milestone: 1.4.1
Component: GDAL_Raster Version: 1.4.0
Severity: major Keywords:
Cc:

Description (last modified by warmerdam)

Using gdalwarp with -co interleave=pixel produces larger files than gdal_translate does with the same option. Combining pixel interleaving with resampling and compression makes even larger files than when no compression at all is used. When the 3 are used together I get file sizes up to 200% larger than no compression at all.

Original:
	328,390 source.tif

Various combinations of gdalwarp -co [options]:
	# lzw	-co compress=lzw
	# pix	-co pixel=interleave
	# rb	-rb

	165,284 gw_lzw.tif
	263,640 gw_lzw-rb.tif
	304,716 gw_pix-lzw.tif
	340,604 gw_pix-rb.tif
	340,604 gw_pix.tif
	524,786 gw_lzw-pix-rb.tif ***

The above pushed through gdal_translate -co compress=lzw -co interleave=pixel:
	123,292 tr_gw_lzw.tif
	123,292 tr_gw_pix-lzw.tif
	123,292 tr_gw_pix.tif
	123,408 tr_source.tif

...and with gdal_translate -co compress=none
	328,274 tr_nocomp_gw_lzw-rb.tif
	...etc.

...and with gdal_translate -co compress=none -co interleave=pixel:
	328,274 tr_nocomp-pix_gw_lzw-rb.tif
	...etc.

Command lines to produce these files:

gdalwarp -co interleave=pixel source.tif gw_pix.tif
gdalwarp -co compress=lzw source.tif gw_lzw.tif
gdalwarp -co compress=lzw -co interleave=pixel -rb source.tif gw_lzw-pix-rb.tif

Attachments (2)

source.tif (320.7 KB) - added by matt.wilkie@… 13 years ago.
source image
gw_lzw-pix-rb.tif (512.5 KB) - added by matt.wilkie@… 13 years ago.
rb, pix interleave, and lzw combined, 1.6 times larger than original

Download all attachments as: .zip

Change History (8)

Changed 13 years ago by matt.wilkie@…

Attachment: source.tif added

source image

Changed 13 years ago by matt.wilkie@…

Attachment: gw_lzw-pix-rb.tif added

rb, pix interleave, and lzw combined, 1.6 times larger than original

comment:1 Changed 13 years ago by matt.wilkie@…

Workaround for when this bug causes failures when output is over 4gb:

   gdalwarp -of HFA [list of input] xxx.img
   gdal_translate -co compress=lzw xxx.img xxx_lzw.tif
 
If you need to reassign nodata lost above:

   gdalwarp --co compress=lzw -srcnodata 255 -dstnodata 255 xxx_lzw.tif xxx-lzw-nodata.tif

And now properly compress and interleave final output:

   gdal_translate -co compress=lzw -co interleave=pixel xxx_lzw-nodata.tif final.tif

comment:5 Changed 13 years ago by warmerdam

Description: modified (diff)
Milestone: 1.4.1
Severity: normalmajor
Status: newassigned

comment:6 Changed 13 years ago by warmerdam

Milestone: 1.4.11.5.0

I don't see any problems with compress=lzw on it's own, but with interleave=pixel the output file is indeed much too large. It appears this is because on closing and flushing out the file the buffers are passed to GTiffRasterBand::IWriteBlock() one band at a time. So when band 1 is written, bands 2 and 3 will be set to zero. When band 2 is written the compressed result will be bigger, so the block is moved to the end of the file, and the old space is lost. When band 3 is written the same happens again. The result is an output file roughly twice the size it should be.

The solution would appear to be smarts in geotiff.cpp's GTiffRasterBand::IWriteBlock() to collect results from all bands in one pass. I'll look into this, but I am dubious this is something I could push into the GDAL 1.4. branch, so I'll likely just pursue it in trunk.

comment:7 Changed 13 years ago by warmerdam

Component: defaultGDAL_Raster
Milestone: 1.5.01.4.1
Priority: highestnormal
Resolution: fixed
Status: assignedclosed

I have applied the fix, completely contained in IWriteBlock(). On reflection the fix is contained enough that I have retrofit it to 1.4 branch as well. r11081 and r11082.

comment:8 Changed 13 years ago by maphew

thank you frank!

comment:9 Changed 13 years ago by warmerdam

Thanks for the good bug report. I just wish i'd addressed it sooner. I've been negligent addressing bugs for quite a long time. Hopefully with Mateusz' help some of the backlog can be worked through. Thanks for reviewing your own bug submissions.

Note: See TracTickets for help on using tickets.