Ticket #1470 (closed defect: fixed)

Opened 2 years ago

Last modified 2 years ago

gdalwarp with pixel interleave, resample, and compress makes huge files

Reported by: matt.wilkie@gov.yk.ca Assigned to: warmerdam
Priority: normal Milestone: 1.4.1
Component: GDAL_Raster Version: 1.4.0
Severity: major Keywords:
Cc:

Description (Last modified by warmerdam)

Using gdalwarp with -co interleave=pixel produces larger files than gdal_translate does with the same option. Combining pixel interleaving with resampling and compression makes even larger files than when no compression at all is used. When the 3 are used together I get file sizes up to 200% larger than no compression at all. 

Original:
	328,390 source.tif

Various combinations of gdalwarp -co [options]:
	# lzw	-co compress=lzw
	# pix	-co pixel=interleave
	# rb	-rb

	165,284 gw_lzw.tif
	263,640 gw_lzw-rb.tif
	304,716 gw_pix-lzw.tif
	340,604 gw_pix-rb.tif
	340,604 gw_pix.tif
	524,786 gw_lzw-pix-rb.tif ***

The above pushed through gdal_translate -co compress=lzw -co interleave=pixel:
	123,292 tr_gw_lzw.tif
	123,292 tr_gw_pix-lzw.tif
	123,292 tr_gw_pix.tif
	123,408 tr_source.tif

...and with gdal_translate -co compress=none
	328,274 tr_nocomp_gw_lzw-rb.tif
	...etc.

...and with gdal_translate -co compress=none -co interleave=pixel:
	328,274 tr_nocomp-pix_gw_lzw-rb.tif
	...etc.

Command lines to produce these files:

gdalwarp -co interleave=pixel source.tif gw_pix.tif
gdalwarp -co compress=lzw source.tif gw_lzw.tif
gdalwarp -co compress=lzw -co interleave=pixel -rb source.tif gw_lzw-pix-rb.tif

Attachments

source.tif (320.7 kB) - added by matt.wilkie@gov.yk.ca on 02/01/07 16:49:48.
source image
gw_lzw-pix-rb.tif (0.5 MB) - added by matt.wilkie@gov.yk.ca on 02/01/07 16:52:54.
rb, pix interleave, and lzw combined, 1.6 times larger than original

Change History

02/01/07 16:49:48 changed by matt.wilkie@gov.yk.ca

  • attachment source.tif added.

source image

02/01/07 16:52:54 changed by matt.wilkie@gov.yk.ca

  • attachment gw_lzw-pix-rb.tif added.

rb, pix interleave, and lzw combined, 1.6 times larger than original

02/09/07 17:22:30 changed by matt.wilkie@gov.yk.ca 

Workaround for when this bug causes failures when output is over 4gb:

   gdalwarp -of HFA [list of input] xxx.img
   gdal_translate -co compress=lzw xxx.img xxx_lzw.tif
 
If you need to reassign nodata lost above:

   gdalwarp --co compress=lzw -srcnodata 255 -dstnodata 255 xxx_lzw.tif xxx-lzw-nodata.tif

And now properly compress and interleave final output:

   gdal_translate -co compress=lzw -co interleave=pixel xxx_lzw-nodata.tif final.tif

03/26/07 21:05:46 changed by warmerdam

  • status changed from new to assigned.
  • severity changed from normal to major.
  • description changed.
  • milestone set to 1.4.1.

03/26/07 22:13:53 changed by warmerdam

  • milestone changed from 1.4.1 to 1.5.0.

I don't see any problems with compress=lzw on it's own, but with interleave=pixel the output file is indeed much too large. It appears this is because on closing and flushing out the file the buffers are passed to GTiffRasterBand::IWriteBlock() one band at a time. So when band 1 is written, bands 2 and 3 will be set to zero. When band 2 is written the compressed result will be bigger, so the block is moved to the end of the file, and the old space is lost. When band 3 is written the same happens again. The result is an output file roughly twice the size it should be.

The solution would appear to be smarts in geotiff.cpp's GTiffRasterBand::IWriteBlock() to collect results from all bands in one pass. I'll look into this, but I am dubious this is something I could push into the GDAL 1.4. branch, so I'll likely just pursue it in trunk.

03/26/07 23:23:05 changed by warmerdam

  • priority changed from highest to normal.
  • component changed from default to GDAL_Raster.
  • status changed from assigned to closed.
  • resolution set to fixed.
  • milestone changed from 1.5.0 to 1.4.1.

I have applied the fix, completely contained in IWriteBlock(). On reflection the fix is contained enough that I have retrofit it to 1.4 branch as well. r11081 and r11082.

03/27/07 18:25:14 changed by maphew

thank you frank!

03/27/07 20:05:20 changed by warmerdam

Thanks for the good bug report. I just wish i'd addressed it sooner. I've been negligent addressing bugs for quite a long time. Hopefully with Mateusz' help some of the backlog can be worked through. Thanks for reviewing your own bug submissions.