Opened 12 years ago

Closed 10 years ago

#1688 closed defect (fixed)

gdalwarp creates GeoTIFFs bigger than gdal_translate

Reported by: msieczka Owned by: warmerdam
Priority: normal Milestone: 1.7.0
Component: Algorithms Version: svn-trunk
Severity: major Keywords: gdalwarp
Cc:

Description

Analogous options settings in gdal_translate and gdalwarp result in GeoTIFF's of significantly different sizes. Example:

$ gdal_translate vegcover3.tif vegcover3_gt.tif

vegcover3_gt.tif size is 28628 bytes. Identical to input. OK.

$ gdalwarp vegcover3.tif vegcover3_gw.tif

vegcover3_gw.tif size is 36336 bytes. That's 27% bigger than the input!

The bigger the output GeoTIFF, the smaller the difference in size. At circa 230 kB, the difference becomes almost neglectable.

However, if one creates lots of small GeoTIFFs, this waists his disk and memory space considerably - eg. 500 10 kB GeoTIFFs from gdal_translate occupy 5 MB, while from gdalwarp it's 10 MB.

The attached graph (gt_vs_gw.pdf; X axis should read "gdal_translate output size [kB]") shows how the size ratio changes with output raster size. Each raster was created and "measured" by script (gt_vs_gw.txt) and those measurements were used to create the graph. The very input raster is attached too (one.tif).

Frank Warmerdam explained the reason:

"On careful review, the difference in the file in question is because gdal_translate uses the TIFFWriteScanline() interface to write the output file from within GTiffDataset::CreateCopy?(), and this only writes as much of the final 'strip' of the file as is required to complete image area. But gdalwarp goes through the blockio interface which writes the complete final strip, even the portion that falls off the end of the file."

There was a discussion about this issue on GDAL dev list in May 2007 [1] continued in June [2].

[1]http://lists.maptools.org/pipermail/gdal-dev/2007-May/012997.html [2]http://lists.maptools.org/pipermail/gdal-dev/2007-June/013255.html

Attachments (4)

gt_vs_gw.pdf (28.6 KB) - added by msieczka 12 years ago.
gt_vs_gw.txt (580 bytes) - added by msieczka 12 years ago.
one.tif (1.9 KB) - added by msieczka 12 years ago.
warpchunkorder.patch (1.2 KB) - added by nowakpl 12 years ago.

Download all attachments as: .zip

Change History (13)

Changed 12 years ago by msieczka

Attachment: gt_vs_gw.pdf added

Changed 12 years ago by msieczka

Attachment: gt_vs_gw.txt added

Changed 12 years ago by msieczka

Attachment: one.tif added

comment:1 Changed 12 years ago by warmerdam

This may related to #1723.

comment:2 Changed 12 years ago by Even Rouault

This is one is interesting... Particularly the 'gdal_translate X.tiff X.tiff'. I think it's probably not a good idea to try that. I've tried it on a few TIFF files and it fails miserabily with :

ERROR 1: TIFFFillStrip:test.tiff: Read error at scanline 4294967295; got 0 bytes, expected 7986
ERROR 1: TIFFReadEncodedStrip() failed.

ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0

It fails with a 121x121 16-bit 1 canal TIFF, a 4096x2048 RGB 8-bit TIFF, etc... The 1x1 one.tif works great though... But as it is really small, it's probably read in memory just once and never read after the first write.

My conclusion would be to detect and prevents the user from ever trying "gdal_translate X X". I'm realizing it probably deserves a specific ticket for that.

comment:3 Changed 12 years ago by Even Rouault

Mea maxima culpa, I read too quicky the original post. msieczka was doing "gdal_translate vegcover3.tif vegcover3_gt.tif" and not "gdal_translate vegcover3.tif vegcover3.tif. Sorry for that.

comment:4 Changed 12 years ago by msieczka

Severity: normalmajor
Version: 1.4.1svn-trunk

I wrote:

"The bigger the output GeoTIFF, the smaller the difference in size. At circa 230 kB, the difference becomes almost neglectable."

That doesn't hold true. You can get huge gdalwarp:gdal_translate ratios with input data of any size.

Max I remember I had was 13:1.

For example, with the 290 MB GeoTIFF available at [1], 'gdalwarp -co "COMPRESS=LZW"' vs 'gdal_translate -co "COMPRESS=LZW"' ratio is 5:1. For the same data the 'gdalwarp -co "COMPRESS=LZW"' vs 'gdalwarp -co "COMPRESS=NONE"' is about 3:1.

Compressed raster must not be bigger than uncompressed. gdalwarp and gdal_translate should create rasters of circa the same size.

[1]http://www.biol.uni.wroc.pl/sieczka/udostepnione/M-33-21-A-b-4.tif

comment:5 Changed 12 years ago by maphew

I suggest that until such a time as this complex bug is fixed* the -co compress option be dropped from gdalwarp, or that a warning be issued when used.

comment:6 Changed 12 years ago by maphew

Frank Warmerdam:

It is not possible to drop driver specific creation options from gdalwarp. Gdalwarp passes driver options to the drivers and has no gdalwarp specific knowledge about them.

c.f.

I've added a note to this effect to the User Docs section of the wiki: http://trac.osgeo.org/gdal/wiki/UserDocs/GdalWarp I don't know how often this page is found by people, it may be a good idea to add a similar note to the main docs.

Changed 12 years ago by nowakpl

Attachment: warpchunkorder.patch added

comment:7 Changed 12 years ago by nowakpl

I have added a patch that orders the chunks produced by warping to be more cache friendly. Below are the results, gdal-r14052_x86-64_ubuntu-gutsy are without it and gdal-r14052+warpchunkorder_x86-64_ubuntu-gutsy with it.

time /opt/gdal-r14052_x86-64_ubuntu-gutsy/bin/gdal_translate -co COMPRESS=DEFLATE AS15-M-0083.tif gdaltranslate.tiff Input file size is 26143, 24889 0...10...20...30...40...50...60...70...80...90...100 - done.

real 2m17.111s user 2m7.980s sys 0m2.610s

time /opt/gdal-r14052_x86-64_ubuntu-gutsy/bin/gdalwarp AS15-M-0083.tif gdalwarp.tiff Creating output file that is 26143P x 24889L. Processing input file AS15-M-0083.tif. 0...10...20...30...40...50...60...70...80...90...100 - done.

real 5m31.607s user 0m29.650s sys 0m25.590s

time /opt/gdal-r14052+warpchunkorder_x86-64_ubuntu-gutsy/bin/gdalwarp AS15-M-0083.tif gdalwarp+warpchunkorder.tiff Creating output file that is 26143P x 24889L. Processing input file AS15-M-0083.tif. 0...10...20...30...40...50...60...70...80...90...100 - done.

real 1m44.590s user 0m28.000s sys 0m25.530s

time /opt/gdal-r14052+warpchunkorder_x86-64_ubuntu-gutsy/bin/gdalwarp -co COMPRESS=DEFLATE -wm 2 --config GDAL_CACHEMAX 128 AS15-M-0083.tif gdalwarpdeflate+warpchunkorder.tiff Creating output file that is 26143P x 24889L. Processing input file AS15-M-0083.tif. 0...10...20...30...40...50...60...70...80...90...100 - done.

real 2m39.383s user 2m21.150s sys 0m3.060s

-rw-r--r-- 1 root root 1301545604 2007-07-02 06:32 AS15-M-0083.tif -rw-r--r-- 1 root root 1064405380 2008-03-21 14:09 gdaltranslate.tiff -rw-r--r-- 1 root root 1064604506 2008-03-21 14:34 gdalwarpdeflate+warpchunkorder.tiff -rw-r--r-- 1 root root 1302372290 2008-03-21 14:15 gdalwarp.tiff -rw-r--r-- 1 root root 1304672874 2008-03-21 14:30 gdalwarp+warpchunkorder.tiff

comment:8 Changed 12 years ago by nowakpl

-rw-r--r-- 1 root root 1301545604 2007-07-02 06:32 AS15-M-0083.tif
-rw-r--r-- 1 root root 1064405380 2008-03-21 14:09 gdaltranslate.tiff
-rw-r--r-- 1 root root 1064604506 2008-03-21 14:34 gdalwarpdeflate+warpchunkorder.tiff
-rw-r--r-- 1 root root 1302372290 2008-03-21 14:15 gdalwarp.tiff
-rw-r--r-- 1 root root 1304672874 2008-03-21 14:30 gdalwarp+warpchunkorder.tiff

comment:9 Changed 10 years ago by Even Rouault

Component: GDAL_RasterAlgorithms
Keywords: gdalwarp added
Milestone: 1.7.0
Resolution: fixed
Status: newclosed

r18277 /trunk/gdal/alg/gdalwarpoperation.cpp: Warp algorithm : reduce destination file size, especially when it has compression, by sorting destination chunks from top to bottom and left to right (contributed by Adam Nowacki); also aligns chunk dimension to block size (#1688)

Note: See TracTickets for help on using tickets.