Opened 13 years ago

Closed 12 years ago

#1507 closed defect (fixed)

GDALRasterIO crashes for width values higher than 2048 with LZW enabled

Reported by: luizcpg@… Owned by: Mateusz Łoskot
Priority: normal Milestone: 1.4.3
Component: GDAL_Raster Version: 1.4.1
Severity: normal Keywords:
Cc:

Description (last modified by Mateusz Łoskot)

Env : x86 32 bits - Linux Gdal 1.3.1

with width > 2048 and LZW disabled, work ,but the output file size is nosense.

GDAL_CACHEMAX = 10 or 100 has no diference.

Attachments (1)

gdal_example.c (7.8 KB) - added by luizcpg@… 13 years ago.
code demonstrating the problem.

Download all attachments as: .zip

Change History (19)

Changed 13 years ago by luizcpg@…

Attachment: gdal_example.c added

code demonstrating the problem.

comment:1 Changed 13 years ago by warmerdam

Mateusz, 

Can you try and reproduce this.  We can consult on a solution, if one is needed.
I *suspect* this is something fixed in recent versions of libtiff so be very
aware of libtiff version.  Likely best to test with curren gdal+built-in-libtiff
and then gdal-1.3.1+build-in-libtiff if it doesn't occur with current. 

comment:4 Changed 13 years ago by warmerdam

Description: modified (diff)
Milestone: 1.4.1

comment:5 Changed 13 years ago by warmerdam

Priority: highestnormal

comment:6 Changed 13 years ago by Mateusz Łoskot

Description: modified (diff)
Status: newassigned

comment:7 Changed 13 years ago by Mateusz Łoskot

Description: modified (diff)

Dear Reporter,

Please

comment:8 Changed 13 years ago by Mateusz Łoskot

Ah, please forget the comment:7 above. I just wanted to ask for sample code as it's no longer available on the rafb.net, but then I noticed it's attached to this report. Sorry for messing the report.

comment:9 Changed 13 years ago by Mateusz Łoskot

Dear luizcpg,

My first question is, please could you be more specific about the meaning of:

the output file size is nosense ?

I did some tests of this issue and compiled / run your example program and here are my results:

mloskot:~/dev/gdal/bugs/1507$ ls -lh *.tif
-rw-r--r-- 1 mloskot mloskot 102M 2007-04-05 23:26 bad.tif
-rw-r--r-- 1 mloskot mloskot  81M 2007-04-05 23:26 good.tif
  1. Simple check of the good.tif
mloskot:~/dev/gdal/bugs/1507$ gdalinfo -stats good.tif 
GDAL: GDALOpen(good.tif) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Size is 2048, 2048
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-46.619302400000002,-23.480697599999999)
Pixel Size = (0.000018850000000,-0.000018850000000)
Metadata:
  AREA_OR_POINT=Area
Corner Coordinates:
Upper Left  ( -46.6193024, -23.4806976) ( 46d37'9.49"W, 23d28'50.51"S)
Lower Left  ( -46.6193024, -23.5193024) ( 46d37'9.49"W, 23d31'9.49"S)
Upper Right ( -46.5806976, -23.4806976) ( 46d34'50.51"W, 23d28'50.51"S)
Lower Right ( -46.5806976, -23.5193024) ( 46d34'50.51"W, 23d31'9.49"S)
Center      ( -46.6000000, -23.5000000) ( 46d36'0.00"W, 23d30'0.00"S)
Band 1 Block=256x256 Type=UInt32, ColorInterp=Gray
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 2 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 3 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 4 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 5 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=4294967295.000, Maximum=4294967295.000, Mean=4294967295.500, StdDev=nan
  Metadata:
    STATISTICS_MINIMUM=4294967295
    STATISTICS_MAXIMUM=4294967295
    STATISTICS_MEAN=4294967295.5
    STATISTICS_STDDEV=nan
GDAL: GDALClose(good.tif)
GDAL: GDALDeregister_GTiff() called.
  1. Check what GDAL says about the bad.tif
mloskot:~/dev/gdal/bugs/1507$ gdalinfo -stats bad.tif 
GDAL: GDALOpen(bad.tif) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Size is 2049, 2049
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-46.619311825000004,-23.480688175000001)
Pixel Size = (0.000018850000000,-0.000018850000000)
Metadata:
  AREA_OR_POINT=Area
Corner Coordinates:
Upper Left  ( -46.6193118, -23.4806882) ( 46d37'9.52"W, 23d28'50.48"S)
Lower Left  ( -46.6193118, -23.5193118) ( 46d37'9.52"W, 23d31'9.52"S)
Upper Right ( -46.5806882, -23.4806882) ( 46d34'50.48"W, 23d28'50.48"S)
Lower Right ( -46.5806882, -23.5193118) ( 46d34'50.48"W, 23d31'9.52"S)
Center      ( -46.6000000, -23.5000000) ( 46d36'0.00"W, 23d30'0.00"S)
Band 1 Block=256x256 Type=UInt32, ColorInterp=Gray
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 2 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 3 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 4 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=0.000, Maximum=0.000, Mean=0.000, StdDev=0.000
  Metadata:
    STATISTICS_MINIMUM=0
    STATISTICS_MAXIMUM=0
    STATISTICS_MEAN=0
    STATISTICS_STDDEV=0
Band 5 Block=256x256 Type=UInt32, ColorInterp=Undefined
  Minimum=4294967295.000, Maximum=4294967295.000, Mean=4294967295.500, StdDev=nan
  Metadata:
    STATISTICS_MINIMUM=4294967295
    STATISTICS_MAXIMUM=4294967295
    STATISTICS_MEAN=4294967295.5005
    STATISTICS_STDDEV=nan
GDAL: GDALClose(bad.tif)
GDAL: GDALDeregister_GTiff() called.

Summarizing, I don't see or I can not recognize the problem.

Could you confirm if my results are correct or not?

comment:10 Changed 13 years ago by Mateusz Łoskot

Frank,

According to Luiz's and my tests, the problem occurs in GDAL 1.3.x and 1.4.0 but seems to be fixed in current SVN.

As it's not fixed in 1.4.0, should I port the fix to the stable 1.4 branch ?

comment:11 Changed 13 years ago by Mateusz Łoskot

Just for records, perhaps, this issue is related to #757 and / or #1359 somehow.

comment:12 Changed 13 years ago by Mateusz Łoskot

I have to confess it's hard to compare changes that have been submitted to the current trunk since 1.3.1/1.4.0 releases, even only those in gdal/frmts/gtiff directory.

There has been a big number of commits between subsequent tags and trunk:

1.3.11.4.0trunk
r8551r10463r11180

The libtiff has been updated twice since 1.3.1.

Here is the complete list of revision logs starting at r11180 and listed back.

comment:13 Changed 13 years ago by warmerdam

Milestone: 1.4.11.4.2

Deferring to 1.4.2. It's too late for non-critical changes in 1.4.1.

comment:14 Changed 13 years ago by luiz

Version: unspecified1.4.1

Frank and Mateusz, bad news.

After some tests with new ( gdal 1.4.1 ) downloaded from www.gdal.org something has changed. In the old version, width values ranging from 1 to 2048 were working, but for width values higher than 2048 GDALRasterIO (with LZW enabled) it was craching and with LZW disabled the output file was nonsense (huge file). Now in this new version( 1.4.1) the only thing that seems to have changed is that: the system will crash for width values over 8100 insted of 2048. Otherwise the simptons are the same. In other words, the bug is still there.

printf("fast without LZW , working... 2048 x 2048\n"); plotColorMapAndSave(ba, "/tmp/good.tif", 3, indexes, r, g, b, a); printf("slow without LZW , bad ... 2049 x 2049\n"); BufferArea?* ba2 = createBufferAreaTest(8200,8200); <--- correct the bug example this way. plotColorMapAndSave(ba2 , "/tmp/bad.tif", 3, indexes, r, g, b, a);

comment:15 Changed 13 years ago by Mateusz Łoskot

Luiz,

I compiled GDAL from current SVN soruces and I did make the test with your new values and here is what I got:

slow without LZW , bad    ... 2049 x 2049
GDAL: GDALDriver::Create(GTiff,/tmp/bad.tif,8200,8200,5,UInt32,0x108b1b80)
line: 0
line: 1
line: 2
...
line: 8101
...
line: 8197
line: 8198
line: 8199
GDAL: GDALClose(/tmp/bad.tif)
GDAL: 270600 block reads on 1089 block band 1 of /tmp/bad.tif.
GDAL: GDALDeregister_GTiff() called.
  • After ~30 minutes I got
-rw-r--r-- 1 mloskot mloskot 1.4G 2007-04-14 01:35 bad.tif
  • Here is what gdalinfo says about the bad.tif file
mloskot:~/dev/gdal/bugs/1507$ gdalinfo /tmp/bad.tif 
GDAL: GDALOpen(/tmp/bad.tif) succeeds as GTiff.
Driver: GTiff/GeoTIFF
Size is 8200, 8200
Coordinate System is:
GEOGCS["WGS 84",
    DATUM["WGS_1984",
        SPHEROID["WGS 84",6378137,298.2572235629972,
            AUTHORITY["EPSG","7030"]],
        AUTHORITY["EPSG","6326"]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433],
    AUTHORITY["EPSG","4326"]]
Origin = (-46.677285000000005,-23.422715000000000)
Pixel Size = (0.000018850000000,-0.000018850000000)
Metadata:
  AREA_OR_POINT=Area
OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs 
Corner Coordinates:
Upper Left  ( -46.6772850, -23.4227150) ( 46d40'38.23"W, 23d25'21.77"S)
Lower Left  ( -46.6772850, -23.5772850) ( 46d40'38.23"W, 23d34'38.23"S)
Upper Right ( -46.5227150, -23.4227150) ( 46d31'21.77"W, 23d25'21.77"S)
Lower Right ( -46.5227150, -23.5772850) ( 46d31'21.77"W, 23d34'38.23"S)
Center      ( -46.6000000, -23.5000000) ( 46d36'0.00"W, 23d30'0.00"S)
Band 1 Block=256x256 Type=UInt32, ColorInterp=Gray
Band 2 Block=256x256 Type=UInt32, ColorInterp=Undefined
Band 3 Block=256x256 Type=UInt32, ColorInterp=Undefined
Band 4 Block=256x256 Type=UInt32, ColorInterp=Undefined
Band 5 Block=256x256 Type=UInt32, ColorInterp=Undefined
GDAL: GDALClose(/tmp/bad.tif)
GDAL: GDALDeregister_GTiff() called.

I think it works for me, doesn't it?

comment:16 Changed 13 years ago by luiz

Mateusz, I compiled GDAL (1.4.1 from www.gdal.org) and did this test as you did and my results were:

using this settings: BufferArea?* ba2 = createBufferAreaTest(8200,8200);

ps: It was too slow as you said ( ~20 min in my machine )

[luiz@core2duo gdal]$ gcc gdal_example.c -lgdal gdal_example.c: In function ‘prepareOutputRaster’: gdal_example.c:104: warning: cast to pointer from integer of different size [luiz@core2duo gdal]$ time ./a.out . . . line: 8097 line: 8098 line: 8099 done.

[luiz@core2duo tmp]$ ls -lah bad.tif -rw-rw-r-- 1 luiz luiz 4.9G Apr 13 21:11 bad.tif

[luiz@core2duo tmp]$ gdalinfo bad.tif Driver: GTiff/GeoTIFF Size is 8200, 8200 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.2572235630016,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]

Origin = (-46.677285000000005,-23.422715000000000) Pixel Size = (0.000018850000000,-0.000018850000000) Metadata:

AREA_OR_POINT=Area

Corner Coordinates: Upper Left ( -46.6772850, -23.4227150) ( 46d40'38.23"W, 23d25'21.77"S) Lower Left ( -46.6772850, -23.5772850) ( 46d40'38.23"W, 23d34'38.23"S) Upper Right ( -46.5227150, -23.4227150) ( 46d31'21.77"W, 23d25'21.77"S) Lower Right ( -46.5227150, -23.5772850) ( 46d31'21.77"W, 23d34'38.23"S) Center ( -46.6000000, -23.5000000) ( 46d36'0.00"W, 23d30'0.00"S) Band 1 Block=256x256 Type=UInt32, ColorInterp?=Gray Band 2 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 3 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 4 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 5 Block=256x256 Type=UInt32, ColorInterp?=Undefined


now compressing with LZW

using this... papszOptions = CSLSetNameValue( papszOptions, "COMPRESS", "LZW");

[luiz@core2duo gdal]$ gcc gdal_example.c -lgdal gdal_example.c: In function ‘prepareOutputRaster’: gdal_example.c:104: warning: cast to pointer from integer of different size [luiz@core2duo gdal]$ time ./a.out . . . line: 0 line: 1 Segmentation fault


using this settings: BufferArea?* ba2 = createBufferAreaTest(8100,8100);

Executing time: ~31s

[luiz@core2duo tmp]$ ls -lah bad.tif -rw-rw-r-- 1 luiz luiz 1.3G Apr 13 21:24 bad.tif

[luiz@core2duo gdal]$ gcc gdal_example.c -lgdal gdal_example.c: In function ‘prepareOutputRaster’: gdal_example.c:104: warning: cast to pointer from integer of different size [luiz@core2duo gdal]$ time ./a.out . . . line: 8097 line: 8098 line: 8099 done. real 0m31.017s user 0m2.898s sys 0m3.993s


now compressing with LZW

using this... papszOptions = CSLSetNameValue( papszOptions, "COMPRESS", "LZW");

[luiz@core2duo tmp]$ ls -lah bad.tif -rw-rw-r-- 1 luiz luiz 4.4M Apr 13 21:32 bad.tif

Executing time: ~11s

[luiz@core2duo gdal]$ gcc gdal_example.c -lgdal gdal_example.c: In function ‘prepareOutputRaster’: gdal_example.c:104: warning: cast to pointer from integer of different size [luiz@core2duo gdal]$ time ./a.out . . . line: 8097 line: 8098 line: 8099 done. real 0m11.381s user 0m10.721s sys 0m0.274s


Same results for LZW enabled or not. Everything Working...

[luiz@core2duo tmp]$ gdalinfo bad.tif Driver: GTiff/GeoTIFF Size is 8100, 8100 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.2572235630016,

AUTHORITY["EPSG","7030"]],

AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0], UNIT["degree",0.0174532925199433], AUTHORITY["EPSG","4326"]]

Origin = (-46.676342500000004,-23.423657500000001) Pixel Size = (0.000018850000000,-0.000018850000000) Metadata:

AREA_OR_POINT=Area

Image Structure Metadata:

COMPRESSION=LZW

Corner Coordinates: Upper Left ( -46.6763425, -23.4236575) ( 46d40'34.83"W, 23d25'25.17"S) Lower Left ( -46.6763425, -23.5763425) ( 46d40'34.83"W, 23d34'34.83"S) Upper Right ( -46.5236575, -23.4236575) ( 46d31'25.17"W, 23d25'25.17"S) Lower Right ( -46.5236575, -23.5763425) ( 46d31'25.17"W, 23d34'34.83"S) Center ( -46.6000000, -23.5000000) ( 46d36'0.00"W, 23d30'0.00"S) Band 1 Block=256x256 Type=UInt32, ColorInterp?=Gray Band 2 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 3 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 4 Block=256x256 Type=UInt32, ColorInterp?=Undefined Band 5 Block=256x256 Type=UInt32, ColorInterp?=Undefined


What do you mean?

comment:17 Changed 12 years ago by warmerdam

Milestone: 1.4.21.4.3

Putting off to 1.4.3.

comment:18 Changed 12 years ago by Even Rouault

I finally figured what happens. Frank's initial intuition was good. It's a libtiff related problem. When using gdal 1.4.1 with external libtiff support (stock libtiff 3.8.2), I hit the bug. When using gdal 1.4.1 with internal libtiff support, no bug.

Here's the patch that backported to stock libtiff 3.8.2 solves the problem :

--- libtiff/tif_lzw.c   2006-03-21 17:42:50.000000000 +0100
+++ ../gdal-1.4.1/frmts/gtiff/libtiff/tif_lzw.c 2007-04-10 17:15:34.000000000 +0200
@@ -1,4 +1,4 @@
-/* $Id: tif_lzw.c,v 1.28 2006/03/16 12:38:24 dron Exp $ */
+/* $Id: tif_lzw.c,v 1.29 2006/09/27 22:39:00 fwarmerdam Exp $ */

 /*
  * Copyright (c) 1988-1997 Sam Leffler
@@ -251,6 +251,9 @@

        (void) s;
        assert(sp != NULL);
+        if( sp->dec_codetab == NULL )
+            LZWSetupDecode( tif );
+
        /*
         * Check for old bit-reversed codes.
         */
@@ -350,6 +353,7 @@

        (void) s;
        assert(sp != NULL);
+        assert(sp->dec_codetab != NULL);
        /*
         * Restart interrupted output operation.
         */
@@ -734,6 +738,10 @@

        (void) s;
        assert(sp != NULL);
+
+        if( sp->enc_hashtab == NULL )
+            LZWSetupEncode( tif );
+
        sp->lzw_nbits = BITS_MIN;
        sp->lzw_maxcode = MAXCODE(BITS_MIN);
        sp->lzw_free_ent = CODE_FIRST;
@@ -803,6 +811,9 @@
        (void) s;
        if (sp == NULL)
                return (0);
+
+        assert(sp->enc_hashtab != NULL);
+
        /*
         * Load local state.
         */

So, luiz, I guess you should either build gdal with internal libtiff support or upgrade your libtiff to 3.9.0beta

comment:19 Changed 12 years ago by Mateusz Łoskot

rouault,

Thanks for investigating the problem.

Frank,

I suppose I have nothing to do in GDAL regarding this bug and I can close it as wontfix, am I right?

comment:20 Changed 12 years ago by warmerdam

Resolution: fixed
Status: assignedclosed

As noted, it appears that the issue is fixed in the gdal 1.4 branch version of libtiff as well as the version of libtiff included in trunk, so I am closing the bug report.

If anyone finds that the bug reports in 1.4 branch or trunk please reopen. It is understood that the bug will still occur if current versions of GDAL are used with old releases of libtiff.

Note: See TracTickets for help on using tickets.