Opened 12 years ago

Closed 12 years ago

#3106 closed defect (fixed)

gdal_translate using CRL create read error

Reported by: cumpston Owned by: warmerdam
Priority: normal Milestone: 1.7.0
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords: INGR
Cc: ilucena

Description

It seems as through any time I try to use gdal_translate with a Intergraph CRL file as input the processing get to about the 90% point and I get the following error message...

The instruction as "0x100f4e40" referenced memory at "0x01bd3000". The memory could not be "read".

A sample command line looks like: gdal_translate test.crl test.tif

I can provide sample CRLs if you want, just let me know how to get them to you. They are being generated by Microstation 95 and seem to work in other applications.

Thanks,

Mark C.

Attachments (2)

12_11378_07152009_094556_36.crl (475.8 KB) - added by cumpston 12 years ago.
14_11378_07152009_094556_36.crl (478.3 KB) - added by cumpston 12 years ago.

Download all attachments as: .zip

Change History (15)

comment:1 Changed 12 years ago by warmerdam

Component: defaultGDAL_Raster
Keywords: INGR added
Status: newassigned

Mark,

Yes, please attach or otherwise provide me a sample file and I'll dig into this.

comment:2 Changed 12 years ago by warmerdam

Mark? Can you provide the data?

Changed 12 years ago by cumpston

Changed 12 years ago by cumpston

comment:3 Changed 12 years ago by cumpston

I sent some samples to your pobox.com email a few weeks ago and then again just now. I have also uploaded them here. Let me know if you need anything else,

Take care,

Mark C.

comment:4 Changed 12 years ago by warmerdam

Mark,

Sorry for dropping the ball on the files.

I have tested development trunk and 1.6 branch on the files on linux and windows and I'm not seeing any problems, even under valgrind (a memory checking tool).

What version of GDAL are you using?

Best regards,

comment:5 Changed 12 years ago by cumpston

Sorry about that, I guess the failure mode depends on the file size.

Most of the CRLs that I have are 10-20MB and I can't upload them here, the max size seems to be 1 MB.

If you look at the .tif files that come out of the translate though you will see a different kind of problem....it looks like the row length is changing for each row.

You can get a large file sample at: http://www.mcumpston.com/2885_11312_07152009_092830_5.crl

I just downloaded gdalwin32exe160.zip and that's the version of GDAL I'm using.

Let me know if you need anything else, Thanks for the help,

Mark C.

comment:6 Changed 12 years ago by warmerdam

Cc: ilucena added

I was unable to reproduce a crash, but I certainly see the image corruption. I did however turn up a valgrind report which might be related to the crash.

==9017== Invalid read of size 2
==9017==    at 0x520E75C: INGR_DecodeRunLengthPaletted(unsigned char*, unsigned char*, unsigned, unsigned, unsigned*) (IngrTypes.cpp:1009)
==9017==    by 0x520E964: INGR_Decode(INGR_Format, unsigned char*, unsigned char*, unsigned, unsigned, unsigned*) (IngrTypes.cpp:923)
==9017==    by 0x521273B: IntergraphRLEBand::IReadBlock(int, int, void*) (IntergraphBand.cpp:609)
==9017==    by 0x538B506: GDALRasterBand::GetLockedBlockRef(int, int, int) (gdalrasterband.cpp:1233)
==9017==    by 0x53A200D: GDALRasterBand::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int) (rasterio.cpp:89)
==9017==    by 0x536670B: GDALDataset::IRasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int*, int, int, int) (gdaldataset.cpp:1427)
==9017==    by 0x536628C: GDALDataset::RasterIO(GDALRWFlag, int, int, int, int, void*, int, int, GDALDataType, int, int*, int, int, int) (gdaldataset.cpp:1620)
==9017==    by 0x539EF7D: GDALDatasetCopyWholeRaster (rasterio.cpp:2071)
==9017==    by 0x51750A4: GTiffDataset::CreateCopy(char const*, GDALDataset*, int, char**, int (*)(double, char const*, void*), void*) (geotiff.cpp:6346)
==9017==    by 0x536D66E: GDALDriver::CreateCopy(char const*, GDALDataset*, int, char**, int (*)(double, char const*, void*), void*) (gdaldriver.cpp:635)
==9017==    by 0x536D83D: GDALCreateCopy (gdaldriver.cpp:674)
==9017==    by 0x403F08: ProxyMain(int, char**) (gdal_translate.cpp:618)
==9017==    by 0x404D62: main (gdal_translate.cpp:962)
==9017==  Address 0xCA7E954 is 0 bytes after a block of size 21,907,748 alloc'd
==9017==    at 0x4C21C16: malloc (vg_replace_malloc.c:149)
==9017==    by 0x53C3DD6: VSIMalloc (cpl_vsisimple.cpp:300)
==9017==    by 0x53A55D8: CPLMalloc (cpl_conv.cpp:128)
==9017==    by 0x52144FB: IntergraphRLEBand::IntergraphRLEBand(IntergraphDataset*, int, int, int) (IntergraphBand.cpp:501)
==9017==    by 0x52165B2: IntergraphDataset::Open(GDALOpenInfo*) (IntergraphDataset.cpp:344)
==9017==    by 0x536704D: GDALOpen (gdaldataset.cpp:1999)
==9017==    by 0x5367597: GDALOpenShared (gdaldataset.cpp:2111)
==9017==    by 0x40351E: ProxyMain(int, char**) (gdal_translate.cpp:390)
==9017==    by 0x404D62: main (gdal_translate.cpp:962)

Ivan - is this a ticket you would be interested in digging into? It does not appear the image corruption relates to the changes last year to treat big one tile images as a series of scanlines since the image corruption also occurs with 1.5.x.

comment:7 Changed 12 years ago by ilucena

Resolution: fixed
Status: assignedclosed

Frank,

I tested those two files 12_11378_07152009_094556_36.crl and 14_11378_07152009_094556_36.crl in Windows XP and OpenSUSE and that is what I got:

 gdalinfo 12_11378_07152009_094556_36.crl 
ERROR 4: `12_11378_07152009_094556_36.crl' not recognised as a supported file format.

gdalinfo failed - unable to open '12_11378_07152009_094556_36.crl'.

But I believe a found a solution that works well for the sample file 2885_11312_07152009_092830_5.crl and for the gdal_autotest/gdrivers/ingr.py test case 9.

ilucena@think:~/Data> gdalinfo 12_11378_07152009_094556_36.crl 
ERROR 4: `12_11378_07152009_094556_36.crl' not recognised as a supported file format.

gdalinfo failed - unable to open '12_11378_07152009_094556_36.crl'.
ilucena@think:~/Data> more log.txt
==11541== Memcheck, a memory error detector.
==11541== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==11541== Using LibVEX rev 1804, a library for dynamic binary translation.
==11541== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==11541== Using valgrind-3.3.0, a dynamic binary instrumentation framework.
==11541== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==11541== For more details, rerun with: -v
==11541== 
==11541== 
==11541== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 9 from 1)
==11541== malloc/free: in use at exit: 0 bytes in 0 blocks.
==11541== malloc/free: 155,083 allocs, 155,083 frees, 2,362,119,246 bytes alloca
ted.
==11541== For counts of detected errors, rerun with: -v
==11541== All heap blocks were freed -- no leaks are possible.

Committed revision 17655.

comment:8 Changed 12 years ago by cumpston

I just built gdal-svn-trunk-2009.09.18 and the large CRLs seem to complete now, but the image that comes out is really distorted.

Are you all going to continue to work on this?

Thanks,

Mark C.

comment:9 Changed 12 years ago by ilucena

Files generated by Intergraphs's ISRU (Image Station Raster Utility) does not carry a key code 0x5901 like yours. ISRU uses only the 0x5900 as a break point on RLE. That doesn't mean that your file is corrupted or malformed. ISRU can read, display and convert it properly as it. We don't have a good source of documentation for that algorithm so it took me hours looking at the hexadecimal editor to figure it out what was happening. I add that 0x5901 code to the logic of the INGR decoder and the converted image is perfect now but I am afraid we are going to find more of those "surprise" codes along the way.

comment:10 Changed 12 years ago by cumpston

Thanks for all the help with this. I know it must be frustrating having to deal with something like this without good documentation. I really wish the organization that produces these files would adopt a more supported image format, but unfortunately I don't have much say in this....

Thanks again,

Mark C.

comment:11 Changed 12 years ago by warmerdam

Milestone: 1.6.3
Resolution: fixed
Status: closedreopened

The fixes have been applied in trunk by Ivan - palette fix (r17655) and RLE color (r17660).

Ivan - any thoughts on whether these fixes are risky to back port? I'd like to take them into the 1.6 and 1.6-esri branches if practical.

comment:12 Changed 12 years ago by ilucena

I revised it on linux/valgrind and it is running clean with the provided dataset:

valgrind gdal_translate 2885_11312_07152009_092830_5.crl out.tif==26401== Memcheck, a memory error detector.
==26401== Copyright (C) 2002-2007, and GNU GPL'd, by Julian Seward et al.
==26401== Using LibVEX rev 1804, a library for dynamic binary translation.
==26401== Copyright (C) 2004-2007, and GNU GPL'd, by OpenWorks LLP.
==26401== Using valgrind-3.3.0, a dynamic binary instrumentation framework.
==26401== Copyright (C) 2000-2007, and GNU GPL'd, by Julian Seward et al.
==26401== For more details, rerun with: -v
==26401== 
Input file size is 30480, 38100
0...10...20...30...40...50...60...70...80...90...100 - done.
==26401== 
==26401== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 9 from 1)
==26401== malloc/free: in use at exit: 0 bytes in 0 blocks.
==26401== malloc/free: 155,095 allocs, 155,095 frees, 2,362,119,354 bytes allocated.
==26401== For counts of detected errors, rerun with: -v
==26401== All heap blocks were freed -- no leaks are possible.

That is the only dataset I have that contains the code 0x5901 so it is hard to say. Files generated by ISRU compressed as "Run Length Encoded Color" don't have that code, like "frmt10.cot" on gdal_autotest for example.

comment:13 Changed 12 years ago by warmerdam

Milestone: 1.6.41.7.0
Resolution: fixed
Status: reopenedclosed

I'm going to leave this in trunk unless someone comes up with a need to backport it.

Note: See TracTickets for help on using tickets.