Opened 16 months ago

Closed 16 months ago

Last modified 16 months ago

#6982 closed defect (invalid)

GDAL crash for single tile in multi-threaded application

Reported by: muzammil360 Owned by: warmerdam
Priority: normal Milestone:
Component: GDAL_Raster Version: unspecified
Severity: normal Keywords:
Cc:

Description

Hi guys, I am trying to run GDAL in a multi-threaded application. What I am trying to do is read the same 256*256 tile from a geoTIFF dataset in multiple threads (std::thread).

If I run only 1 thread, there is no issue. But if I run 2 or 3 threads, sometime they work and at other time, the application crash. The application crashes more often when more threads are used.

I understand that this is probably because some functions are not thread-safe. Is there any alternative way? Can I write my own thread-safe implementation of RasterIO? And more importantly, is a fix planned in the upcoming releases?

Here is my error log on console:


ERROR 1: TIFFFillTile:Read error at row 4294967295, col 4294967295; got 0 bytes, expected 159996

ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted at scanline 2304

ERROR 1: LZWDecode:Corrupted LZW table at scanline 2048

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 9, Y offset 8

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: GetBlockRef? failed at X block offset 9, Y block offset 8

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 8, Y offset 9

ERROR 1: GetBlockRef? failed at X block offset 8, Y block offset 9

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 8, Y offset 8

ERROR 1: GetBlockRef? failed at X block offset 8, Y block offset 8

Warning 1: LZWDecode:LZWDecode: Strip -1 not terminated with EOI code

ERROR 1: LZWDecode:Not enough data at scanline 2304 (short 15 bytes)

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 9, Y offset 9

ERROR 1: GetBlockRef? failed at X block offset 9, Y block offset 9


Thank you for your time.

Change History (3)

comment:1 Changed 16 months ago by Even Rouault

Resolution: invalid
Status: newclosed

The rule is that only one single thread must access a GDALDataset* object at a single time. This is by design. So open a GDALDataset* for each thread. See https://trac.osgeo.org/gdal/wiki/FAQMiscellaneous#IstheGDALlibrarythread-safe

comment:2 Changed 16 months ago by muzammil360

Hi Even. I am making an image viewer applications and it requires multi-threading as a must for performance reasons.

I have already tested that approach along with my manual mutex implementation in C++.

If I open a new dataset in every thread, then it takes approximately double the time to complete the process than it takes with my mutex implementation.

Is any parallelism planned for RasterIO function in future releases?

comment:3 Changed 16 months ago by Even Rouault

I'm a bit surprised you get better performance with a mutex protecting a single GDALDataset* object rather than operating on different GDALDataset*. The reason might be that your different threads read in very close areas so the block caching make things faster. I don't have personnaly immediate plans to have a mutex protected GDALDataset* but one possible approach would be to follow the approach taken into ogr/ogrsf_frmts/generic/ogrmutexeddatasource.cpp and generalize it to GDALDataset (since OGRDataSource is now mostly a simple extension of GDALDataset), to have a GDALMutexedDataset class that would take a regular GDALDataset* instance and protect all calls with a mutex. If you want to tackle that, a pull request against the github repository would be welcome

Note: See TracTickets for help on using tickets.