GDAL crash for single tile in multi-threaded application

[muzammil360]

Hi guys, I am trying to run GDAL in a multi-threaded application. What I am trying to do is read the same 256*256 tile from a geoTIFF dataset in multiple threads (std::thread).

If I run only 1 thread, there is no issue. But if I run 2 or 3 threads, sometime they work and at other time, the application crash. The application crashes more often when more threads are used.

I understand that this is probably because some functions are not thread-safe. Is there any alternative way? Can I write my own thread-safe implementation of RasterIO? And more importantly, is a fix planned in the upcoming releases?

Here is my error log on console:

---

ERROR 1: TIFFFillTile:Read error at row 4294967295, col 4294967295; got 0 bytes, expected 159996

ERROR 1: TIFFReadEncodedTile() failed. ERROR 1: LZWDecode:Wrong length of decoded string: data probably corrupted at scanline 2304

ERROR 1: LZWDecode:Corrupted LZW table at scanline 2048

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 9, Y offset 8

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: GetBlockRef failed at X block offset 9, Y block offset 8

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 8, Y offset 9

ERROR 1: GetBlockRef failed at X block offset 8, Y block offset 9

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 8, Y offset 8

ERROR 1: GetBlockRef failed at X block offset 8, Y block offset 8

Warning 1: LZWDecode:LZWDecode: Strip -1 not terminated with EOI code

ERROR 1: LZWDecode:Not enough data at scanline 2304 (short 15 bytes)

ERROR 1: TIFFReadEncodedTile() failed.

ERROR 1: E:\nir.tif, band 1: IReadBlock failed at X offset 9, Y offset 9

ERROR 1: GetBlockRef failed at X block offset 9, Y block offset 9

---

Thank you for your time.

[Even Rouault]

The rule is that only one single thread must access a GDALDataset* object at a single time. This is by design. So open a GDALDataset* for each thread. See ​https://trac.osgeo.org/gdal/wiki/FAQMiscellaneous#IstheGDALlibrarythread-safe

[muzammil360]

Hi Even. I am making an image viewer applications and it requires multi-threading as a must for performance reasons.

I have already tested that approach along with my manual mutex implementation in C++.

If I open a new dataset in every thread, then it takes approximately double the time to complete the process than it takes with my mutex implementation.

Is any parallelism planned for RasterIO function in future releases?

[Even Rouault]

I'm a bit surprised you get better performance with a mutex protecting a single GDALDataset* object rather than operating on different GDALDataset*. The reason might be that your different threads read in very close areas so the block caching make things faster. I don't have personnaly immediate plans to have a mutex protected GDALDataset* but one possible approach would be to follow the approach taken into ogr/ogrsf_frmts/generic/ogrmutexeddatasource.cpp and generalize it to GDALDataset (since OGRDataSource is now mostly a simple extension of GDALDataset), to have a GDALMutexedDataset class that would take a regular GDALDataset* instance and protect all calls with a mutex. If you want to tackle that, a pull request against the github repository would be welcome