Building pyramids causes segfault

[gsherman]

Building pyramids segfaults. Image size was 8192x4096.

Backtrace:

(gdb) bt
#0  0xb62424c7 in memset () from /lib/tls/i686/cmov/libc.so.6
#1  0x00002000 in ?? ()
#2  0xb6be5c85 in _TIFFmemset () from /usr/local/lib/libgdal.so.1
#3  0xb6bd4ab2 in LZWPreDecode () from /usr/local/lib/libgdal.so.1
#4  0xb6be372c in TIFFStartTile () from /usr/local/lib/libgdal.so.1
#5  0xb6be330d in TIFFFillTile () from /usr/local/lib/libgdal.so.1
#6  0xb6be2bce in TIFFReadEncodedTile () from /usr/local/lib/libgdal.so.1
#7  0xb6ab8bd5 in GTiffDataset::LoadBlockBuf () from /usr/local/lib/libgdal.so.1
#8  0xb6aba436 in GTiffRasterBand::IReadBlock () from /usr/local/lib/libgdal.so.1
#9  0xb6c10691 in GDALRasterBand::GetLockedBlockRef () from /usr/local/lib/libgdal.so.1
#10 0xb6c22235 in GDALRasterBand::IRasterIO () from /usr/local/lib/libgdal.so.1
#11 0xb6c0f4fc in GDALRasterBand::RasterIO () from /usr/local/lib/libgdal.so.1
#12 0xb6c1efc8 in GDALDownsampleChunk32R () from /usr/local/lib/libgdal.so.1
#13 0xb6c20156 in GDALRegenerateOverviews () from /usr/local/lib/libgdal.so.1
#14 0xb6abd0c1 in GTiffDataset::IBuildOverviews () from /usr/local/lib/libgdal.so.1
#15 0xb6bff0b6 in GDALDataset::BuildOverviews () from /usr/local/lib/libgdal.so.1
#16 0xb7ee7529 in QgsRasterLayer::buildPyramids (this=0x82abfa0, theRasterPyramidList=@0xbfd31eac,
    theResamplingMethod=@0xbfd31eb0) at qgsrasterlayer.cpp:4216
#17 0xb65ccaf1 in QgsRasterLayerProperties::on_buttonBuildPyramids_clicked (this=0x8319d80)
    at qgsrasterlayerproperties.cpp:478

[timlinux]

Moved to milestone 0.8.2 since we wont be fixing any further issues before the 0.8.1 release

[warmerdam]

I believe an upgrade to GDAL 1.4.4 with internal libtiff, or libtiff 3.9 (from CVS) will resolve this problem. It is a libtiff bug, not a GDAL or QGIS bug.

[msieczka]

I have just checked in QGIS r8190, built against GDAL 1.5.0+SVN 2008-03-04 (internal libtiff). No crashes building overviews in QGIS whatsoever. Checked with several Byte and Uint16 GeoTIFFs, all having colortables, compressed with lzw, deflate or packbits.

I still believe however that building external overviews in Erdas Imagine format would be more bulletproof and universal. The benefits of external overviews:

1. The input data are not modified.

2. Ergo, input data remain compatible with software that don't tolerate internal overviews and

3. there's no need to duplicate data for safety/usability reasons before building overviews.

4. Other software support external Erdas Imagine overviews that GDAL creates - eg. ArcGIS, ERDAS Imagine.

What do developers think about it?

For now, to create an external Erdas Imagin overwievs from CLI use eg.:

gdaladdo --config USE_RRD YES some.tif 2 4 8 16

[msieczka]

Since my last #comment:4 regarding overwievs, I have learned that external overviews in *GeoTIFF* are also a nice option. They meet points 1-3 too, and although they are not so compatible as Erdas overviews, they are more compressable beacuse you can use DEFLATE for TIFF, which is more efficient than Erdas RLC compression method (the only available for Erdas, applied by default to Erdas overviews; and can't be disabled for overviews AFAIK).

Compare (the input is a 361 MB uncompressed GeoTIFF):

$ gdaladdo --config USE_RRD YES atif.tif 2 4 8 16
$ ls -sh atif.aux
125M atif.aux

$ chmod a-w atif.tif
$ gdaladdo --config COMPRESS_OVERVIEW DEFLATE atif.tif 2 4 8 16
$ ls -sh atif.tif.ovr
103M atif.tif.ovr

So, ideally, QGIS should let the user choose if he wants internal overviews (and if/what compresssion scheme) or external overviews - in Erdas or GeoTIFF format (the former for better compatibility, the latter for space saving - if chosen let the user choose DEFLATE/LZW/PACKBITS; hint that DEFLATE compresses best and better than Erdas compressions).

[dickeya]

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file? From my understanding of aux files, this should contain a small amount of information, usually histogram / coordinate / projection info / etc. The rrd should contain the pyramids, and in my local tests, this is the case.

[msieczka]

Replying to dickeya:

One question:

In the first example you show a ".aux" file of 125M. Shouldn't this be the ".rrd" file?

My *.aux file is a direct output of a command like 'gdaladdo --config USE_RRD YES some.tif 2 4 8 16' and the command does not produce anything else. If you get a file with different extension then I don't know why. I'm using GDAL 1.5.2 + SVN r15027 FWIW.

The rrd should contain the pyramids, and in my local tests, this is the case.

Do you mean gdaladdo output? Please copy&paste the command syntax, gdalinfo of input file, the list of output files, gdalinfo of what you believe is the overviews file and your GDAL version.

[dickeya]

As it turns out, I had HFA_USE_RRD set to YES in my environment. After removal, gdaladdo only created an aux file.

Thanks.

[timlinux]

With r8968 you can now specify internal or external overviews. Compressed overviews are not yet supported since ​http://www.gdal.org/classGDALDataset.html#2aa6f88b3bbc840a5696236af11dde15 provides no obvious way to do this (I would need to go and read gdaladdo code and see how they did it there). If you really want this feature, please file a separate enhancement ticket since we are wondering way off the topic of the original ticket.

Best regards

Tim

[msieczka]

Replying to timlinux:

With r8968 you can now specify internal or external overviews.

Cool. Smallish issue - there is a redundant test for file write access if the user chooses to build external overviews. It effectively disables external overviews creation for read-only rasters. As we are not going to alter the file, we don't care if it is writeable.

Compressed overviews (...) . If you really want this feature, please file a separate enhancement ticket since we are wondering way off the topic of the original ticket.

Done in #1199.

[timlinux]

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull()) 1602 { 1603 if (res == "ERROR_WRITE_ACCESS") 1604 { 1605 QMessageBox::warning(this, tr("Write access denied"), 1606 tr("Write access denied. Adjust the file permissions and try again.\n\n") ); 1607 } 1608 else if (res == "ERROR_WRITE_FORMAT") 1609 { 1610 QMessageBox::warning(this, tr("Building pyramids failed."), 1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") ); 1612 } 1613 else if (res == "FAILED_NOT_SUPPORTED") 1614 { 1615 QMessageBox::warning(this, tr("Building pyramids failed."), 1616 tr("Building pyramid overviews is not supported on this type of raster.") ); 1617 } 1618 }

If gdal reports an error my feeling is that we should report it. Perhaps I will just add a message saying that the warning can be ignored if building external overviews and the dir is writable.

Regards

Tim

[msieczka]

Replying to timlinux:

Hi

Its strange, because that message is propogated from GDAL as shown in the following snippet:

1601 if (!res.isNull()) 1602 { 1603 if (res == "ERROR_WRITE_ACCESS") 1604 { 1605 QMessageBox::warning(this, tr("Write access denied"), 1606 tr("Write access denied. Adjust the file permissions and try again.\n\n") ); 1607 } 1608 else if (res == "ERROR_WRITE_FORMAT") 1609 { 1610 QMessageBox::warning(this, tr("Building pyramids failed."), 1611 tr("The file was not writeable. Some formats do not support pyramid overviews. Consult the GDAL documentation if in doubt.") ); 1612 } 1613 else if (res == "FAILED_NOT_SUPPORTED") 1614 { 1615 QMessageBox::warning(this, tr("Building pyramids failed."), 1616 tr("Building pyramid overviews is not supported on this type of raster.") ); 1617 } 1618 }

If gdal reports an error my feeling is that we should report it.

Hmm, probably GDAL reports this error in error. It is logical to discard it IMO.

Perhaps I will just add a message saying that the warning can be ignored if building external overviews and the dir is writable.

I think QGIS should not bother about theis error at all and just proceed. There is no point in reporting that the file is not writeable if we don't try to modify it.

Another small issue I noticed: even if QGIS fails to build the overviews for whatever reason, the icons for selected overviews change like if the overviews were actually created. Re-entering raster's properties shows they were really were not.

Regards

Tim

[(none)]

Milestone Version 1.0.0 deleted