issue creating large geotiffs from virt driver

[aleks]

I'm having some trouble creating geotiffs in certain scenarios and i can't figure out if it's something different happening when it works vs when it doesn't work or if it's simply a bug in gdal.

so I have two test cases attached.. one where a large (100k x 100k pixels) geotiff creates successfully and one where a similar image with just different data does not create successfully.

I define "successfully" if the default system viewer on my platform is able to open/recognize the image. they are from a different 100km square region in the UK but otherwise should be identical so they should both work fine.

The two test cases are of 1:10000 scale raster maps. One is the NK data set which fine and one is the NL data set which will not be recognized as a valid file. Both are created from using gdal_translate on a .vrt file pointing to 100 smaller tiles. The command used to create them both is:

$ for i in NK.vrt NL.vrt; do gdal_translate -mo TIFFTAG_XRESOLUTION=254 -mo TIFFTAG_RESOLUTIONUNIT=2 -mo TIFFTAG_IMAGEDESCRIPTION='1:10000 TILE '${i/.vrt/} -mo TIFFTAG_YRESOLUTION=254 -a_srs EPSG:27700 -co NUM_THREADS=ALL_CPUS -co BIGTIFF=NO -co COMPRESS=DEFLATE -of GTiff $i ${i/.vrt/.TIFF}; done

I am using BIGTIFFF=no as i found the system won't be able to read the files ever if i use YES, I also tried to use -co PROFILE=BASELINE to form a perhaps more compliant TIFF file but the system still claims the created file is damaged/unreadable format.

BTW, my entire data set (of which these two files are just a sample of good vs bad), is 57 TIFFs, and 14 of these are unreadable (and with all these, the created .TIFF files strangely always result in under 100 meg mark when bad, seems any file produced over 100 megs end up being one of the 43 working .TIFF files).

I have uploaded both the good/bad created TIFFs, as well as the .vrt used to make them, as well as the source 200 .TIFF files the .vrt references:

​https://www.dropbox.com/s/6cwahje7stunx7v/bug-report-20170102-aleks.zip?dl=0

The attached file is ~263 megs big, md5 sum is: 5475898a5e3cf625d09f8ffb1b17ebef

My platform is Mac OS X 10.8.5 (build 12F2560) on a macbook pro (15" retina model from 2012). And the default system viewer is called 'Preview'. It is also built-into the system and can 'quick-view' images showing a thumbnail/preview, instead of opening the standalone 'Preview' app. Neither works for the second test case.

I am using GDAL as compiled by the Anaconda python distribution with the following package versions:

gdal 2.1.0

geos 3.5.0

geotiff 1.4.1

libgdal 2.1.0

libtiff 4.0.6

Here is the output of tiffutil -dump on both images:

$ tiffutil -dump NL.TIFF Magic: 0x4949 <little-endian> Version: 0x2a Directory 0: offset 8 (0x8) next 0 (0) ImageWidth (256) LONG (4) 1<100000> ImageLength (257) LONG (4) 1<100000> BitsPerSample (258) SHORT (3) 3<8 8 8> Compression (259) SHORT (3) 1<8> Photometric (262) SHORT (3) 1<2> ImageDescription (270) ASCII (2) 16<1:10000 TILE NL\0> StripOffsets (273) LONG (4) 100000<800605 803879 807282 810711 814012 817478 820906 824427 835485 839015 842414 828084 831701 845955 850050 858550 854155 862868 867187 871462 884478 875830 880180 893161 ...> SamplesPerPixel (277) SHORT (3) 1<3> RowsPerStrip (278) SHORT (3) 1<1> StripByteCounts (279) LONG (4) 100000<3274 3403 3429 3301 3466 3428 3521 3657 3530 3399 3541 3617 3784 4095 4105 4318 4395 4319 4275 4368 4480 4350 4298 4306 ...> XResolution (282) RATIONAL (5) 1<254> YResolution (283) RATIONAL (5) 1<254> PlanarConfig (284) SHORT (3) 1<1> ResolutionUnit (296) SHORT (3) 1<2> Predictor (317) SHORT (3) 1<1> SampleFormat (339) SHORT (3) 3<1 1 1> Copyright (33432) ASCII (2) 37<ORDNANCE SURVEY CROWN CO ...> 33550 (0x830e) DOUBLE (12) 3<1 1 0> 33922 (0x8482) DOUBLE (12) 6<0 0 0 0 800000 0> 34735 (0x87af) SHORT (3) 36<1 1 0 8 1024 0 1 1 1025 0 1 1 1026 34737 34 0 2049 34737 10 34 2054 0 1 9102 ...> 34736 (0x87b0) DOUBLE (12) 7<446.448 -125.157 542.06 0.15 0.247 0.842 -20.489> 34737 (0x87b1) ASCII (2) 45<OSGB 1936 / British Nati …>

$ tiffutil -dump NK.TIFF Magic: 0x4949 <little-endian> Version: 0x2a Directory 0: offset 8 (0x8) next 0 (0) ImageWidth (256) LONG (4) 1<100000> ImageLength (257) LONG (4) 1<100000> BitsPerSample (258) SHORT (3) 3<8 8 8> Compression (259) SHORT (3) 1<8> Photometric (262) SHORT (3) 1<2> ImageDescription (270) ASCII (2) 16<1:10000 TILE NK\0> StripOffsets (273) LONG (4) 100000<800605 800923 801559 801877 801241 802195 802513 802831 803785 803149 803467 804103 804421 804739 805375 805693 805057 806011 806329 806647 806965 807283 807601 807919 ...> SamplesPerPixel (277) SHORT (3) 1<3> RowsPerStrip (278) SHORT (3) 1<1> StripByteCounts (279) LONG (4) 100000<318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 318 ...> XResolution (282) RATIONAL (5) 1<254> YResolution (283) RATIONAL (5) 1<254> PlanarConfig (284) SHORT (3) 1<1> ResolutionUnit (296) SHORT (3) 1<2> Predictor (317) SHORT (3) 1<1> SampleFormat (339) SHORT (3) 3<1 1 1> Copyright (33432) ASCII (2) 37<ORDNANCE SURVEY CROWN CO ...> 33550 (0x830e) DOUBLE (12) 3<1 1 0> 33922 (0x8482) DOUBLE (12) 6<0 0 0 400000 900000 0> 34735 (0x87af) SHORT (3) 36<1 1 0 8 1024 0 1 1 1025 0 1 1 1026 34737 34 0 2049 34737 10 34 2054 0 1 9102 ...> 34736 (0x87b0) DOUBLE (12) 7<446.448 -125.157 542.06 0.15 0.247 0.842 -20.489> 34737 (0x87b1) ASCII (2) 45<OSGB 1936 / British Nati ...>

[aleks]

seems there's a size limit for the built in file-attachment system, so i uploaded them to dropbox:

​https://www.dropbox.com/s/6cwahje7stunx7v/bug-report-20170102-aleks.zip?dl=0

[Jukka Rahkonen]

Both files, NL.TIFF and NK.TIFF opens without problems in QGIS. The main issue is perhaps in the viewer you are using but I do not know to whom in Apple you should write and ask for their opinion.

At first sight this does not feel like a bug in GDAL. Instead of creating bug reports right ahead I recommend to send mail to gdal-dev mailing list which is also serving as gdal users mailing list.

I would do a bunch of tests by converting NL.TIFF (if that is the one that makes the Apple viewer to fail) with gdal_translate to other kind of tiff files by playing with the parameters and see it viewer is happy with some variant.

[Even Rouault]

I concur with Jukka. Both files seem to be perfectly sane under QGIS or when running gdalinfo -checksum on them. It is likely a defect of the viewer. Very few general purpose image viewers are expeced to properly deal with such big images. What is surprising though is that both files have same dimension and structure. Only Apple can investigate what is going wrong I'm afraid.

I'm aware that some GIS vendors have or had issue reading files with DEFLATE compression. Perhaps you could try LZW instead. But I would expect such a problem to be of the black&white type: either this never works, either this always works, but not for half of the images.

You could also try other variations like -co TILED=YES / -co INTERLEAVE=BAND .

Closing as not a GDAL bug. Feel free to reopen if you have more precise elements that would indicate there's really an isue.