Internal TIFF mask behavior in uDig/qgis

[crschmidt]

Using an internal TIFF mask on a TIFF seems to have strange behavior in uDig: When opening the image with the mask, the mask seems 'reversed' -- the data that is supposed to be transparent (in my case, an edge) is displayed, and the center portion of the image is not.

With some tweaking of settings (changing the projection of the map) the result seems to be that the mask can be ignored, but I am unable to reproduce this in a way that I can share.

The way I produced the image was:

download http://gis2.amherstma.gov/massgis/sid_tiles_30cm/22829020.zip
./gdalmask/gdalmask -internal -nearblack -near 3 -nb 5 22829020_15.sid out.tiff (where gdalmask is winkey's tool from his sandbox)

This produces a mask with a 1px edge. To get a wider edge (more visible), you can:

gdalwarp -co TILED=YES -dstalpha -t_srs EPSG:4326  out.tiff  warped_file.tif
./gdalmask/gdalmask -co TILED=YES -co COMPRESS=JPEG -nearblack -near 0 -internal warped_file.tif final_file.tif

This creates an image with several pixels of 'mask' on the outside, which then does not render properly in uDig if you add this image to a map.

[crschmidt]

final_file.tif (or something that nearly approximates the above process) is available at ​http://hypercube.telascience.org/~crschmidt/final_file.tif . This one is compressed with YCbCr jpg and not tiled, but fails in the same way in uDig.

[crschmidt]

When using qgis 1.5.0, downloaded from the KyngChaos binaries page, the image renders without taking the mask into account, in general. However, with a 1024 x 613 rendering area, at certain scales, the image fails to render.

The times when it fails to render seem to be not-obvious: when zooming to the full extent of the file, at a scale of 1:3876, the image fails to render. However, dragging the map may cause it to render, and setting the scale via the scale control at the bottom to 1:3000 will also cause it to render.

SEtting it to 1:3900 will also fail, but setting the scale to 1:5500 will cause it to render (even though both of them are displaying the full image). 1:11000 fails, 1:22000 works, 1:44000 works, as do more zoomed out views.

No internal overviews have been built, and no file alongside the source file seems to exist; I have not selected any overview building in the qgis UI, and the "Contrast Enhancements" is set to "no Stretch" in the GUI.

[crschmidt]

Building the same file with either external mask (.msk file) or no mask, the result does not behave the same way; the mask is ignored (in the former case) or not present (in the latter), but the image appears to render at all scales, which is not the case with the internally masked file.

[crschmidt]

A followup to the 'tweaking of settings' comment above: it appears that 'overzooming' -- that is, zooming in beyond 1:1 -- will cause uDig to render the image. I assume that the warping kernel probably requests more data than it needs, which is why reprojection can trigger this case even when more zoomed out. However, in cases where the whole image is shown, uDig seems to consistently display the data as if the mask was reversed, and when zoomed in, displays as if the mask was not there, so in no case does the mask seem to work as expected.

[crschmidt]

tiffinfo on the file:

final_file.tif:
Magic: 0x4949 <little-endian> Version: 0x2a <ClassicTIFF>
Directory 0: offset 29953652 (0x1c90e74) next 4618 (0x120a)
ImageWidth (256) SHORT (3) 1<11419>
ImageLength (257) SHORT (3) 1<8467>
BitsPerSample (258) SHORT (3) 3<8 8 8>
Compression (259) SHORT (3) 1<7>
Photometric (262) SHORT (3) 1<6>
StripOffsets (273) LONG (4) 530<29385040 24444 29395704 109424 156872 204980 253034 29439200 350087 398230 29482396 29525282 543167 29568853 640879 29612077 731889 779086 29654920 29697723 922653 970862 29740093 1066482 ...>
SamplesPerPixel (277) SHORT (3) 1<3>
RowsPerStrip (278) SHORT (3) 1<16>
StripByteCounts (279) LONG (4) 530<10664 30437 43496 41975 42886 42663 43313 43196 42845 42627 42886 43571 43927 43224 43186 42843 41991 42436 42803 42370 42983 41793 43259 42687 ...>
PlanarConfig (284) SHORT (3) 1<1>
SampleFormat (339) SHORT (3) 3<1 1 1>
JPEGTables (347) UNDEFINED (7) 574<0xff 0xd8 0xff 0xdb 00 0x43 00 0x8 0x6 0x6 0x7 0x6 0x5 0x8 0x7 0x7 0x7 0x9 0x9 0x8 0xa 0xc 0x14 0xd ...>
33550 (0x830e) DOUBLE (12) 3<1.60124e-06 1.60124e-06 0>
33922 (0x8482) DOUBLE (12) 6<0 0 0 -71.1663 42.375 0>
34735 (0x87af) SHORT (3) 32<1 1 0 7 1024 0 1 2 1025 0 1 1 2048 0 1 4326 2049 34737 7 0 2054 0 1 9102 ...>
34736 (0x87b0) DOUBLE (12) 2<298.257 6.37814e+06>
34737 (0x87b1) ASCII (2) 8<WGS 84|\0>

Directory 1: offset 4618 (0x120a) next 0 (0)
SubFileType (254) LONG (4) 1<4>
ImageWidth (256) SHORT (3) 1<11419>
ImageLength (257) SHORT (3) 1<8467>
BitsPerSample (258) SHORT (3) 1<1>
Compression (259) SHORT (3) 1<32946>
Photometric (262) SHORT (3) 1<4>
StripOffsets (273) LONG (4) 530<60082 60333 156606 156785 252850 252946 301544 350000 398134 446064 494232 543076 592308 640792 731708 731795 826744 826831 922472 922567 1017858 1017955 1114368 1114462 ...>
SamplesPerPixel (277) SHORT (3) 1<1>
RowsPerStrip (278) SHORT (3) 1<16>
StripByteCounts (279) LONG (4) 530<251 406 179 87 96 88 87 87 96 87 87 91 87 87 87 94 87 88 95 86 97 87 94 86 ...>
PlanarConfig (284) SHORT (3) 1<1>
SampleFormat (339) SHORT (3) 1<1>

[crschmidt]

Further investigation has determined that:

1. "A reader application can use the mask to determine which parts of the image to display. Main image pixels that correspond to 1-bits in the transparency mask are imaged to the screen or printer, but main image pixels that correspond to 0-bits in the mask are not displayed or printed." -- ​http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf
2. GDAL appears to be correctly writing internal masks as 1 bit/1sample per pixel.

External masks appear to follow RFC15, with 255 as a default value; assuming that the internal mask for TIFF is translating 0 to 0 and non-0 to 1 (which I expect it is, but don't have an easy ability to check) this appears to be doing the correct thing.

[crschmidt]

For the record, winkey confirmed on IRC that the file above seems to fail similarly in his qgis on Linux.

[crschmidt]

When opening the image in question in Photoshop, the following error occurs:

22:21:37 < darkblue_B> PS "Could not complete your request becuse of an invalid

SOS,DHT,DQT or EOI JPEG marker is found before JPEG SOI marker"

It's possible that there is some sort of JPEG corruption which is causing this behavior in qgis.

[Even Rouault]

Before throwing files into other apps, it is always good to check if GDAL can read it... and it cannot. gdalinfo -checksum on the file leads to the following error message :

Band 1 Block=11419x16 Type=Byte, ColorInterp=Red
ERROR 1: JPEGLib:Not a JPEG file: starts with 0xff 0xd9
ERROR 1: TIFFReadEncodedStrip() failed.
ERROR 1: IReadBlock failed at X offset 0, Y offset 47
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 47
ERROR 3: Checksum value couldn't be computed due to I/O read error.

Looking at the output of tiffdump, I see : "StripOffsets (273) LONG (4) 530<29385040...>" This means that the first JPEG strip is located right at the end of the file. It might be strange/wrong at the beginning, but in fact while reviewing the code, it is not surprising when looking at how gdalmask/nearblack works. It does a first pass from top to bottom and a second pass from bottom to top, so strips are written twice and the last write is the write of the first strip at the end of the second pass.

There's probably a bug in geotiff.cpp/libtiff when updating in place the JPEG compressed main IFD I think, perhaps due to the constant switching between reading/writing the main JPEG compressed IFD and the DEFLATE compressed mask.

--> Likely workaround, and advisable way to process even if there was not that bug : use gdalmask without any compression, and then use gdal_translate -co COMPRESS=JPEG (it will copy the mask if it exists) on the result of gdalmask. This will likely result in a correct file that is much much smaller !

Update in place of libtiff compressed files leads to rewriting already written strips, and in that case libtiff put them at the end of the file when the new strip size is larger than the previous size. And the place of the previous strip is lost forever...

[crschmidt]

EvenR: Thanks! I realize now that I wasted a bunch of time yesterday by not noticing this minor detail, so I appreciate you pointing it out to me :) One more question; it appears that by default, gdal_translate writes out the .msk externally rather than inside the file, so doing the last gdal_translate step, the mask sits outside the file. Is this a creation option I'm failing to see?

[Even Rouault]

I didn't manage to obtain the same file as Chris when following his instructions. I guess there's a missing x 2 resizing somewhere and a missing -co PHOTOMETRIC=YCBCR in the final gdalmask. So here's what I've tried :

download http://gis2.amherstma.gov/massgis/sid_tiles_30cm/22829020.zip
gdal_translate 22829020.sid 22829020.tif -outsize 200% 200%
gdalmask -internal -nearblack -near 3 -nb 5 22829020.tif 22829020_after_gdalmask.tif
gdalwarp -co TILED=YES -dstalpha -t_srs EPSG:4326 22829020_after_gdalmask.tif 22829020_after_warp.tif
gdalmask -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR -nearblack -near 0 -internal 22829020_after_warp.tif buggy.tif

The last gdalmask finishes prematurely with the following error :

0...10...20...30...40...50.ERROR 1: JPEGLib:Not a JPEG file: starts with 0xd0 0x
c9
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: JPEGLib:Not a JPEG file: starts with 0xd0 0xc9
ERROR 1: TIFFReadEncodedTile() failed.
ERROR 1: An error occured while writing a dirty block

When enabling CPL_DEBUG, I see a lot of "GTiff: directory moved during flush in FlushDirectory()" messages during the process.

If I remove -internal, I don't get the error (and just one "directory moved" warning at the end of the process).

If I remove the GDAL_CACHEMAX to another value, this will also hide the error. When set to something huge like 1 GB, this results in a small file due to all I/O defered to the final FlushCache(). When set to 30 MB, no "JPEGLib error" but a lot of "directory moved" warnings.

So the error seems to be really related to the constant switching between writing the main imagery and the mask.

I did instead try as I suggested in the previous post :

gdalmask -co TILED=YES -nearblack -near 0 -internal 22829020_after_warp.tif 22829020_after_warp_and_mask.tif
gdal_translate -co TILED=YES -co COMPRESS=JPEG -co PHOTOMETRIC=YCBCR 22829020_after_warp_and_mask.tif final.tif --config GDAL_TIFF_INTERNAL_MASK YES

Note the configuration option GDAL_TIFF_INTERNAL_MASK=YES (that appears to be documented in the "Internal nodata masks" section of ​http://gdal.org/frmt_gtiff.html ;-))

I had a error message "Error fetching directory count" coming from TIFFRewriteDirectory() when writing the internal mask, without any apparent consequence in the file. But I found a fix commited in r20953 that solves that last error. (The "JPEGLib:Not a JPEG file" still remains when using gdalmask -co COMPRESS=JPEG)

[crschmidt]

Regarding the file being different: The difference that I did was to add -co PHOTOMETRIC=YCBCR, and to remove -co TILED=YES. I'm sorry I didn't document this better; after I created the ticket, I went to upload the file and realized it would take 30 minutes, so I switched to PHOTOMETRIC; doing so in tiled mode caused the "JPEGLib:Not a JPEG file: starts with 0xd0 0x" that you also saw, so I had to then drop TILED.

I'm assuming that this is one of those cases where doing these in separate stages (warp/mask/compress) is going to result in a smaller file, and I should just accept it. :) I was actually going to create a seperate ticket for the JPEG error, but it seems like that is really the only underlying issue in the weird behavior in Qgis. (uDig still reads the mask backwards, but that's a Java problem, not a GDAL problem, now that I've confirmed the spec.)

Thanks for pointing out GDAL_TIFF_INTERNAL_MASK; I was reading RFC 15 (http://trac.osgeo.org/gdal/wiki/rfc15_nodatabitmask) because I knew I'd seen it somewhere, but didn't find it there. (Too many documentations for me to manage in my head.) One further question (to clutter up this already confused ticket): the file with an internal mask seems much larger, and it looks like that tiff directory is not compressed:

TIFF Directory at offset 0x3c34 (15412)
  Subfile Type: transparency mask (4 = 0x4)
  Image Width: 11419 Image Length: 8467
  Tile Width: 256 Tile Length: 256
  Bits/Sample: 1
  Sample Format: unsigned integer
  Compression Scheme: None
  Photometric Interpretation: transparency mask
  Samples/Pixel: 1
  Planar Configuration: single image plane

This time I've actually read ​http://gdal.org/frmt_gtiff.html and don't see anything obvious there to cause that internal mask to be deflate compressed. THe external mask (.msk), however, is deflate compressed. Additionally, the initial file (produced with gdalmask's -internal) is "Compression (259) SHORT (3) 1<32946>" (different from the external's "Compression (259) SHORT (3) 1<8>"), so I fear I'm probably doing something wrong to cause this :) If this is a better question better answered elsewhere, feel free to say so :)

[Even Rouault]

As far as the internal mask is concerned, Frank changed internal mask to be compressed with nCompress = COMPRESSION_DEFLATE (method 32946) in r20820 (previously was uncompressed). I can't explain the tiffdump output you have.

However it turns that when specifying COMPRESS=DEFLATE as a creation option (like it is done for external overviews), this is turned into nCompress = COMPRESSION_ADOBE_DEFLATE (method 8) internally. Hence the difference you saw.

I've changed the internal mask to use COMPRESSION_ADOBE_DEFLATE now (r20954), but it shouldn't have any impact (as both methods AFAIK are equivalent). Perhaps you should retry now.

[Even Rouault]

Additional fix in r20956 : GTiff: CreateMaskBand?() didn't inherit properly the bIsTiled from the main IFD due to change done in r20953

[crschmidt]

Ah! I knew Frank had made that change recently, but failed to realize that I was running on my system gdal_translate instead of SVN, which explains the problem. Sorry for the noise, just a silly error on my part. Thanks again!

[Even Rouault]

Closing as I think there's no longer any issue.