gdalwarp from 1 bit min-is-black tiff misses most of image in vertical stripes

[kyngchaos]

Another one (trying to project these geotiffs) - I encountered a couple 1 bit geotiffs where the tiff "photometric interpretation" is min-is-black. GDAL sees this as:

  Image Structure Metadata:
    NBITS=1
  Color Table (RGB with 2 entries)
    0: 0,0,0,255
    1: 255,255,255,255

When projected, only every 8th vertical line is projected. All else comes out black. It may be coincidence, but the tiff rows-per-strip is 8. Or it just may be related to the 8bit nature of GDAL's view of a 1bit image.

Occurs with both 1.5.3 and 1.6.0. The #2759 fix didn't help.

[Even Rouault]

I've generated a all white 1 bit GeoTIFF with rows-per-strip=8 but couldn't reproduce any problem when warping it.

Can you provide a sample and the gdalwarp commandline you've used ?

[kyngchaos]

Is the image you created min-is-black? Most software will create 1 bit as min-is-white.

I tried windowing out a small area with gdal_translate (and converting the whole image) and got the same problem. The image is proprietary. If I crop out a bit with Photoshop, it'll change it to min-is-white (and drop the projection).

tiffcp will keep min-is-black, though drop the projection. That could work using -s_srs option, but I'm also having problems with the projection (see email to list).

I can't verify that it will happen with a 1bit colormap image because of the projection problem. The images I'm using are true bitmap images, not 1bit colormapped. I'll attach a bitmap image I created with min-is-black, but no projection info, and rows per strip = 102 (that was the default for the software I used).

[kyngchaos]

bitmap image with min-is-black

[Even Rouault]

Yes, it was min-is-black. I'm attaching my sample 1bit.tif that has the following characteristics, so you can try it :

Driver: GTiff/GeoTIFF
Files: 1bit.tif
Size is 1000, 1000
Coordinate System is:
PROJCS["WGS 84 / UTM zone 31N",
    GEOGCS["WGS 84",
        DATUM["WGS_1984",
            SPHEROID["WGS 84",6378137,298.2572235629972,
                AUTHORITY["EPSG","7030"]],
            AUTHORITY["EPSG","6326"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4326"]],
    PROJECTION["Transverse_Mercator"],
    PARAMETER["latitude_of_origin",0],
    PARAMETER["central_meridian",3],
    PARAMETER["scale_factor",0.9996],
    PARAMETER["false_easting",500000],
    PARAMETER["false_northing",0],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","32631"]]
Origin = (500000.000000000000000,5000000.000000000000000)
Pixel Size = (1.000000000000000,-1.000000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  (  500000.000, 5000000.000) (  3d 0'0.00"E, 45d 9'12.52"N)
Lower Left  (  500000.000, 4999000.000) (  3d 0'0.00"E, 45d 8'40.11"N)
Upper Right (  501000.000, 5000000.000) (  3d 0'45.80"E, 45d 9'12.52"N)
Lower Right (  501000.000, 4999000.000) (  3d 0'45.79"E, 45d 8'40.11"N)
Center      (  500500.000, 4999500.000) (  3d 0'22.90"E, 45d 8'56.31"N)
Band 1 Block=1000x8 Type=Byte, ColorInterp=Palette
  Image Structure Metadata:
    NBITS=1
  Color Table (RGB with 2 entries)
    0: 0,0,0,255
    1: 255,255,255,255


$ tiffdump 1bit.tif
1bit.tif:
Magic: 0x4949 <little-endian> Version: 0x2a
Directory 0: offset 127760 (0x1f310) next 0 (0)
ImageWidth (256) SHORT (3) 1<1000>
ImageLength (257) SHORT (3) 1<1000>
BitsPerSample (258) SHORT (3) 1<1>
Compression (259) SHORT (3) 1<1>
Photometric (262) SHORT (3) 1<1>
StripOffsets (273) LONG (4) 125<1384 2384 3384 4384 5384 6384 7384 8384 9384 10384 11384 12384 13384 14384 15384 16384 17384 18384 19384 20384 21384 22384 23384 24384 ...>
SamplesPerPixel (277) SHORT (3) 1<1>
RowsPerStrip (278) SHORT (3) 1<8>
StripByteCounts (279) LONG (4) 125<1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 ...>
PlanarConfig (284) SHORT (3) 1<1>
SampleFormat (339) SHORT (3) 1<1>
33550 (0x830e) DOUBLE (12) 3<1 1 0>
33922 (0x8482) DOUBLE (12) 6<0 0 0 500000 5e+06 0>
34735 (0x87af) SHORT (3) 32<1 1 0 7 1024 0 1 1 1025 0 1 1 1026 34737 22 0 2049 34737 7 22 2054 0 1 9102 ...>
34737 (0x87b1) ASCII (2) 30<WGS 84 / UTM zone 31N|WG ...>

And I do : gdalwarp -t_srs EPSG:32630 1bit.tif 1bit_warped.tif

Could you also append the result of gdalinfo and tiffdump on your source image ?

[Even Rouault]

Georeferenced 1bit TIFF with min-is-black and row-per-strip=8

[kyngchaos]

gdalwarp worked for me with your image. BUt, try this:

gdal_translate 1bit.tif 1bit1.tif

It gives me a striped copy.

[Even Rouault]

What do you mean by "a striped copy" ? I've tried warping the result of 1bit1.tif and it shows fine too. Maybe it's an issue with your viewer ? I use OpenEV.

Hum, did you try to delete the output file of the gdalwarp and retry ? Maybe an issue with updating a specific file ?

[kyngchaos]

vertical stripes, as in the warp-stripes.tif attachement, though I guess that also appears normal to you...

something like this: O=white, @=black

@@@@@@@O@@@@@@@O
@@@@@@@O@@@@@@@O
@@@@@@@O@@@@@@@O
@@@@@@@O@@@@@@@O

hmmm... interesting... In Photoshop CS (no CS2+ available to check) and ArcGIS 9.2, my warped and copied images, and your images copied-only, appear with stripes. Your image warped appears correct in Photoshop (can't check Arc). In GraphicConverter (a Mac shareware image program), they all appear correct. They are also correct in OSX Preview. My warp-stripes.tif appears striped even in GraphicConverter, maybe because it was cropped and resaved from Photoshop.

In Photoshop, the color index numbers read out strangely (in the 1bit sample):

128, 64, 32, 16, 8, 4, 2, 1
128, 64, 32, 16, 8, 4, 2, 1

instead of all 1's. Of course, the palette only has 1 assigned, to white, so the others are defaulting to 0 (black). So it may be one of those unusual variations (like the band interleaving that GDAL used to default to) that some software (unfortunately big-name) has problems with.

[kyngchaos]

Here's a dump of the copied version of your 1bit.tif:

Magic: 0x4949 <little-endian> Version: 0x2a <ClassicTIFF>
Directory 0: offset 130512 (0x1fdd0) next 0 (0)
ImageWidth (256) SHORT (3) 1<1000>
ImageLength (257) SHORT (3) 1<1000>
BitsPerSample (258) SHORT (3) 1<1>
Compression (259) SHORT (3) 1<1>
Photometric (262) SHORT (3) 1<3>
StripOffsets (273) LONG (4) 16<512 8637 16762 24887 33012 41137 49262 57387 65512 73637 81762 89887 98012 106137 114262 122387>
SamplesPerPixel (277) SHORT (3) 1<1>
RowsPerStrip (278) SHORT (3) 1<65>
StripByteCounts (279) LONG (4) 16<8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125 8125>
PlanarConfig (284) SHORT (3) 1<1>
Colormap (320) SHORT (3) 6<0 65535 0 65535 0 65535>
SampleFormat (339) SHORT (3) 1<1>
33550 (0x830e) DOUBLE (12) 3<1 1 0>
33922 (0x8482) DOUBLE (12) 6<0 0 0 500000 5e+06 0>
34735 (0x87af) SHORT (3) 32<1 1 0 7 1024 0 1 1 1025 0 1 1 1026 34737 22 0 2049 34737 7 22 2054 0 1 9102 ...>
34737 (0x87b1) ASCII (2) 30<WGS 84 / UTM zone 31N|WG ...>

[Even Rouault]

So, if I understand well (please use precise filenames otherwise I get lost), 1bit.tif shows correctly in all viewers, but 1bit1.tif not.

Hum... maybe that those viewers don't support a paletted image (Photometric=3) of 1 bit, which is what GDAL outputs naturally... Which is different from a min-is-black image (Photometric=1) of 1 bit...

In fact, when I produced my 1bit.tif, I produced 1bit1.tif first with some GDAL python scripting and had to do 2 manual steps :

gdal_translate 1bit_photometric3_with_palette.tif 1bit_photometric1_with_palette.tif -co PHOTOMETRIC=MINISBLACK

cp 1bit_photometric1_with_palette.tif 1bit.tif

and then in python: import gdal ds = gdal.Open('1bit.tif', gdal.GA_Update) ds.GetRasterBand(1).SetColorTable(None) ds = None

[kyngchaos]

Hum... maybe that those viewers don't support a paletted image (Photometric=3) of 1 bit,

Could be. With your plain warp command with no creation options, gdalwarp defaults to outputting an 8bit colormapped image with only 1bit used

Bits/Sample: 8

Samples/Pixel: 1

But when I add NBITS=1 to the warp, I again get a striped image in Photoshop.

-co PHOTOMETRIC=MINISBLACK

hmm... so, looks like gdal_translate can create a true bitmap image. But when I copied 1bit.tif and set that to MINISWHITE,

gdal_translate -co "PHOTOMETRIC=MINISWHITE" 1bit.tif 1bitw.tif

it inverted the black/white - that is, it didn't change the values, just the interpretation of them. Is that what that SetColorTable(None) is taking care of - some bogus colortable left in the image?

Warping the 1bit.tif (MINISBLACK) image directly to MINISWHITE didn't work either - I got a 1bit colormap image (or is this what the SetColorTable(None) is taking care of?).

I'm wondering then if there is a bug as reported (seems not). And if not, is there a bug in the non-palette 1bit creation, since you have to strip out the colortable with python.

Is there some way to force a non-palette 1bit image (and set MINISWHITE), especially in gdalwarp? Or flip the interpretation AND the values?

Sorry, lots of questions there.. I'm starting to confuse myself here also. Need to sleep on it.

[Even Rouault]

William,

I'm a bit(well... completely...) lost when trying to review that ticket. Is it still valid ? Sounds a bit like some image viewers might be buggy when dealing with 1bit images.

About your last comment,

1) 'gdal_translate -co "PHOTOMETRIC=MINISWHITE" 1bit.tif 1bitw.tif' is NOT expected to change the pixel values from 0 to 1 or vice versa. It will just set the PHOTOMETRIC TIFF tag.

2) the color table you see reported by GDAL for 1 bit images is a bit "virtual". It is not written in the TIFF at all. When GDAL sees a one bit band, according to the value of the PHOTOMETRIC tag (MINISWHITE or MINISBLACK), it will expose the appropriate color table so that image viewers don't have to be aware of that detail (concept of PHOTOMETRIC tag being MINISWHITE or MINISBLACK) of the TIFF spec.

[kyngchaos]

After so long, I'm a little lost on the state of this also. I have Photoshop CS4 now, and have the same problem with the images, so no improvement there.

As a check, I 'postprocessed' the warped image and got something readable by Photoshop:

gdalwarp -t_srs EPSG:32630 1bit.tif 1bit_warped3.tif
gdal_translate -co PHOTOMETRIC=MINISBLACK -co NBITS=1 1bit_warped3.tif 1bit_warped1.tif

I don't know if your python step to delete the color table is necessary (I'm now at GDAL 1.6.2 now), except maybe to be tidy. tiffinfo still shows it has a color map, though also 1 bit/sample, and Photoshop reports it as 'bitmapped'.

So, I guess you could say this bug is history, a non-bug really. Unless the leftover color table in a 1bit photometric=1 image is a bug.

A couple notes on software:

Photoshop is a major program to be having issues, but Adobe uses a custom TIFF library that just doesn't support some TIFF features (and it's even worse placing TIFFs in Illustrator - more limitations). I don't know if this issue is a bug or missing support.

The others I mentioned, GraphicConverter and OSX Preview, use the system image library on OSX which is libtiff-based for TIFF support, so it's no surprise that they have no problem with the images.

I don't know what Arc uses, or Windows if it uses Windows image APIs.

[Jukka Rahkonen]

It is time to check the status of this issue. Does William have new Photoshop versions to test with?

[kyngchaos]

same problem with CS5. frankly, hard to care now - rare case, haven't had another image like this since reporting.

[Even Rouault]

Let's close that