JP2KAK YCbCr problems with band selection through DirectRasterIO()

[Coatman]

Summary

1. When QGIS depends upon a version of GDAL that is older than about 10 months, then QGIS properly displays JPEG2000 images thru Kakadu and thru Jasper

2. When QGIS depends upon a version of GDAL that is newer than about 10 months, JP2 images are properly georeferenced, and displayed with the proper number of columns and rows, but the colors are altered. Thru Kakadu, the colors appear to be run through a color Look Up Table (LUT). Thru Jasper, the colors can be all black.

Example images are here ​http://homepage.mac.com/gregcoats/jp2.html

Since 2 May 2008, there have been 42 comments posted about this issue at ​http://trac.osgeo.org/qgis/ticket/996

In comparing the code in GDAL 1.6.0 to 1.5.0, I find that the Kakadu based frmts/jp2kak/jp2kakdataset.cpp in 1.6.0 has different code than 1.5.0 after the comment in 1.6.0 on line 312 "Figure out the color interpretation for this band." I find that the Jasper based frmts/jpeg2000/jpeg2000dataset.cpp in 1.6.0 has different code than 1.5.0 around the comment in 1.6.0 on line 738 "JPEG2000 driver ignores color table. Consider using color table expansion (-expand option in gdal_translate)".

Any help would be greatly appreciated.

[Coatman]

The original report was posted at May 2, 2008 3:46:51 PM EDT on ​http://trac.osgeo.org/qgis/ticket/996 and specified that we were using Kakadu version 6.0, the current version of Kakadu at that time. As of 8 December 2008, the current version of Kakadu is 6.1.1, which has several improvements and no known reported bugs, so I would recommend working with Kakadu 6.1.1, but a solution with either version of Kakadu would be most welcome. Thank you.

[warmerdam]

This ticket does not explain how to reproduce a problem with GDAL utilities.

Also, please reference one specific jpeg2000 file demonstrating a problem, and ideally a reference tiff or png demonstrating what the image should look like.

[Coatman]

More detailed information, and --most importantly-- small JP2 images created specifically for testing with this ticket are available as a free download from ​http://homepage.mac.com/gregcoats/jp2.html Success is properly displaying these JP2 test images. As a reference, Kakadu provides a free viewer for JP2 images, called kdu_show for Windows and kdu_macshow for Mac OS X. Jasper provides a free viewer for JP2 images for Windows, Mac OS X, and Linux called jiv. Failure is improperly displaying these test images, especially displaying these JP2 images with altered colors, and in particular with the Kakadu JP2 plugin gdal_JP2KAK.dylib changing the colors of these JP2 test images by running them through a color Look Up Table (LUT).

[Coatman]

< This ticket does not explain how to reproduce a problem with GDAL utilities. >

The problem can be easily reproduced with gdal_translate, and a small, sample GeoTIF, 800 columns by 100 rows, that can be downloaded from ​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo.tif

Using the GDAL Jasper-based JPEG2000, and without using Kakadu, I used GDAL 1.6 gdal_translate to create from a GeoTIF a GeoJP2, and then to create from that GeoJP2 a GeoTIF. QGIS version 1.0.0-Kore-preview2, for Mac OS X 10.5, available from the Quantum GIS Downloads web site, and from the kyngchaos web site, both display the GeoTIF created by gdal_translate properly geo-referenced, with the proper number of columns and rows, but with the colors being ALL BLACK. Additionally, all of the other image viewing applications that I have display the GeoTIF created by gdal_translate as all black.

To summarize, I am reporting that a _GeoTIF_, created by gdal_translate from a GeoJP2 that gdal_translate created, is being displayed as all black, by QGIS and by 5 other image viewing applications.

The sample GeoTIF, 800 columns by 100 rows, that gdal_translate converts to GeoJP2, can be downloaded from ​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo.tif

gdal_translate --version

GDAL 1.6.0, released 2008/12/04

gdal_translate --formats | grep 2000

JP2KAK (rw): JPEG-2000 (based on Kakadu)

JPEG2000 (rw): JPEG-2000 part 1 (ISO/IEC 15444-1)

gdal_translate -ot Byte -of JPEG2000 -co "FORMAT=JP2" -co "mode=int" rgbwcmyk01_YeGeo.tif rgbwcmyk01_YeGeo_GDAL16_JPEG2000.jp2

Input file size is 800, 100 0...10...20...30...40...50...60...70...80...90...100 - done.

export GDAL_SKIP=JP2KAK

gdal_translate -ot Byte -of GTiff -co "COMPRESS=NONE" rgbwcmyk01_YeGeo_GDAL16_JPEG2000.jp2 rgbwcmyk01_YeGeo_GDAL16_JPEG2000.tif

Input file size is 800, 100 0...10...20...30...40...50...60...70...80...90...100 - done.

[Coatman]

A further, byte by byte comparison of the original uncompressed, 800 columns by 100 rows, GeoTIF, that can be downloaded from ​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo.tif , to the uncompressed .tif created by gdal_translate using Jasper, from a .jp2 created by gdal_translate from the original .tif, shows that the basic tiffinfo and listgeo metadata is identical (this is good). But od shows that this 800 column by 100 row by RGB image has 800 * 100 = 80,000 consecutive 00 00 00 pixel triples, representing black, that causes image viewing applications to display the .tif created by gdal_translate as all black.

[Even Rouault]

What you report here is already known as a Jasper issue : #2399

[warmerdam]

I was hoping this ticket could focus on the JP2KAK issue(s). Suggestions on how to reproduce that with gdal_translate are welcome.

[Coatman]

Using QGIS with the Kakadu based JP2KAK plugin, based on Kakadu 6.1.1, to display JP2 images, I observe that JP2 images are displayed properly georeferenced, and with the proper number of columns and rows and

1 JP2 images created from .tif images with Kakadu kdu_compress are displayed by QGIS with a color shift

2 JP2 images created from .tif images with geojasper are displayed by QGIS with a color shift

3 JP2 images created from .tif images with gdal_translate -of JP2KAK are properly displayed! That is good news, but I wonder why? Does gdal_translate -of JP2KAK write some extra information, that it later depends upon to properly display the image, that Kakadu and geojasper do not write?

Sample images and the color shift can be seen at ​http://homepage.mac.com/gregcoats/jp2.html

1.6/gdal/frmts/jp2kak/jp2kakdataset.cpp

[warmerdam]

I have determined that the new optimized DirectRasterIO() code does not properly handle YCbCr images such as ​http://homepage.mac.com/gregcoats/jp2/images/18stj940125_reduce4_reversible.jp2 .

I am digging into how to address this.

[warmerdam]

Hmm, I'm back to having problems reproducing this. It works fine for me in my dev build on 64bit linux. But the version built into the last FWTools OpenEV for Linux does exhibit the problem. This is nearly a year old and I'm not sure what version of Kakadu I was using when I built it. My local builds use Kakadu 6.0.

What version of Kakadu was used in the qgis-linux builds that exhibited the problem?

[Coatman]

Starting with the GeoTIF file ​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo.tif that consists of 8 adjoining 100x100 squares of pure red, green, blue, white, cyan, magenta, yellow, and black, I created three GeoJP2 files using Kakadu version 6.1.1, Geojasper version 1.3.2, and JP2KAK version GDAL 1.6.0 released 2008/12/04. These images are available at

​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo_JP2KAK.jp2

​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo_kakadu.jp2

​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo_geojasper.jp2

Kakadu kdu_show properly displays all three JP2 images. Geojasper geojiv properly displays all three JP2 images. QGIS using GDAL JP2KAK properly displays only the image it created, and displays the Kakadu created and Geojasper created JP2 images properly georeferenced, with the proper number of columns and rows, but with a color shift, as though the colors are being altered through a color look up table.

Even though all three JP2 files were created using reversible compression, and all three JP2 files contain georeferencing info, ls shows that the JP2KAK file is 1.7 times larger than the Kakadu created and geojasper created .jp2 files.

2751 Jan 5 11:53 rgbwcmyk01_YeGeo_geojasper.jp2

2803 Jan 5 11:46 rgbwcmyk01_YeGeo_kakadu.jp2

4645 Jan 5 11:53 rgbwcmyk01_YeGeo_JP2KAK.jp2

JP2 files are composed of multiple JP2 boxes. When I examined the internal structure of the three JP2 files, I discovered that the JP2KAK created .jp2 contains, after the initial JP2 header box, a JP2 colr box, 1786 bytes long. The Kakadu created JP2, and the geojasper created JP2 do not contain a JP2 colr box. I would assert that an image with only 8 colors, does not need a color look up table. The extra 1786 bytes in the JP2 colr box nicely accounts for the JP2KAK created .jp2 file being larger than the Kakadu created and geojasper created .jp2 files.

So JP2KAK wrote a(n unnecessary) color look up table to the .jp2 file it created, and apparently JP2KAK only properly displays the colors of .jp2 files it creates with a color look up table box.

There is a comment on line 2487 of jp2kakdataset.cpp about "Write a JP2 pseudo color table". With the example image, Kakadu and Geojasper do not write a pseudo color table, so I do not see the need for JP2KAK to write a pseudo color table.

[Coatman]

My initial report ​http://trac.osgeo.org/qgis/ticket/996 noted in its first line "working with QGIS version 0.9.2, GDAL version 1.5.0, and Kakadu version 6.0". Kakadu version 6.1.1 was released on 7 Dec 2008, and I am now using it as the basis for JP2KAK. To be clear, since my initial report, QGIS with GDAL JP2KAK has displayed all of my Kakadu and geojasper created .jp2 files with a color shift, as though the colors are being altered through a color look up table. For reference, Kakadu version 6.0 was released on 15 Aug 2007. The previous Kakadu version was 5.2.6 released on 10 Jan 2007.

[warmerdam]

I have confirmed that the problem only occurs with YCbCr color space images handled through the DirectRasterIO() code path (instead of block oriented IO) and where individual bands are read rather than all the bands at once. The fix is to do the band selection on "output components" rather than "codestream components".

Generally speaking this shows up in QGIS and OpenEV because they pull stuff one band at a time, while gdal_translate generally reads all three bands at once.

The fix is in trunk (r16046), 1.6 branch (r16047) and 1.5 branch (r16048).

If it would be convenient to provide a small (less than 20KB) sample file demonstrating this problem, I'll add it to the test suite.

Please move the above issues about colormaps into a new ticket so this ticket can remain focused on one defect.

[Coatman]

I am pleased to report that, under Mac OS X 10.5, I changed, as directed, line 1562 of 1.6/gdal/frmts/jp2kak/jp2kakdataset.cpp and then built a new gdal_JP2KAK.dylib plugin. Now QGIS, using Kakadu based JP2KAK, properly displays JPEG2000 images. Many thanks to Frank!

[Coatman]

< If it would be convenient to provide a small (less than 20KB) sample file demonstrating this problem, I'll add it to the test suite. >

You are welcome add the 2803 byte GeoJP2 image file, that demonstrated the now fixed problem, available at ​http://homepage.mac.com/gregcoats/jp2/images/rgbwcmyk01_YeGeo_kakadu.jp2

[warmerdam]

I have added jp2kak tests based on this .jp2 file in trunk (r16054).

Thanks for the sample file.