GDAL JPEG compressed NITF image is not compatible with other NITF viewer

[thchiou]

When I used GDAL to save a compressed NITF image, I was not able to get other NITF viewer to open resulting image.

The image I tested is a 3 band image and I set compression type as “C3” (JPEG compression). I tried to reopen the compressed image with ELT4000 (commercial NITF tool), ESRI ArcMap, and Opticks (open source image analysis tool) and none of the tools would display the compressed image. And all three tools are able to open the none-compressed NITF file just fine.

I also tried to compress an NITF image with ELT4000. A ELT compressed image can be open by ELT4000, GDAL, and ESRI ArcMap. This leads me to believe there are some incompatibility issue with GDAL generated compressed NITF file.

I don’t know if the following information is relevant, but I ran CIVA, the official NITF validation tool, against the GDAL compressed NITF file. And below is an error the validation identified:

The combination of Image Representation(IREP) = "YCbCr601" and Image Compression(IC) = "C3" does not allow Image Mode(IMODE) = "B".” I checked the NITF specification(2500C) and those combination is not listed in the the complexity level table in the Appendix A (page 127).

I would appreciate if any one can shed some light one this issue. If you need additional information feel free to contact me: kevin.chiou@…

I will attach following files:

1. An uncompressed NITF file
2. A GDAL compressed NITF file
3. An ELT4000 compressed NITF file

[thchiou]

An uncompressed NITF file

[thchiou]

GDAL compressed NITF file

[thchiou]

ELT compressed NITF file

[Even Rouault]

Kevin,

that's almost a pleasure to have such a precise bug report ;-) Yes, I think you pointed out the issue. MIL-STD-2500C, page 129, indicates that IMODE must be set to P for JPEG 24bit Color YCbCr601. GDAL currently always writes IMODE=B. I'm attaching a variation of bike_gdal_compressed.ntf, bike_gdal_compressed_manual_edited.ntf, where I've just changed with an hex editor the right byte to set IMODE=P. Could you check that's enough to make other NITF readers happy ? If so, I'll fix the code.

[Even Rouault]

Variation of bike_gdal_compressed.ntf where IMODE was set to P

[thchiou]

Hi, Even. I appreciate the quick response and the compliment :)

But sorry, no, that doesn’t fix the problem. Still cannot open the image with other image viewers. Although the official validation tool, CIVA, think the file is valid now.

I also tried to compress a 1 band image, so it has IREP=”Mono”, IC=”C3” and IMODE=”B.” Other viewers are not able to open a single band GDAL compressed NITF image either.

I also tried to compress a JPEG image with GDAL and open it with other viewer and that works fine, so it’s not a pure JPEG compression issue.

Again thanks for looking into the issue. Let me know if anything else I can help with.

[Even Rouault]

Hum, this get more complicated now. By examining the JPEG stream from bike-d-c-2-elt-compressed.ntf, I see the presence of a particular JPEG segment, the NITF application data segment, that is required by MIL-STD-188-198 (page 59), which is required according the spec. So, I took again my favorite hexadecimal heditor and pasted the APP6 segment from bike-d-c-2-elt-compressed.ntf into the GDAL compressed one. Could you check if bike_gdal_compressed_manual_edited.ntf works with other viewers ?

[Even Rouault]

I meant bike_gdal_compressed_manual_edited_app6.ntf

[Even Rouault]

Revised version with APP6 segment

[thchiou]

Yup, that works on both ArcMap and ELT4000. And I guess Opticks doesn't support compressed NITF file in general.

[Even Rouault]

Ok, I've commited in GDAL trunk (r18617) code to emit the required APP6 application data marker and set appropriate IMODE value. If you're in position to build GDAL from source, I'd appreciate your testing.

Otherwise, you could check the files I've generated with the following command lines and which are enclosed in the attached bike_after_r18617.zip :

gdal_translate -of NITF -co IC=C3 bike-d-c-2.ntf bike_c3_after_r18617.ntf
gdal_translate -of NITF -co IC=M3 bike-d-c-2.ntf bike_m3_after_r18617.ntf
gdal_translate -of NITF -co IC=C3 -co BLOCKSIZE=256 bike-d-c-2.ntf bike_c3_block256_after_r18617.ntf
gdal_translate -of NITF -co IC=M3 -co BLOCKSIZE=256 bike-d-c-2.ntf bike_m3_block256_after_r18617.ntf
gdal_translate -b 1 -of NITF -co IC=C3 bike-d-c-2.ntf bike_mono_c3_after_r18617.ntf
gdal_translate -b 1 -of NITF -co IC=M3 bike-d-c-2.ntf bike_mono_m3_after_r18617.ntf
gdal_translate -b 1 -of NITF -co IC=C3 -co BLOCKSIZE=256 bike-d-c-2.ntf bike_mono_c3_block256_after_r18617.ntf
gdal_translate -b 1 -of NITF -co IC=M3 -co BLOCKSIZE=256 bike-d-c-2.ntf bike_mono_m3_block256_after_r18617.ntf

[Even Rouault]

Archive with 8 NITF JPEG compressed files generated by GDAL after r18617

[thchiou]

Thanks for putting together a fix in such short amount of time. Here is the result of viewing the images with different NITF viewers. There are some good news, but mostly bad ones :(

File/Tool	ArcMap	OpenEV (GDAL)	ELT4000
bike_c3_after_r18617	Yes	Yes	No (3)
bike_c3_block256_after_r18617	Partial (1)	Yes	No (3)
bike_m3_after_r18617	No (2)	Yes	No (3)
bike_m3_block256_after_r18617	No (2)	Yes	No (3)
bike_mono_c3_after_r18617	Yes	Yes	No (3)
bike_mono_c3_block256_after_r18617	Partial (1)	Yes	No (3)
bike_mono_m3_after_r18617	No	Yes	No (3)
bike_mono_m3_block256_after_r18617	No	Yes	No (3)

Notes:

(1) ArcMap only displays upper left corner of the image (probably first 256 row and columns) for block256 images

(2) ArcMap is unable to render M3 compression. Image opens but is blank (could be an ArcMap compression support issue)

(3) ELT4000 refuses to open any of the images with the error "Error: File Read"

All images passed CIVA validation.
Also, I am building GDAL from source. I am running some additional testing and will let you know if I notice anything.

[Even Rouault]

There was a stupid typo now fixed by r18662 which hopefully will enable ELT4000 to read (part of) the new regenerated image set I'm attaching (bike_after_r18662.zip). I guess there's no improvement to expect from ArcMap.

Could you also tell me how the following images display (to check conformance of hopefully reference images) :

​http://www.gwg.nga.mil/ntb/baseline/software/testfile/Nitfv2_1/ns3304a.nsf ​http://www.gwg.nga.mil/ntb/baseline/software/testfile/Nitfv2_1/ns3301j.nsf

and the attached output files generated by :

gdal_translate -of NITF ../autotest/gdrivers/tmp/cache/ns3304a.nsf ns3304a_gdal_mono_c3_512_after_r18662.ntf -co IC=C3 -co BLOCKSIZE=512
gdal_translate -of NITF ../autotest/gdrivers/tmp/cache/ns3301j.nsf ns3301j_gdal_mono_m3_256_after_r18662.ntf -co IC=M3 -co BLOCKSIZE=256

[Even Rouault]

Archive with 8 NITF JPEG compressed files generated by GDAL after r18662

[thchiou]

Good news, ELT4K can now read most image except for C3 block images. Below is the updated result.

File/Tool with r18662	ArcMap	OpenEV (GDAL)	ELT4000
bike_c3_after_r18662	Yes	Yes	Yes
bike_c3_block256_after_r18662	Partial (1)	Yes	No (5)
bike_m3_after_r18662	No (2)	Yes	Yes
bike_m3_block256_after_r18662	No (2)	Yes	Yes
bike_mono_c3_after_r18662	Yes	Yes	Yes
bike_mono_c3_block256_after_r18662	Partial (1)	Yes	No (5)
bike_mono_m3_after_r18662	No	Yes	Yes
bike_mono_m3_block256_after_r18662	No	Yes	Yes
ns3304a.nsf	No (3)	Yes	Yes
ns3304a_gdal_mono_c3_512_after_r18662.ntf	No (3)	Yes	No (6)
ns3301j.nsf	No (4)	Yes	Yes
ns3301j_gdal_mono_m3_256_after_r18662.ntf	No (4)	Yes	Yes

Notes:

(1) ArcMap only displays upper left corner of the image (probably first 256 row and columns) for block256 images
(2) ArcMap is unable to render M3 compression. Image opens but is blank (could be an ArcMap compression support issue)
(3) ArcMap is not able to display either of ns3304a.nsf images. It shows 4 black sqauare at four corners and a white background.
(4) ArcMap is not able to display either of ns3301j.nsf. It shows a black background with some tiny horizontal lines.
(5) ELT4000 refuses to open the images with the error "Error: File Read"
(6) ELT4000 read error. "ns3304a_gdal_mono_c3_512_after_r18662.ntf could not be loaded. return code was 1. Contact ..."

All images passed CIVA validation.

Since ELT4K is a NITF specific tool, resolving image errors with ELT4K is a lot more important then figure out why ESRI ArcMap cannot handle some of the images.
Also, I am fine with the current state of the fix since I don't use compression with blocks. But if you need me to run additional tests on the images, I am more than happy to do so.

[Even Rouault]

OK, that's good news. The array shows that ArcMap has some limitations since he cannot read ns3304a.nsf and ns3301j.nsf

I'm going to give a last try for the C3 block images, but it's a very blind guess. By reading the MIL-STD-188-198 spec a bit more and comparing NGA images with their GDAL "clones", I see that the presence of "restart intervals" is required in the JPEG NITF profile, but not in baseline JPEG as currently used by GDAL. I'm not clear why ELT4000 would need them to be present in C3 block images and not in other cases. As I'm not familiar with this notion, I'm attaching mcu.zip (from a small patch not commited) that contains 3 series of images (bike_c3_block256, bike_mono_c3_block256 and ns3304a_gdal_mono_1024x512_c3_512), each one in 5 versions (no mcu -> as you got previously, with mcu=16,32,64 or 128). If my understanding of the spec is right, I'd expect that ELT4000 succeeds with mcu16, mcu32 and mcu64 for ns3304a_gdal_mono_1024x512_c3_512 (and fails for mcu128 and nomcu); and succeeds with mcu16 and mcu32 for bike_c3_block256 and bike_mono_c3_block256 (and fails for mcu64, mcu128 and nomcu). Could you give it a try ?

[Even Rouault]

3 x 5 images, multiblock, IC=C3, with different values of nRestartInterval

[thchiou]

Guess what...ELT4000 can open all of them except the nomcu ones.

[Even Rouault]

Ok, I've pushed a new fix in r18664 that autosets a value of restart interval to the maximum value allowed by the spec (that apparently ELT4000 does not strictly check). Here's (hopefully) a final set of images if you want to check : after_r18664.zip

It includes the 8 'classical' variants of bike (mono/RGB, C3/M3, single block/multi block), as well as a 50x50 RGB one in 6 variants (C3/M3, single block, block size =25, block size =16) to test a bit for unusual situations (non-power-of-2 image size, non-power-of-2 block size, or block size not multiple of image size).

[Even Rouault]

Set of 14 JPEG compressed NITF after r18664 + 1 uncompressed NITF

[thchiou]

Nice job. All of them work with ELT4000 and OpenEV. Again thank you for making the fix. And I think NITF compression feature is pretty solid now.

Also, just to give you a heads up, I have being making some minor updates to my local copy of NITF driver. Most of the them are minor tweaks to make the official validation tool happy. I am looking to submit them in coming weeks.

[Even Rouault]

Ok, I've backported the whole set of changes in 1.7 branch (r18667).