Opened 15 years ago

Closed 15 years ago

#2964 closed defect (fixed)

11 bit JPEG2000 compressed NITIF shows as 8 bit

Reported by: gaopeng Owned by: warmerdam
Priority: normal Milestone: 1.6.1
Component: GDAL_Raster Version: 1.6.0
Severity: normal Keywords: NITF JP2KAK 11bit
Cc: ysiddiqui

Description

The following test image is a single band 11-bit NITF with JPEG2000 compression. It shows an 8-bit image.

ftp://GDAL@ftp.esri.com/04AUG08WV011300008AUG04182920-P1BS-052034510010_01_P001.ntf

Change History (12)

comment:1 by warmerdam, 15 years ago

Cc: ysiddiqui added
Status: newassigned

Gao,

Can you determine which underlying JPEG2000 implementation is being used? (JP2KAK, JP2MrSID, or JP2ECW)? Often there are clues in the debug output if you do "gdalinfo --debug on your.ntf".

I have added Yusuf to the ticket as he mentioned the same issue with a similar file yesterday.

comment:2 by gaopeng, 15 years ago

The underlying JPEG2000 implementation is JP2KAK. The oCodeStream.get_bit_depth(nBand-1) returns 11. The image is from Digital Globe.

Here is the output from gdalinfo --debug on

C:\Program Files\FWTools2.2.8>gdalinfo --debug on
meeks2\Data\NITF\NCDS\SanDie go\04AUG08WV011300008AUG04182920-P1BS-052034510010_01_P001.ntf ECW: Got mutex. ECW: FileInfo: SizeXY=35840,31744 Bands=1

OriginXY=0,0 CellIncrementXY=1,1

GDAL: GDALOpen(J2K_SUBFILE:3884,513142054,\\meeks2\Data\NITF\NCDS\SanDiego\04AUG 08WV011300008AUG04182920-P1BS-052034510010_01_P001.ntf, this=003FEE30) succeeds as JP2ECW (pid=3312, responsiblePID=3312). Warning 1: Image with 11 bits per sample will not be interpreted properly. GDAL: GDALOpen(
meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV011300008AUG04182920-P1 BS-052034510010_01_P001.ntf, this=003FE4C8) succeeds as NITF (pid=3312, responsi blePID=3312). Driver: NITF/National Imagery Transmission Format Files:
meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV011300008AUG04182920-P1BS-05203 4510010_01_P001.ntf


meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV011300008AUG04182920-P1BS-05203

4510010_01_P001.ntf.aux.xml Size is 35840, 31744 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0,

AUTHORITY["EPSG","8901"]],

UNIT["degree",0.0174532925199433,

AUTHORITY["EPSG","9108"]],

AXIS["Lat",NORTH], AXIS["Long",EAST], AUTHORITY["EPSG","4326"]]

GeoTransform =

-117.277711507, 6.351708722030561e-006, -4.375417851839874e-009 32.78930789977986, 2.557735055851492e-007, -4.944222173216818e-006

Metadata:

NITF_FHDR=NITF02.10 NITF_CLEVEL=06 NITF_STYPE=BF01 NITF_OSTAID=DG NITF_FDT=20080804201528 NITF_FTITLE=WV01_SENSOR:2008-08-04T18:29:20.481958Z NITF_FSCLAS=U NITF_FSCLSY=US NITF_FSCODE= NITF_FSCTLH= NITF_FSREL= NITF_USE00A_ANGLE_TO_NORTH=270 NITF_USE00A_MEAN_GSD=022.7 NITF_USE00A_DYNAMIC_RANGE=02047 NITF_USE00A_OBL_ANG=23.59 NITF_STDIDC_ACQUISITION_DATE=20080804182920 NITF_STDIDC_MISSION=WV01 NITF_STDIDC_PASS=13 NITF_STDIDC_OP_NUM=000 NITF_STDIDC_START_SEGMENT=AA NITF_STDIDC_REPRO_NUM=00 NITF_STDIDC_REPLAY_REGEN=000 NITF_STDIDC_START_COLUMN=001 NITF_STDIDC_START_ROW=00001 NITF_STDIDC_END_SEGMENT=AA NITF_STDIDC_END_COLUMN=035 NITF_STDIDC_END_ROW=00031 NITF_STDIDC_COUNTRY=US NITF_STDIDC_WAC= NITF_STDIDC_LOCATION=3248N11710W NITF_USE00A_ROLL_ANG=-22.99 NITF_USE00A_N_REF=00 NITF_USE00A_REV_NUM=04889 NITF_USE00A_N_SEG=001 NITF_USE00A_MAX_LP_SEG=031680 NITF_USE00A_SUN_EL=+65.3 NITF_USE00A_SUN_AZ=123.8 NITF_FSDCTP= NITF_FSDCDT= NITF_FSDCXM= NITF_FSDG= NITF_FSDGDT= NITF_FSCLTX= NITF_FSCATP= NITF_FSCAUT= NITF_FSCRSN= NITF_FSSRDT= NITF_FSCTLN= NITF_FSCOP=00000 NITF_FSCPYS=00000 NITF_ENCRYP=0 NITF_FBKGC=126,126,126 NITF_ONAME=DigitalGlobe NITF_OPHONE=+1(800)496-1225 NITF_IID1=P1031AA100 NITF_IDATIM=20080804182920 NITF_TGTID= NITF_IID2=04AUG08WV011300008AUG04182920-P1BS-052034510010_01_P001 NITF_ISCLAS=U NITF_ISCLSY=US NITF_ISCODE= NITF_ISCTLH= NITF_ISREL= NITF_ISDCTP= NITF_ISDCDT= NITF_ISDCXM= NITF_ISDG= NITF_ISDGDT= NITF_ISCLTX= NITF_ISCATP= NITF_ISCAUT= NITF_ISCRSN= NITF_ISSRDT= NITF_ISCTLN= NITF_ISORCE=DigitalGlobe NITF_PVTYPE=INT NITF_IREP=MONO NITF_ICAT=VIS NITF_ABPP=11 NITF_PJUST=R NITF_IMAGE_COMMENTS=The imagery and metadata data on this media is the propert

y of DigitalGlobe and is licensed for use only. All use must be

in accordance with the terms of the license that accompanies

the media. If the license is purchased under contract NMA 301 99 D003 , use is in accordance with the license therein

NITF_IC=C8 NITF_IMODE=B

Image Structure Metadata:

COMPRESSION=JPEG2000

Subdatasets:

SUBDATASET_1_NAME=NITF_IM:0:\\meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV01130000

8AUG04182920-P1BS-052034510010_01_P001.ntf

SUBDATASET_1_DESC=Image 1 of
meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV0113000

08AUG04182920-P1BS-052034510010_01_P001.ntf

SUBDATASET_2_NAME=NITF_IM:1:\\meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV01130000

8AUG04182920-P1BS-052034510010_01_P001.ntf

SUBDATASET_2_DESC=Image 2 of
meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV0113000

08AUG04182920-P1BS-052034510010_01_P001.ntf RPC Metadata:

LINE_OFF=15839 LINE_SCALE=15931 SAMP_OFF=17576 SAMP_SCALE=17974 LONG_OFF=-117.1639 LONG_SCALE=0.1139 LAT_OFF=32.7155 LAT_SCALE=0.0829 HEIGHT_OFF=70 HEIGHT_SCALE=501 LINE_NUM_COEFF=-0.001038033 0.05698506 -1.051104 0.005695194 0.0004562016 -4.0

19947e-006 1.020762e-005 -0.0007883984 0.001150701 -4.949970000000001e-007 -3.64 9007e-008 2.055249e-006 5.303055e-006 2.076904e-006 -2.908066e-005 -0.000144348 -3.841197e-005 1.81126e-007 1.218707e-006 2.199924e-007

LINE_DEN_COEFF=1 0.0005796357 0.0002159086 1.408437e-005 1.214994e-005 -5.5964

56e-008 7.366359e-007 2.698178e-005 -9.008356e-005 3.65334e-005 -5.847226e-007 1 .249256e-007 8.333085e-005 4.440523e-008 -5.171251e-006 -0.0004764925 -4.685569e -008 0 4.965944e-006 0

SAMP_NUM_COEFF=-0.009553556 0.982324 0.0007050203 -0.02127772 -0.002046597 0.0

002744625 -0.0001695849 0.009100619000000001 0.0003254278 -7.200068e-006 -3.0572 92e-006 6.481815e-005 -1.91651e-005 -6.906865e-006 2.286248e-005 0.0001812801 -6 .54437e-008 6.154844e-006 -1.866099e-006 1.546958e-007

SAMP_DEN_COEFF=1 0.0004517674 0.00201399 -0.000473017 -8.128729999999999e-006

-4.180401e-006 1.165732e-006 1.025112e-005 1.601425e-005 -7.263943e-006 0 -4.257 468e-007 -3.923566e-007 5.664512e-008 -5.250244000000001e-007 1.741876e-006 -1.9 09208e-008 -1.068307e-007 -3.177835e-008 1.531909e-008

MIN_LONG=-117.22085 MAX_LONG=-117.10695 MIN_LAT=32.67405 MAX_LAT=32.75695

OGRCT: Source: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs OGRCT: Target: +proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs Corner Coordinates: Upper Left (-117.2777115, 32.7893079) (117d16'39.76"W, 32d47'21.51"N) Lower Left (-117.2778504, 32.6323585) (117d16'40.26"W, 32d37'56.49"N) Upper Right (-117.0500663, 32.7984748) (117d 3'0.24"W, 32d47'54.51"N) Lower Right (-117.0502052, 32.6415254) (117d 3'0.74"W, 32d38'29.49"N) Center (-117.1639583, 32.7154167) (117d 9'50.25"W, 32d42'55.50"N) Band 1 Block=35840x1 Type=UInt16, ColorInterp=Gray

Overviews: arbitrary

GDAL: GDALClose(J2K_SUBFILE:3884,513142054,\\meeks2\Data\NITF\NCDS\SanDiego\04AU G08WV011300008AUG04182920-P1BS-052034510010_01_P001.ntf, this=003FEE30) (pid=331 2, responsiblePID=3312) GDAL: GDALClose(
meeks2\Data\NITF\NCDS\SanDiego\04AUG08WV011300008AUG04182920-P 1BS-052034510010_01_P001.ntf, this=003FE4C8) (pid=3312, responsiblePID=3312) GDAL: GDALDeregister_GTiff() called.

comment:3 by warmerdam, 15 years ago

Gao,

I notice:

GDAL: GDALOpen(J2K_SUBFILE:3884,513142054,\\meeks2\Data\NITF\NCDS\SanDiego\04AUG
08WV011300008AUG04182920-P1BS-052034510010_01_P001.ntf, this=003FEE30) succeeds as JP2ECW

So the file is opened with the JP2ECW driver, not the JP2KAK driver. In the band report I see

Band 1 Block=35840x1 Type=UInt16

Which indicates the image is being treated as 16bit. How did you establish that it was being read as 8bit? I tried a similar NITF from Yusuf, and it seemed the image had an effective dynamic range of 9 bits - the highest pixel values I found were in the area of 500 or so.

comment:4 by gaopeng, 15 years ago

The gdalinfo I used is from FWTools2.2.8, not from my build of GDAL.

I stepped into NITF driver open, and then JP2KAK open. I tried to fix it by setting UInt16 in JP2KAKRasterBand::JP2KAKRasterBand, but it didn't work. The output pixel values are scaled from 11 bit to 16.

comment:5 by warmerdam, 15 years ago

Keywords: JP2KAK 11bit added

I have confirmed that the JP2KAK driver when operating on these 11bit jpeg2000 datastreams will treat the result as eight bit. The CodeStream.get_bit_depth() is actually returning 11 for the bands of this data. Some additional work will be required...

comment:6 by warmerdam, 15 years ago

Resolution: fixed
Status: assignedclosed

I have confirmed that the JP2KAK driver was treating this dataset as 8bit. I have added logic to treat 9 to 15 bit data as GDT_UInt16 or GDT_Int16 and added logic in transfer_bytes() to do this properly.

The change has been applied in trunk (r16871) and 1.6 branch (r16872).

comment:7 by gaopeng, 15 years ago

I tested, and it doesn't seem to work. The pixel values are scaled to UIng16 value range instead of staying within 11 bit range. This causes problem since NITF returns ABP as 11 bit, and ArcGIS uses it to stretch.

Also, the newly added code path is not executed. It went to the following instead:

if (src.get_buf32() != NULL) { Decompressed samples have a 32-bit representation (integer or float)

assert(precision >= 8); Else would have used 16 bit representation kdu_sample32 *sp = src.get_buf32(); if (!src.is_absolute() && eOutType != GDT_Byte ) { Transferring normalized floating point data.

float scale16 = (float)(1<<16); int val;

for (; width > 0; width--, sp++, dest+=gap) {

if( eOutType == GDT_Int16 ) {

val = (int) (sp->fval*scale16); *((GInt16 *) dest) = (GInt16) MAX(MIN(val,32767),-32768);

} else if( eOutType == GDT_UInt16 ) {

HERE >>>>> val = (int) (sp->fval*scale16) + 32768; *((GUInt16 *) dest) = (GUInt16) MAX(MIN(val,65535),0);

} else if( eOutType == GDT_Float32 )

*((float *) dest) = sp->fval;

}

}

comment:8 by warmerdam, 15 years ago

Resolution: fixed
Status: closedreopened

Gao,

It would appear your file is different than Yusuf's but unfortunately your file is 500MB which strains my download limits so I'm hesitant to fetch it.

Would it be convenient to inspect the result result from comp.get_bit_depth(true) circa the following line in ProcessTile()?

    bool use_shorts = (comp.get_bit_depth(true) <= 16);

I'm wondering if it actually returns 11 as Yusuf's file did. Also we need to confirm you actually go through ProcessTile() as opposed to ProcessYCbCrTile().

comment:9 by gaopeng, 15 years ago

It did go through ProcessTile(). The use_shorts is false.

The line above returns 11:

int bit_depth = comp.get_bit_depth();

comment:10 by warmerdam, 15 years ago

Gao,

Could you try changing the code you reference above to look like this and see if it gives desirable results? It should scale things to the indicated precision, instead of stretching to full 16bit range. If this works ok, I may need to apply this a bit more widely. I am a bit nervous about possible side effects.

        assert(precision >= 8); // Else would have used 16 bit representation
        kdu_sample32 *sp = src.get_buf32();
        if (!src.is_absolute() && eOutType != GDT_Byte )
        { // Transferring normalized floating point data, but preserving 
          // precision.
            float scale16 = (float)(1<<precision);
            float offset = (1<<(precision-1));
            int val;

            for (; width > 0; width--, sp++, dest+=gap)
            {
                if( eOutType == GDT_Int16 )
                {
                    val = (int) (sp->fval*scale16);
                    *((GInt16 *) dest) = (GInt16) MAX(MIN(val,32767),-32768);
                }
                else if( eOutType == GDT_UInt16 )
                {
                    val = (int) (sp->fval*scale16) + offset;
                    *((GUInt16 *) dest) = (GUInt16) MAX(MIN(val,65535),0);
                }
                else if( eOutType == GDT_Float32 )
                    *((float *) dest) = sp->fval;
            }
        }

comment:11 by gaopeng, 15 years ago

It works.

comment:12 by warmerdam, 15 years ago

Resolution: fixed
Status: reopenedclosed

I have applied the above change in trunk (r16877) and 1.6 branch (r16878). I also added a modest test to the test suite (r16876) and added broader NBITS support on creation and in the IMAGE_STRUCTURE metadata (trunk only). There was also a fix in 1.6 and trunk related to writing lossless unsigned 16bit files (in r16877/r16878).

I think this slays this issue...

Note: See TracTickets for help on using tickets.