Opened 9 years ago

Closed 9 years ago

Last modified 9 years ago

#3848 closed defect (fixed)

Unable to load CIB datasets with GDAL

Reported by: morabit Owned by: warmerdam
Priority: highest Milestone: 1.8.0
Component: GDAL_Raster Version: 1.7.2
Severity: blocker Keywords:
Cc:

Description

In our project we are using GDAL 1.7.2 to load map data in various formats as CADRG, DTED, CIB, etc.
All it's working fine but we have a big problem with the CIB format.
For this type of datasets, the GDALOpen function fails with the following error message:
'For <frameFileName>, assert 'geoTransf[GEOTRSFRM_ROTATION_PARAM1] == 0 && geoTransf[GEOTRSFRM_ROTATION_PARAM2] == 0' failed.'
We checked the source code and discovered that this was due to a series of assert failures in file rpftocdataset.cpp, lines 733,740.
We tried to comment out this asserts, but now the same function fails with the following error message:
'Ignoring NITF RPF Location table since it seems to be corrupt.'
We are pretty sure that our datasets are not corrupted, as the same datasets are loaded without any problems by Global Mapper v12.00 and FalconView? 4.3.0.
The gdalinfo tool seems to handle these files with no problems but if we use it with the -mm option the following errors are reported:
......
......
Band 1 Block=256x256 Type=Byte, ColorInterp?=Palette
ERROR 6: File lacks VQ LUTs, unable to decode image
ERROR 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 6: File lacks VQ LUTs, unable to decode imagery
ERROR 1: IReadBlock failed at X offset 1, Y offset 0
......
......
ERROR 1: Failed to compute min/max, no valid pixels found in sampling.

The openev tool, that uses GDAL internally, is able to load them only if we force the use of the OGDI driver.
The datasets come from a military collection with limited access and cannot be shared completely, so we can attach only a truncated NITF frame file. However, it should include all the relevant metadata values since the problem seems to come from there.

Attachments (2)

0AA43516.I42 (258.7 KB) - added by morabit 9 years ago.
nitf frame file
00JGWR16.I22 (962.6 KB) - added by morabit 9 years ago.
Second nitf frame file

Download all attachments as: .zip

Change History (9)

Changed 9 years ago by morabit

Attachment: 0AA43516.I42 added

nitf frame file

comment:1 Changed 9 years ago by morabit

Component: defaultGDAL_Raster
Owner: changed from warmerdam to morabit
Priority: highhighest
Severity: normalblocker

comment:2 Changed 9 years ago by morabit

Owner: changed from morabit to warmerdam

comment:3 Changed 9 years ago by Even Rouault

Milestone: 1.8.0
Resolution: fixed
Status: newclosed

OK, there are several issue that are triggered by that file :

  • One was a real bug in GDAL that didn't read the subframe mask table in all cases. This could also be shown with a file of the NITF 2.0 test suite. Fixed in r21173 and tested in r21174
  • IMHO, the rest are inconsistencies/errors of the attached file and not GDAL bugs. They are workaround in r21175 :

a) The RPFIMG TRE size coded in the file isn't consistent. It is bigger than the advertized total size of the TREs. Instead of erroring out and discarding the whole RPFIMG data, I decided to truncate it to the available remaining size of the TREs. Looks like it work with that file. Hopefully, the code that reads the RPFIMG shouldn't crash if it is smaller than needed.

b) The RPFHDR isn't located at the place indicated by the RPF location table. This has been an issue noticed in the past on a few files (see #1373, #1714). But in the case of that ticket, I only found the error to be the RPFHDR location. So basically I test that the VQ tables are at the indicated offset (and don't use the default logic that tries to look them a few bytes after if not found at the indicated offset). If so, we cross the finger and go on...

Changed 9 years ago by morabit

Attachment: 00JGWR16.I22 added

Second nitf frame file

comment:4 Changed 9 years ago by morabit

Resolution: fixed
Status: closedreopened

Hi rouault,

thank you for the quick response end for the fix.

We drive our requirement tests on our collection of CIB datasets
and now almost all the datasets are loaded with no errors
but some still have problems.
The returned error message is:
<frameFileName>, assert '(entry->vertInterval - (-geoTransf[GEOTRSFRM_NS_RES])) / entry->horizInterval < 0.01' failed

After comment out this assert we get once again this error:
Ignoring NITF RPF Location table since it seems to be corrupt.

Running our application with a version of gdaldev18
compiled in debug mode we find that the workaround call to
NITFLoadVQTables(psImage, FALSE) at nitfimage.c, line 3242 returns false
because the pasLocations created with NITFReadRPFLocationTable
doesn't include the record with LocId?=LID_CompressionLookupSubsection.
These datasets are loaded without any problems by Global Mapper v12.00.
We attach a truncated version of one of the NITF frame files included in these datasets.
Thank you for your attention.

comment:5 Changed 9 years ago by Even Rouault

Resolution: fixed
Status: reopenedclosed

The cause for the error "<frameFileName>, assert '(entry->vertInterval - (-geoTransf[GEOTRSFRM_NS_RES])) / entry->horizInterval < 0.01' failed" is that the RPFIMG is being discarded, and that the information from the CoverageSectionSubHeader? are not read. So in that case the IGEOLO record is only used for geolocation, but as it isn't precise enough, it doesn't match with what is expected from the content of the A.TOC file.

Ok, so in that second file, there's no VQ table since the image file isn't compressed (I somehow though that CIB images were always VQ compressed). I've added a fallback test that check the consistency of the CoverageSectionSubHeader? to deal with that case. Commited in r21181.

I'd encourage you reporting the issue with the data producer. Those images perhaps happen to work in other software packages since they probably don't make the same consistency check as done in GDAL, but I think the images aren't still conformant with the spec (or there's some aspect of the spec we've missed, which is possible as it is awfully complicated...)

comment:6 in reply to:  5 Changed 9 years ago by morabit


Replying to rouault:

The cause for the error "<frameFileName>, assert '(entry->vertInterval - (-geoTransf[GEOTRSFRM_NS_RES])) / entry->horizInterval < 0.01' failed" is that the RPFIMG is being discarded, and that the information from the CoverageSectionSubHeader? are not read. So in that case the IGEOLO record is only used for geolocation, but as it isn't precise enough, it doesn't match with what is expected from the content of the A.TOC file.

Ok, so in that second file, there's no VQ table since the image file isn't compressed (I somehow though that CIB images were always VQ compressed). I've added a fallback test that check the consistency of the CoverageSectionSubHeader? to deal with that case. Commited in r21181.

I'd encourage you reporting the issue with the data producer. Those images perhaps happen to work in other software packages since they probably don't make the same consistency check as done in GDAL, but I think the images aren't still conformant with the spec (or there's some aspect of the spec we've missed, which is possible as it is awfully complicated...)

Thank you very much for your support, now everything's work fine.
We are reporting all those issues to our data producer as you said.
Can you deliver this change in your very next official release, i.e 1.7.4,
instead of milestone version 1.8.0 ?
Our software delivery is due in two weeks by now and we'd like to deliver
the official version of the GDAL libray, if this is possible.
Thank you

Best regards

Bruno M.

comment:7 Changed 9 years ago by Even Rouault

I'm not too keen to backport such ugly workarounds in the 1.7.X branch. And it is unlikely that 1.7.4 will be released before 1.8.0 : it could be released after 1.8.0, or there might be no 1.7.4 at all. The 1.8 developement forked off from the 1.7.0 tag at the beginning of the year, so they can have independant release schedules. If we trust the past dev cycles, 1.8.0 will probably be released at the end of this month or more likely at the beginning of 2011.

Note: See TracTickets for help on using tickets.