Doesn't use external overviews when reading NITF with JPEG2000 compression

[gaopeng]

Both BlockIO and direct RasterIO do not use external overviews when reading NITF images with JPEG2000 compression.

To reproduce

- Build overviews on a NITF image with jp2 compression. - Follow the code path for BlockIO and RasterIO

Can provide a test image (but very large) if needed.

[warmerdam]

I have confirmed that any access that ends up in JP2KAKDataset::DirectRasterIO() will ignore external overviews - instead letting the Kakadu SDK satisfy the request from the jpeg2000 data stream. As far as I can see this is unrelated to whether the data is in an NITF file, or a standalone JPEG2000 .jp2 file. So I'm treating this as a JP2KAK issue.

[warmerdam]

I have added logic to JP2KAK::TestUseBlockIO() to avoid directio if there are appropriate external overviews available in trunk (r18281) and 1.6-esri (r18283).

Let me know if problems persist.

I have not ported these changes into the 1.6 general branch as I feel external overview support for jpeg2000 is a fairly specialized feature.

[gaopeng]

The test nitf has two subdatasets. When opening the first subdataset, it goes into the "if" branch, and it can't find the external overview file.

/* -------------------------------------------------------------------- */ /* Check for overviews. */ /* -------------------------------------------------------------------- */

if( !EQUAL(poOpenInfo->pszFilename,pszFilename) )

poDS->oOvManager.Initialize( poDS, ":::VIRTUAL:::" );

else

poDS->oOvManager.Initialize( poDS, pszFilename );

[warmerdam]

While investigating a found some situations with non-jpeg2000 datastreams in the nitf file that buildoverviews can get into a recursive infinite loop and crash. Fixed in 1.6-esri (r18299) and trunk (r18300).

I do not have any multi-image jpeg2000 encoded nitf files. However I have encountered a problem with multi-image jpeg compressed nitf files with finding overviews once they are built. I will investigate this angle in case it is the same problem.

[warmerdam]

Small fix to jpeg handling in trunk only (r18301). I jpeg and jpeg2000 compressed nitf files are likely running into the same problem.

Briefly, when subdatasets are not in play, the jpeg2000 dataset's ovmanager can actually find the abc.ntf.ovr file because it knows it's physical name is abc.ntf. But in the jpeg2000 in a subdataset case we need access to the NITF file's .aux.xml data to get the overview filename in the jpeg2000 driver. Uggg.

I haven't worked out how to make this work yet...

I'll continue tomorrow.

[warmerdam]

In trunk (r18312) I have made some changes so that OVERVIEW_FILE metadata on the NITFDataset (in .aux.xml) file is propagated to JPEG and JPEG2000 datasets.

I also modified IBuildOverviews() to clear jpeg2000 level overviews (with a dummy IBuildOverviews() on poJ2KDataset) and to propagate on overview_file metadata after generic overviews are built at the NITF level to the jpeg/jpeg2000 datasets.

I also avoided setting the physical name for gdalpamdataset/nitfdataset unless subdatasets are in use.

All this is aimed at ensuring external overviews for nitf files are handled at the nitfdataset level primarily so that information about them can be properly saved to the .aux.xml referencing the proper subdataset.

With this change external overviews on nitf files with or without internal jpeg or jpeg2000 compression and with or without subdatasets should work.

I don't actually have a jpeg2000 nitf file with multiple images though, so I only sort-of tested this case using multi-jpeg image nitf files.

Also, additional (vaguely related) filelist related improvements (r18305).

[warmerdam]

Patch also applied in 1.6-esri branch (r18313).

Gao, could you test this with your file?

[gaopeng]

It still can't find the newly created ovr. The problem seems to be that the stored ovr path at the time of overview creation is a local path, something like:

E:\Temp\Kabul_uncompressed\other\08MAY18184148-P1BS-005765570010_02_P002.NTF_0.ovr

When the image is accessed using a UNC path on a network, e.g.

\\rberger\temp\Kabul_uncompressed\other\08MAY18184148-P1BS-005765570010_02_P002.NTF

the ovr path E:\Temp is not a valid path.

[warmerdam]

Gao,

Would you mind creating a new ticket for this? It sounds like the issue you are describing is that the OVERVIEW_FILE value saved out to be relative to the file associated with the base dataset rather than a (potentially) absolute path.

I assume this could arise with any of the overview-on-subdataset situations including uncompressed multi-image nitf, netcdf, hdf4, etc.

[gaopeng]

I manually modified the OVERVIEW_FILE path. But it still can't find the overview.

I uploaded a sample nitf, nitf_ovr.zip, up to  ftp://GDAL@ftp.esri.com.

[warmerdam]

The path issue has been handled via #3287. I believe all issues specific to this ticket are corrected.

[warmerdam]

Gao,

I have tested with the contents of nitf_ovr.zip and it seemed to work ok with the changes from 3287 incorporated. I deleted the .aux.xml and .ovr file and ran:

gdaladdo NITF_IM:0:08MAY18184148-P1BS-005765570010_02_P002.NTF 2 4 8 16 32 64

then ran the script:

import gdal

print gdal.VersionInfo('')
ds = gdal.Open('NITF_IM:0:08MAY18184148-P1BS-005765570010_02_P002.NTF')

ds.ReadRaster(0,0,35000,33000,400,400)

ds = None

The .aux.xml used :::BASE::: for the filename, and when the script ran I confirmed that the debug messages indicated there was no direct .jp2 imagery access meaning everything was done via the tiff file.

Before this I also manually edited the original .aux.xml and removed the path, inserting :::BASE::: and it also worked fine. However, it should be noted that the file has only small overviews so it was necessary to make a read request with dramatic decimation before the overviews kick in.

As far as I can establish everything is working fine.

[gaopeng]

Frank,

It didn't work for me because my .ovr is read-only. The following statement in gdaldefaultoverviews.cpp tried to open the ovr for update, and it failed.

            poODS = (GDALDataset *) GDALOpen(osOvrFilename,GA_Update);

Also, the overviews were built for 7 levels, but only 2 levels were actually built in the .ovr. The original embedded jp2 image has 5 levels, which is accounted as the part of 7 levels.

[warmerdam]

I have corrected the access mode in trunk (r18333) and 1.6-esri (r18334).

I would note that a new ticket (#3294) has been setup to address the requirement to mix internal and external overviews.