Out of memory error in NITFLoadAttributeSection

[rprinceley]

Attached NITF triggers an OOM error (nASSSize is -43) in function NITFLoadAttributeSection():

    if (nNextOffset > 0 && nNextOffset - nASSOffset > nASSSize)
        nASSSize = nNextOffset - nASSOffset;

    /* Be sure that the attribute subsection is large enough to hold the */
    /* offset table (otherwise NITFFetchAttribute coud read out of the buffer) */
    if (nASSSize < 8 * nAttrCount)
    {
        CPLError( CE_Warning, CPLE_AppDefined,
                  "Attribute subsection not large enough (%d bytes) to contain %d attributes.",
                  nASSSize, nAttrCount );
        return;
    }

/* -------------------------------------------------------------------- */
/*      Load the attribute table.                                       */
/* -------------------------------------------------------------------- */
    pabyAttributeSubsection = (GByte *) VSIMalloc(nASSSize);
    if( pabyAttributeSubsection == NULL )

[warmerdam]

The problem is the segment size adjustment done by:

    /* OK, now, as often with RPF/CADRG, here is the necessary dirty hack */
    /* -- Begin of lengthy explanation -- */
    /* A lot of CADRG files have a nASSSize value that reports a size */
    /* smaller than the genuine size of the attribute subsection in the */
    /* file, so if we trust the nASSSize value, we'll reject existing */
    /* attributes. This is for example the case for */
    /* http://download.osgeo.org/gdal/data/nitf/0000M033.GN3 */
    /* where nASSSize is reported to be 302 bytes for 52 attributes (which */
    /* is odd since 52 * 8 < 302), but a binary inspection of the attribute */
    /* subsection shows that the actual size is 608 bytes, which is also confirmed*/
    /* by the fact that the next subsection (quite often LID_ExplicitArealCoverageTable but not always) */
    /* begins right after. So if this next subsection is found and that the */
    /* difference in offset is larger than the original nASSSize, use it. */
    /* I have observed that nowhere in the NITF driver we make use of the .nLocSize field */
    /* -- End of lengthy explanation -- */

    if (nNextOffset > 0 && nNextOffset - nASSOffset > nASSSize)
        nASSSize = nNextOffset - nASSOffset;

The nNextOffset is actually less than nASSOffset so nASSize is adjusted to what would be a negative value, but wraps to a large positive value since nASSSize is 32bit unsigned. I will try to come up with some improved logic around the "next segment offset" detection for this hacky workaround.

[warmerdam]

Fixed in trunk (r22075), 1.8 (r22076) and 1.8-esri (r22077).

The problem does not appear in 1.7.x.