Building overviews on an Imagine-created HFA file results in a segfault

[hobu]

#0  0x00279448 in HFAEntry::MarkDirty () from /usr/local/lib/libgdal.so.1
#1  0x00279694 in HFAEntry::HFAEntry () from /usr/local/lib/libgdal.so.1
#2  0x0027c9be in HFACreateLayer () from /usr/local/lib/libgdal.so.1
#3  0x0027334a in HFABand::CreateOverview () from /usr/local/lib/libgdal.so.1
#4  0x0027cfb7 in HFACreateOverview () from /usr/local/lib/libgdal.so.1
#5  0x00278853 in HFARasterBand::BuildOverviews () from /usr/local/lib/libgdal.so.1
#6  0x00274b13 in HFADataset::IBuildOverviews () from /usr/local/lib/libgdal.so.1
#7  0x0034b465 in GDALDataset::BuildOverviews () from /usr/local/lib/libgdal.so.1

You can grab a 70mb copy of the file here:  http://www.tcwd.org/flipfree10_5.img.gz

gdaladdo -r average flipfree10_5.img 2 4 8 16

If I use gdal_translate to turn it into an HFA file using GDAL, things work as expected.

[warmerdam]

Analysis of a similar problem from SANZ shows initial problems as follows.
I believe the issue is that the RRDNamesList isn't being grown. 

==30412== Invalid write of size 1
==30412==    at 0x1B900989: memset (mac_replace_strmem.c:464)
==30412==    by 0x1CAA04E1: HFAField::SetInstValue(char const*, int, unsigned
char*, unsigned, int, char, void*) (hfafield.cpp:488)
==30412==    by 0x1CAA6746: HFAType::SetInstValue(char const*, unsigned char*,
unsigned, int, char, void*) (hfatype.cpp:284)
==30412==    by 0x1CAA080D: HFAField::SetInstValue(char const*, int, unsigned
char*, unsigned, int, char, void*) (hfafield.cpp:636)
==30412==    by 0x1CAA6746: HFAType::SetInstValue(char const*, unsigned char*,
unsigned, int, char, void*) (hfatype.cpp:284)
==30412==    by 0x1CA9F71C: HFAEntry::SetFieldValue(char const*, char, void*)
(hfaentry.cpp:647)
==30412==    by 0x1CA9F74E: HFAEntry::SetStringField(char const*, char const*)
(hfaentry.cpp:660)
==30412==    by 0x1CA9662E: HFABand::CreateOverview(int) (hfaband.cpp:1681)
==30412==    by 0x1CAA49D8: HFACreateOverview (hfaopen.cpp:1922)
==30412==    by 0x1CA989D1: HFARasterBand::BuildOverviews(char const*, int,
int*, int (*)(double, char const*, void*), void*) (hfadataset.cpp:1109)
==30412==    by 0x1CA9C98C: HFADataset::IBuildOverviews(char const*, int, int*,
int, int*, int (*)(double, char const*, void*), void*) (hfadataset.cpp:2221)
==30412==    by 0x1CBB1418: GDALDataset::BuildOverviews(char const*, int, int*,
int, int*, int (*)(double, char const*, void*), void*) (gdaldataset.cpp:1186)
==30412==    by 0x1CBB146B: GDALBuildOverviews (gdaldataset.cpp:1211)
==30412==    by 0x1C87AE59: ossimGdalWriter::buildGdalOverviews()
(ossimGdalWriter.cpp:1287)
==30412==    by 0x1C874D70: ossimGdalWriter::writeFile() (ossimGdalWriter.cpp:367)
==30412==    by 0x1C002807: ossimImageFileWriter::execute() (in
/usr5/pkg/ossim-cvs/lib/libossim.so.1.6.5)
==30412==  Address 0x1C7A146B is 0 bytes after a block of size 547 alloc'd
==30412==    at 0x1B8FE899: malloc (vg_replace_malloc.c:149)
==30412==    by 0x1CBE2301: VSIMalloc (cpl_vsisimple.cpp:342)
==30412==    by 0x1CBD2A70: CPLRealloc (cpl_conv.cpp:337)
==30412==    by 0x1CA9F01E: HFAEntry::MakeData(int) (hfaentry.cpp:344)
==30412==    by 0x1CA96586: HFABand::CreateOverview(int) (hfaband.cpp:1659)
==30412==    by 0x1CAA49D8: HFACreateOverview (hfaopen.cpp:1922)
==30412==    by 0x1CA989D1: HFARasterBand::BuildOverviews(char const*, int,
int*, int (*)(double, char const*, void*), void*) (hfadataset.cpp:1109)
==30412==    by 0x1CA9C98C: HFADataset::IBuildOverviews(char const*, int, int*,
int, int*, int (*)(double, char const*, void*), void*) (hfadataset.cpp:2221)
==30412==    by 0x1CBB1418: GDALDataset::BuildOverviews(char const*, int, int*,
int, int*, int (*)(double, char const*, void*), void*) (gdaldataset.cpp:1186)
==30412==    by 0x1CBB146B: GDALBuildOverviews (gdaldataset.cpp:1211)
==30412==    by 0x1C87AE59: ossimGdalWriter::buildGdalOverviews()
(ossimGdalWriter.cpp:1287)
==30412==    by 0x1C874D70: ossimGdalWriter::writeFile() (ossimGdalWriter.cpp:367)
==30412==    by 0x1C002807: ossimImageFileWriter::execute() (in
/usr5/pkg/ossim-cvs/lib/libossim.so.1.6.5)
==30412==    by 0x1C171341: ossimIgen::outputProduct() (in
/usr5/pkg/ossim-cvs/lib/libossim.so.1.6.5)
==30412==    by 0x804A791: main (in /usr5/pkg/ossim-cvs/bin/igen)
==30412== 

[warmerdam]

For the SANZ case (adding alot overviews to a GDAL generated file) it
is sufficient to increase the size the RRDNamesList is created with. 

Change:
       poRRDNamesList->MakeData( 23+16+8+ 500 /* hack for growth room*/ );
To:
        poRRDNamesList->MakeData( 23+16+8+ 3000 /* hack for growth room*/ );

The proper solution is for the logic that sets the string field to know how
to grow things.  That might/should also correct the problem with files
not generated by GDAL.

[daniel112b]

Patch for ticket #1109

[daniel112b]

I have made a hack for this issue. It's kind of uglu but it seems to work.

What i did was that if the call to SetFieldValue? in HFAEntry::SetStringField? failes and the name of the entry is "RRDNamesList" i enlarge the entry and try again. In order for this to work i had to do some more minor changes like setting nFilePos to 0 in MakeData? and making sure that all file pointers to the changed entry is updated in SetPosition?.

I have not tested this a lot but it seems to work fine. One issue with this is that it wastes the old space in the file but that should not be alot.

[warmerdam]

This needs to be addressed for GDAL 1.5.0 (if not GDAL 1.4.3).

Setting to 1.4.3 for now and I'll review the proposed patch shortly.

[warmerdam]

Patches r12301, r12302, r12303 incorporate a variation on the provided patch in trunk.

r12300 incorporates a test for the issue into the autotest.

r12304 backports fixes to 1.4 branch.