GML read failure for GDAL 1.6+

[mdwheel]

We're trying to rev to use GDAL 1.6 and all seemed well until we ran into issues with GML point data.

We've boiled down a test set to as small as possible while still failing...it's attached.

Here's the error we're getting:

ogr2ogr -f "CSV" /tmp/test.csv /tmp/points.gml

* glibc detected * free(): invalid next size (fast): 0x080b55f0 *

Aborted

We saw this error with Xerces 2.7 and decided to try with Xerces 3.0.1, but it still gives the same error. We're building with GCC 4.1.1. We tried the 2009-04-14 stable branch and it had the same failure.

Also note that we can use ogr2ogr for other input file formats without error, and we can write to GML output format.

This worked with our build of GDAL 1.4.2 and XercesC 2.7.

[mdwheel]

small version of file which still exhibits the failure.

[warmerdam]

Chaitanya,

I'd appreciate your looking into this as soon as practical. If you don't get the glibc error, running it under valgrind might also turn up the problem.

[chaitanya]

The problem occurs when a character with a unicode value 256 (0x100) is encountered in http://trac.osgeo.org/gdal/browser/branches/1.6/gdal/ogr/ogrsf_frmts/gml/trstring.cpp by tr_strcpy( char *pszCString, const XMLCh *panXMLString ) at line177. There was no problem with a few other values greater than 256.

[warmerdam]

Chaitanya,

It was my understanding that the UTF-8 encoding does not include any zero bytes. This is pretty critical for non-utf8 aware manipulation of UTF-8 strings in OGR.

Is there really a legitimate character 0x0100 in unicode? What is it's utf-8 representation?

[chaitanya]

Fixed in trunk (r16785) and 1.6 (r16786).

[warmerdam]

Reviewing the change, it appears the problem was that a buffer overrun would occur if a UTF16 string was processed that included a multiple of 256 as a character. This resulted in allocation of a too-short working buffer. The change was introduced in 1.6 when proper recoding of UTF16 strings was implemented.

Thanks for the fix!

[Even Rouault]

Test added in r16792

[mdwheel]

We built with GDAL 1.6.1 and this bug persists in the 64 bit build but is fixed in the 32 bit build. We can reproduce with the same 1 point gml file and ogr2ogr

[chaitanya]

Replying to mdwheel:

We built with GDAL 1.6.1 and this bug persists in the 64 bit build but is fixed in the 32 bit build. We can reproduce with the same 1 point gml file and ogr2ogr

mdwheel,

Could you give more details on the platform and versions? I couldn't reproduce the bug in 64bit Ubuntu 8.10.

[Even Rouault]

Closing due to lack of feedback from reporter. Please reopen if you can reproduce with latest GDAL and give more details.

[mdwheel]

Hi, This simple example (see file attached above worked in 1.4.2 but not in 1.6.0 or 1.6.1. The error has changed due to a fix above, but an error still persists. See below for example of running it in 1.4.2 and 1.6.1 (using file named points-1.gml which is the same as "points.gml" attached above. thanks, Mark Wheeler

[root@entfq26 ~]# /opt/google/bin/gdalwarp --version GDAL 1.4.2.0, released 2007/06/27 [root@entfq26 ~]# /opt/google/bin/ogr2ogr --debug on -f "CSV" ~/points.csv ~/points-1.gml OGR: OGROpen(/root/points-1.gml/0x98d5cc0) succeeded as GML. [root@entfq26 ~]# ls ~/points.csv planet_osm_point.csv [root@entfq26 ~]# cat ~/points.csv/planet_osm_point.csv name,osm_id,pid,place ,150916687,853,hamlet [root@entfq26 ~]#

fusionqa0:~> /opt/google/bin/ogr2ogr --version GDAL 1.6.1, released 2009/05/11 fusionqa0:~> fusionqa0:~> /opt/google/bin/ogr2ogr --debug on -f "CSV" ~/points.csv ~/points-1.gml OGR: OGROpen(/home/jagadev/points-1.gml/0x408f90) succeeded as GML. ERROR 1: <coordinates> element missing value. fusionqa0:~> ls ~/points.csv planet_osm_point.csv fusionqa0:~> cat ~/points.csv/planet_osm_point.csv fusionqa0:~>

[chaitanya]

Replying to mdwheel:

Actually, the file points-1.gml didn't work in 1.4.2 too.

The 1.4.2 branch misread the name string as zero length string, which is not the desired action. This is because of the simple character translation in 1.4 branch as opposed to the use of recoding API in 1.6 branch

The 1.6.1 branch gave the error message "<coordinates> element missing value." This was fixed in #3156 a few days ago. The 1.6.1 branch probably recognized the name value but the error made the process stop without writing the feature into the csv file.

The 1.6.1 branch you have should work with xerces2 but not with xerces3. To work with xerces3 you need to get the latest working copy of the 1.6 or the trunk.