SXF Encoding error

[alexey]

I use your software GDAL\OGR (ver. 1.11.0). In particular, I use ogr2ogr utility for converting vector geographic information data to a different formats. In a different open sources there are geo-data files which contain a diacritic marks (for example, Estonia data contains umlaut modification: Ä, Ö, Õ and so on). Export of this data by ogr2ogr utility either causes an error or replacement diacritic marks with latin marks.

I have tested an open Estonia map EE.sxf (​http://gistoolkit.ru/download/osm/ee.sxf.zip) with classifier OSM.rsc (​http://gistoolkit.ru/download/classifiers/osm.zip), which was exported to PostgreSql database, Shape(Esri).

Command: ogr2ogr -f PostgreSQL PG:"dbname='SXFData' host='localhost' port='5432' user='postgres' password='1'" d:\EE.sxf

Error: INSERT command for new feature failed. Invalid byte sequence for encoding “UTF8”: 0xd5 0x20

[Jukka Rahkonen]

Are you sure that your SXF file is valid? Driver manual page says that the format is using UTF8 ​http://www.gdal.org/drv_sxf.html but the error seems to mean that something else it tried to insert into PostGIS.

GDAL issue trac is meant for handling real bugs. You could have sent first a mail into gdal-dev mailing list. The developers read it but also many users and some of them may have experience on the SXF format.

-Jukka Rahkonen-

[Even Rouault]

Jukka, actually the driver recodes from various encodings to UTF-8 which is the pivot encoding in OGR. CC'ing Dmitry

[bishop]

Fix in r27686

[alexey]

Replying to bishop:

Fix in r27686

If you backup this lines will be work fine. 986 unsigned nLen = (unsigned(stAttInfo.nScale) + 1) * 2;

987 if(nLen < 2

nLen > nSemanticsSize

nSemanticsSize - nLen < offset )

value[nLen-2] = 0; -not need back for this 986 unsigned nLen = unsigned(stAttInfo.nScale) + 1;

987 if( nLen > nSemanticsSize

nSemanticsSize - nLen < offset )

If you want real test download data from link in start topic. P.S. Apologies for the rediscovery

[bishop]

Replying to alexey:

Replying to bishop:

Fix in r27686

If you backup this lines will be work fine. 986 unsigned nLen = (unsigned(stAttInfo.nScale) + 1) * 2;

987 if(nLen < 2

nLen > nSemanticsSize

nSemanticsSize - nLen < offset )

value[nLen-2] = 0; -not need back for this 986 unsigned nLen = unsigned(stAttInfo.nScale) + 1;

987 if( nLen > nSemanticsSize

nSemanticsSize - nLen < offset )

If you want real test download data from link in start topic. P.S. Apologies for the rediscovery

I don't understand what you write. Can you provide a patch for a trunk? Also I have tested the changes on the link you have provided - everything works well on my PC.

[alexey]

Sorry Bishop, I did not described the problem. After you fix r27686 I am testing the night build on Windows 7 (release-1500-gdal-mapserver and release-1500-x64-gdal-mapserver) gdal. I have no encoding errors, but have some other error: ERROR 1: Invalid index : -1 (Mass spam error >1000) On the next step I downloaded source for stable release gdal 1.11.0, changed file (ogrsxflayer.cpp) as in your fix r27686. And still got the same - ERROR 1: Invalid index : -1. Then I replaced next lines in your fix: 986 unsigned nLen=(unsigned(stAttInfo.nScale)+1)*2

987 if(nLen < 2

nLen > nSemanticsSize

nSemanticsSize-nLen < offset )

with this 986 unsigned nLen=unsigned(stAttInfo.nScale)+1

987 if(nLen > nSemanticsSize

nSemanticsSize-nLen < offset )

Build gdal and got no errors. Can you check this.

[bishop]

Alexey, unfortunately your solution back us to the origin problem of disappearing features with words included such chars as Ä, Ö, Õ.

I found the root of the problem of "ERROR 1: Invalid index : -1" in another place. Please test it. Fix in r27920