Opened 7 years ago

Closed 7 years ago

#5647 closed defect (fixed)

SXF Encoding error

Reported by: alexey Owned by: warmerdam
Priority: normal Milestone:
Component: OGR_SF Version: svn-trunk
Severity: normal Keywords: sxf
Cc: bishop

Description

I use your software GDAL\OGR (ver. 1.11.0). In particular, I use ogr2ogr utility for converting vector geographic information data to a different formats. In a different open sources there are geo-data files which contain a diacritic marks (for example, Estonia data contains umlaut modification: Ä, Ö, Õ and so on). Export of this data by ogr2ogr utility either causes an error or replacement diacritic marks with latin marks.

I have tested an open Estonia map EE.sxf (http://gistoolkit.ru/download/osm/ee.sxf.zip) with classifier OSM.rsc (http://gistoolkit.ru/download/classifiers/osm.zip), which was exported to PostgreSql? database, Shape(Esri).

Command: ogr2ogr -f PostgreSQL PG:"dbname='SXFData' host='localhost' port='5432' user='postgres' password='1'" d:\EE.sxf

Error: INSERT command for new feature failed. Invalid byte sequence for encoding “UTF8”: 0xd5 0x20

Attachments (1)

error.png (97.1 KB) - added by alexey 7 years ago.

Download all attachments as: .zip

Change History (8)

Changed 7 years ago by alexey

Attachment: error.png added

comment:1 Changed 7 years ago by Jukka Rahkonen

Are you sure that your SXF file is valid? Driver manual page says that the format is using UTF8 http://www.gdal.org/drv_sxf.html but the error seems to mean that something else it tried to insert into PostGIS.

GDAL issue trac is meant for handling real bugs. You could have sent first a mail into gdal-dev mailing list. The developers read it but also many users and some of them may have experience on the SXF format.

-Jukka Rahkonen-

comment:2 Changed 7 years ago by Even Rouault

Cc: bishop added
Component: defaultOGR_SF
Keywords: sxf added
Summary: Encoding errorSXF Encoding error

Jukka, actually the driver recodes from various encodings to UTF-8 which is the pivot encoding in OGR. CC'ing Dmitry

comment:3 Changed 7 years ago by bishop

Resolution: fixed
Status: newclosed

Fix in r27686

comment:4 in reply to:  3 ; Changed 7 years ago by alexey

Resolution: fixed
Status: closedreopened

Replying to bishop:

Fix in r27686

If you backup this lines will be work fine. 986 unsigned nLen = (unsigned(stAttInfo.nScale) + 1) * 2;

987 if(nLen < 2
nLen > nSemanticsSize nSemanticsSize - nLen < offset )

value[nLen-2] = 0; -not need back for this 986 unsigned nLen = unsigned(stAttInfo.nScale) + 1;

987 if( nLen > nSemanticsSize
nSemanticsSize - nLen < offset )

If you want real test download data from link in start topic. P.S. Apologies for the rediscovery

comment:5 in reply to:  4 Changed 7 years ago by bishop

Resolution: worksforme
Status: reopenedclosed

Replying to alexey:

Replying to bishop:

Fix in r27686

If you backup this lines will be work fine. 986 unsigned nLen = (unsigned(stAttInfo.nScale) + 1) * 2;

987 if(nLen < 2
nLen > nSemanticsSize nSemanticsSize - nLen < offset )

value[nLen-2] = 0; -not need back for this 986 unsigned nLen = unsigned(stAttInfo.nScale) + 1;

987 if( nLen > nSemanticsSize
nSemanticsSize - nLen < offset )

If you want real test download data from link in start topic. P.S. Apologies for the rediscovery

I don't understand what you write. Can you provide a patch for a trunk? Also I have tested the changes on the link you have provided - everything works well on my PC.

comment:6 Changed 7 years ago by alexey

Resolution: worksforme
Status: closedreopened

Sorry Bishop, I did not described the problem. After you fix r27686 I am testing the night build on Windows 7 (release-1500-gdal-mapserver and release-1500-x64-gdal-mapserver) gdal. I have no encoding errors, but have some other error: ERROR 1: Invalid index : -1 (Mass spam error >1000) On the next step I downloaded source for stable release gdal 1.11.0, changed file (ogrsxflayer.cpp) as in your fix r27686. And still got the same - ERROR 1: Invalid index : -1. Then I replaced next lines in your fix: 986 unsigned nLen=(unsigned(stAttInfo.nScale)+1)*2

987 if(nLen < 2
nLen > nSemanticsSize nSemanticsSize-nLen < offset )

with this 986 unsigned nLen=unsigned(stAttInfo.nScale)+1

987 if(nLen > nSemanticsSize
nSemanticsSize-nLen < offset )

Build gdal and got no errors. Can you check this.

comment:7 Changed 7 years ago by bishop

Resolution: fixed
Status: reopenedclosed
Version: 1.11.0svn-trunk

Alexey, unfortunately your solution back us to the origin problem of disappearing features with words included such chars as Ä, Ö, Õ.

I found the root of the problem of "ERROR 1: Invalid index : -1" in another place. Please test it. Fix in r27920

Note: See TracTickets for help on using tickets.