Opened 5 years ago

Last modified 11 months ago

#2524 reopened defect

wxGUI v.in.ogr wrapper fails to find any layers if file path contains non-latin letters

Reported by: marisn Owned by: grass-dev@…
Priority: normal Milestone: 7.4.5
Component: wxGUI Version: svn-trunk
Keywords: encoding Cc:
CPU: Unspecified Platform: MSWindows Vista

Description

Steps to reproduce:

  • mkdir "šaursliežu dzelzceļš"
  • copy "myshapefile.*" "šaursliežu dzelzceļš/"
  • start GRASS, open vector data import dialog
  • select "C:\balhblah\šaursliežu dzelzceļš\myshapefile.shp" as a data source

Observe how nothing happens. No error messages, no hangs, just nothing. No layers are listed and thus is not possible to continue with data import.

GRASS version: 7.0.0svn GRASS SVN Revision: 63925 Build Date: 2015-01-02 Build Platform: i686-pc-mingw32 GDAL/OGR: 1.11.1 PROJ.4: 4.8.0 GEOS: 3.4.2 SQLite: 3.7.17 Python: 2.7.4 wxPython: 2.8.12.1 Platform: Windows-Vista-6.0.6002-SP2

Change History (17)

comment:1 in reply to:  description Changed 5 years ago by hellik

Replying to marisn:

Steps to reproduce:

  • mkdir "šaursliežu dzelzceļš"
  • copy "myshapefile.*" "šaursliežu dzelzceļš/"
  • start GRASS, open vector data import dialog
  • select "C:\balhblah\šaursliežu dzelzceļš\myshapefile.shp" as a data source

Observe how nothing happens. No error messages, no hangs, just nothing. No layers are listed and thus is not possible to continue with data import.

GRASS version: 7.0.0svn GRASS SVN Revision: 63925 Build Date: 2015-01-02 Build Platform: i686-pc-mingw32 GDAL/OGR: 1.11.1 PROJ.4: 4.8.0 GEOS: 3.4.2 SQLite: 3.7.17 Python: 2.7.4 wxPython: 2.8.12.1 Platform: Windows-Vista-6.0.6002-SP2

Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\gselect.py", line 1920, in OnUpdate

self._reloadLayers()
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\gselect.py", line 1938, in _reloadLayers

input = dsn)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\cor
e\gcmd.py", line 694, in RunCommand

ps = grass.start_command(prog, flags, overwrite, quiet,
verbose, **kwargs)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass
\script\core.py", line 349, in start_command

val = encode(val)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass
\script\utils.py", line 218, in encode

return string.encode(enc)
  File "C:\OSGeo4W\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
24: character maps to <undefined>

comment:2 Changed 5 years ago by annakrat

Keywords: encoding added

The error is the same as in #2205. I have no idea about that one.

I don't get this error on Ubuntu. But when I have non ascii characters in the name of the file (not in the path to it), I got a different error coming from GetValidLayerName? (this function tries to make the name sql compliant). Hopefully fixed in 63943.

comment:3 Changed 4 years ago by martinl

Milestone: 7.0.07.0.5

comment:4 Changed 3 years ago by martinl

Still an issue?

comment:5 Changed 3 years ago by annakrat

Resolution: fixed
Status: newclosed

Tested on Ubuntu and Windows, no problems.

comment:6 Changed 3 years ago by marisn

Milestone: 7.0.57.4.0
Resolution: fixed
Status: closedreopened
Version: svn-releasebranch70svn-trunk

Tested import tool (File -> Import vector -> Common formats) - when choosing a Shapefile, no layers are listed and thus it is still impossible to import anything. This issue affects also v.import and v.in.ogr as all of them seem to share the same UI components.

GRASS versija: 7.3.svn                                                          
GRASS SVN revīzija: r70508                                                      
Būvējuma datums: 2017-01-09                                                     
Būvēšanas platforma: x86_64-w64-mingw32                                         
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platforma: Windows-8-6.2.9200 (OSGeo4W

comment:7 in reply to:  6 Changed 3 years ago by hellik

Replying to marisn:

Tested import tool (File -> Import vector -> Common formats) - when choosing a Shapefile, no layers are listed and thus it is still impossible to import anything. This issue affects also v.import and v.in.ogr as all of them seem to share the same UI components.

GRASS versija: 7.3.svn                                                          
GRASS SVN revīzija: r70508                                                      
Būvējuma datums: 2017-01-09                                                     
Būvēšanas platforma: x86_64-w64-mingw32                                         
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platforma: Windows-8-6.2.9200 (OSGeo4W

confirmed, tested here:

System Info                                                                     
GRASS version: 7.2.0                                                            
GRASS SVN revision: r70156                                                      
Build date: 2016-12-29                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-8-6.2.9200 (OSGeo4W) 
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\gui_
core\gselect.py", line 2055, in OnUpdate

self._reloadLayers()
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\gui_
core\gselect.py", line 2103, in _reloadLayers

input=dsn)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\core
\gcmd.py", line 705, in RunCommand

quiet, verbose, **kwargs))
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\core.py", line 311, in make_command

args.append(opt + b'=' + _make_val(val))
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\core.py", line 98, in _make_val

return encode(val)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\utils.py", line 177, in encode

return string.encode(enc) if enc else string.encode()
  File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
36: character maps to <undefined>
System Info                                                                     
GRASS version: 7.3.svn                                                          
GRASS SVN revision: r70750M                                                     
Build date: 2017-03-14                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-8-6.2.9200 (OSGeo4W) 
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\gselect.py", line 2071, in OnUpdate

self._reloadLayers()
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\gselect.py", line 2119, in _reloadLayers

input=dsn)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\co
re\gcmd.py", line 705, in RunCommand

quiet, verbose, **kwargs))
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\core.py", line 312, in make_command

args.append(opt + b'=' + _make_val(val))
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\core.py", line 99, in _make_val

return encode(val)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\utils.py", line 184, in encode

return string.encode(enc)
  File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
36: character maps to <undefined>

comment:8 Changed 3 years ago by annakrat

Is there any non-ascii character somewhere in the path?

comment:9 Changed 3 years ago by hellik

Tested it then with the example in original report above.

comment:10 Changed 3 years ago by annakrat

Some of the characters are not defined in cp1252, that's where the error comes from. There is nothing else we can do about it except of giving a better error message. When I switch my locale, I didn't get any error but it still doesn't work, just as Maris describes.

This is output I get in terminal when I switch to Czech locale (cp1250). Note that character č is not defined in cp1252 but is defined in cp1250

C:\Users\anna>v.external in=C:\Users\anna\Desktop\déíáýšžč\aaaa.shp
D1/5: G_set_program_name(): v.external
D2/5: G_file_name(): path = C:\Users\anna\Documents\grassdata\nc_basic_spm_grass7\user1
D1/5: GRASS_VECTOR_OGR defined? no
D1/5: dsn: C:\Users\anna\Desktop\dÚÝßřÜ×Ŕ\aaaa.shp
ERROR: Unable to open data source 'C:\Users\anna\Desktop\dÚÝßřÜ×Ŕ\aaaa.shp'


When I remove č, it works:

C:\Users\anna>v.external in=C:\Users\anna\Desktop\déíáýšž\aaaa.shp --o
D1/5: G_set_program_name(): v.external
D2/5: G_file_name(): path = C:\Users\anna\Documents\grassdata\nc_basic_spm_grass7\user1
D1/5: GRASS_VECTOR_OGR defined? no
D1/5: dsn: C:\Users\anna\Desktop\dÚÝßřÜ×\aaaa.shp
D2/5: layer 'aaaa' was found
...

I think I am missing something here.

comment:11 Changed 22 months ago by neteler

Milestone: 7.4.07.4.1

Ticket retargeted after milestone closed

comment:12 Changed 17 months ago by neteler

Milestone: 7.4.17.4.2

comment:13 Changed 14 months ago by mlennert

Is this really an issue that can be handled within GRASS GIS ? It sounds more like a general encoding config issue.

comment:14 Changed 13 months ago by neteler

Can you please try in trunk/Python3?

comment:15 Changed 13 months ago by neteler

Milestone: 7.4.27.4.3

Ticket retargeted after milestone closed

comment:16 Changed 11 months ago by martinl

Milestone: 7.4.37.4.4

Bump milestone to 7.4.4

comment:17 Changed 11 months ago by neteler

Milestone: 7.4.47.4.5

Ticket retargeted after milestone closed

Note: See TracTickets for help on using tickets.