wxGUI v.in.ogr wrapper fails to find any layers if file path contains non-latin letters

[marisn]

Steps to reproduce:

• mkdir "šaursliežu dzelzceļš"
• copy "myshapefile.*" "šaursliežu dzelzceļš/"
• start GRASS, open vector data import dialog
• select "C:\balhblah\šaursliežu dzelzceļš\myshapefile.shp" as a data source

Observe how nothing happens. No error messages, no hangs, just nothing. No layers are listed and thus is not possible to continue with data import.

GRASS version: 7.0.0svn GRASS SVN Revision: 63925 Build Date: 2015-01-02 Build Platform: i686-pc-mingw32 GDAL/OGR: 1.11.1 PROJ.4: 4.8.0 GEOS: 3.4.2 SQLite: 3.7.17 Python: 2.7.4 wxPython: 2.8.12.1 Platform: Windows-Vista-6.0.6002-SP2

[hellik]

Replying to marisn:

Steps to reproduce:

• mkdir "šaursliežu dzelzceļš"
• copy "myshapefile.*" "šaursliežu dzelzceļš/"
• start GRASS, open vector data import dialog
• select "C:\balhblah\šaursliežu dzelzceļš\myshapefile.shp" as a data source

Observe how nothing happens. No error messages, no hangs, just nothing. No layers are listed and thus is not possible to continue with data import.

GRASS version: 7.0.0svn GRASS SVN Revision: 63925 Build Date: 2015-01-02 Build Platform: i686-pc-mingw32 GDAL/OGR: 1.11.1 PROJ.4: 4.8.0 GEOS: 3.4.2 SQLite: 3.7.17 Python: 2.7.4 wxPython: 2.8.12.1 Platform: Windows-Vista-6.0.6002-SP2

Traceback (most recent call last):
  File "C:\OSGeo4W\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\gselect.py", line 1920, in OnUpdate

self._reloadLayers()
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\gui
_core\gselect.py", line 1938, in _reloadLayers

input = dsn)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\gui\wxpython\cor
e\gcmd.py", line 694, in RunCommand

ps = grass.start_command(prog, flags, overwrite, quiet,
verbose, **kwargs)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass
\script\core.py", line 349, in start_command

val = encode(val)
  File "C:\OSGeo4W\apps\grass\grass-7.1.svn\etc\python\grass
\script\utils.py", line 218, in encode

return string.encode(enc)
  File "C:\OSGeo4W\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
24: character maps to <undefined>

[annakrat]

The error is the same as in #2205. I have no idea about that one.

I don't get this error on Ubuntu. But when I have non ascii characters in the name of the file (not in the path to it), I got a different error coming from GetValidLayerName (this function tries to make the name sql compliant). Hopefully fixed in 63943.

[martinl]

Still an issue?

[annakrat]

Tested on Ubuntu and Windows, no problems.

[marisn]

Tested import tool (File -> Import vector -> Common formats) - when choosing a Shapefile, no layers are listed and thus it is still impossible to import anything. This issue affects also v.import and v.in.ogr as all of them seem to share the same UI components.

GRASS versija: 7.3.svn                                                          
GRASS SVN revīzija: r70508                                                      
Būvējuma datums: 2017-01-09                                                     
Būvēšanas platforma: x86_64-w64-mingw32                                         
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platforma: Windows-8-6.2.9200 (OSGeo4W

[hellik]

Replying to marisn:

Tested import tool (File -> Import vector -> Common formats) - when choosing a Shapefile, no layers are listed and thus it is still impossible to import anything. This issue affects also v.import and v.in.ogr as all of them seem to share the same UI components.

GRASS versija: 7.3.svn                                                          
GRASS SVN revīzija: r70508                                                      
Būvējuma datums: 2017-01-09                                                     
Būvēšanas platforma: x86_64-w64-mingw32                                         
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platforma: Windows-8-6.2.9200 (OSGeo4W

confirmed, tested here:

System Info                                                                     
GRASS version: 7.2.0                                                            
GRASS SVN revision: r70156                                                      
Build date: 2016-12-29                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-8-6.2.9200 (OSGeo4W) 

Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\gui_
core\gselect.py", line 2055, in OnUpdate

self._reloadLayers()
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\gui_
core\gselect.py", line 2103, in _reloadLayers

input=dsn)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\gui\wxpython\core
\gcmd.py", line 705, in RunCommand

quiet, verbose, **kwargs))
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\core.py", line 311, in make_command

args.append(opt + b'=' + _make_val(val))
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\core.py", line 98, in _make_val

return encode(val)
  File "C:\OSGEO4~1\apps\grass\grass-7.2.0\etc\python\grass\
script\utils.py", line 177, in encode

return string.encode(enc) if enc else string.encode()
  File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
36: character maps to <undefined>

System Info                                                                     
GRASS version: 7.3.svn                                                          
GRASS SVN revision: r70750M                                                     
Build date: 2017-03-14                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.1.2                                                                     
PROJ.4: 4.9.3                                                                   
GEOS: 3.5.0                                                                     
SQLite: 3.14.1                                                                  
Python: 2.7.5                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-8-6.2.9200 (OSGeo4W) 

Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\Python27\lib\site-packages\wx-2.8
-msw-unicode\wx\lib\filebrowsebutton.py", line 140, in
OnChanged

self.changeCallback(evt)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\gselect.py", line 2071, in OnUpdate

self._reloadLayers()
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\gu
i_core\gselect.py", line 2119, in _reloadLayers

input=dsn)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\gui\wxpython\co
re\gcmd.py", line 705, in RunCommand

quiet, verbose, **kwargs))
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\core.py", line 312, in make_command

args.append(opt + b'=' + _make_val(val))
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\core.py", line 99, in _make_val

return encode(val)
  File "C:\OSGEO4~1\apps\grass\grass-7.3.svn\etc\python\gras
s\script\utils.py", line 184, in encode

return string.encode(enc)
  File "C:\OSGEO4~1\apps\Python27\lib\encodings\cp1252.py",
line 12, in encode

return codecs.charmap_encode(input,errors,encoding_table)
UnicodeEncodeError
:
'charmap' codec can't encode character u'\u013c' in position
36: character maps to <undefined>

[annakrat]

Is there any non-ascii character somewhere in the path?

[hellik]

Tested it then with the example in original report above.

[annakrat]

Some of the characters are not defined in cp1252, that's where the error comes from. There is nothing else we can do about it except of giving a better error message. When I switch my locale, I didn't get any error but it still doesn't work, just as Maris describes.

This is output I get in terminal when I switch to Czech locale (cp1250). Note that character č is not defined in cp1252 but is defined in cp1250

C:\Users\anna>v.external in=C:\Users\anna\Desktop\déíáýšžč\aaaa.shp
D1/5: G_set_program_name(): v.external
D2/5: G_file_name(): path = C:\Users\anna\Documents\grassdata\nc_basic_spm_grass7\user1
D1/5: GRASS_VECTOR_OGR defined? no
D1/5: dsn: C:\Users\anna\Desktop\dÚÝßřÜ×Ŕ\aaaa.shp
ERROR: Unable to open data source 'C:\Users\anna\Desktop\dÚÝßřÜ×Ŕ\aaaa.shp'

When I remove č, it works:

C:\Users\anna>v.external in=C:\Users\anna\Desktop\déíáýšž\aaaa.shp --o
D1/5: G_set_program_name(): v.external
D2/5: G_file_name(): path = C:\Users\anna\Documents\grassdata\nc_basic_spm_grass7\user1
D1/5: GRASS_VECTOR_OGR defined? no
D1/5: dsn: C:\Users\anna\Desktop\dÚÝßřÜ×\aaaa.shp
D2/5: layer 'aaaa' was found
...

I think I am missing something here.

[neteler]

Ticket retargeted after milestone closed

[mlennert]

Is this really an issue that can be handled within GRASS GIS ? It sounds more like a general encoding config issue.

[neteler]

Can you please try in trunk/Python3?

[neteler]

Ticket retargeted after milestone closed

[martinl]

Bump milestone to 7.4.4

[neteler]

Ticket retargeted after milestone closed