gcpmanager fails to list source mapsets if non-ascii characters in grass data dir path

[hamish]

[Hi, one fwd'd from the Debian bug tracker, ​http://bugs.debian.org/676190 --H]

From: Torsti Schulz <torsti.schulz at iki dot fi>
To: Debian Bug Tracking System
Subject: grass-gui: gcpmanager fails to list source mapsets if non-ascii characters
 in grass data dir path
Date: Tue, 05 Jun 2012 14:35:48 +0300

Package: grass-gui
Version: 6.4.2-1
Severity: normal

Hi,

Using the wxpython GUI Georectify tool is impossible if the path to the grass
data directory (e.g. GRASSDATA) contains non ascii characters (e.g.
'/home/user/Työpöytä/GRASSDATA'). Selecting the source location works, but
the 'Select source mapset:' dropdown menu remains greyed out. The command
console shows the following error message, after selecting the source location:

***
Traceback (most recent call last):
  File
"/usr/lib/grass64/etc/wxpython/gui_modules/gcpmanager.py",
line 377, in OnLocation

tmplist = os.listdir(os.path.join(self.grassdatabase,
self.xylocation))
  File "/usr/lib/python2.7/posixpath.py", line 71, in join

path += '/' + b
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 15: ordinal
not in range(128)
***

Changing the grass data directory path to only contain ascii characters (e.g.
'/home/user/GRASSDATA') makes the Georectify tool usable. This seems to an
issue with how the python code handles path names and unicode characters.

Cheers,
Torsti



-- System Information:
Debian Release: wheezy/sid
  APT prefers testing
  APT policy: (700, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=fi_FI.utf8, LC_CTYPE=fi_FI.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages grass-gui depends on:
ii  gnome-terminal [x-terminal-emulator]  3.4.1.1-1
ii  grass-core                            6.4.2-1
ii  lesstif2                              1:0.95.2-1
ii  libc6                                 2.13-33
ii  libfreetype6                          2.4.9-1
ii  libgdal1                              1.9.0-2
ii  libgl1-mesa-glx [libgl1]              8.0.2-2
ii  libglu1-mesa [libglu1]                8.0.2-2
ii  libice6                               2:1.0.8-2
ii  libpng12-0                            1.2.49-1
ii  libsm6                                2:1.2.1-2
ii  libtiff5                              4.0.1-7
ii  libx11-6                              2:1.4.99.901-2
ii  libxext6                              2:1.3.1-2
ii  libxmu6                               2:1.1.1-1
ii  libxt6                                1:1.1.3-1
ii  python                                2.7.2-10
ii  python-opengl                         3.0.1-1
ii  python-wxgtk2.8                       2.8.12.1-11
ii  tcl8.5                                8.5.11-2
ii  tk8.5                                 8.5.11-2
ii  xterm [x-terminal-emulator]           278-1
ii  zlib1g                                1:1.2.7.dfsg-11

Versions of packages grass-gui recommends:
ii  ghostscript     9.05~dfsg-6
ii  python-imaging  1.1.7-4

grass-gui suggests no packages.

[hamish]

see also

• ​http://bugs.debian.org/595815 "sqlite database update failure with Unicode characters in wxpython gui"

and

• ​http://bugs.debian.org/595910 "wxpython: save of workspace on exit crasches on non-ascii layer name and corrupts workspace file"

Hamish

[torsti]

This one is still around (r55643). It can be traced back to 'self.grassdatabase' in 'OnLocation' (gui/wxpython/gcp/manager.py, line 376), which gets its value at line 94:

self.grassdatabase = grass.gisenv()['GISDBASE']

'gisenv' is defined at line 720 in lib/python/script/core.py:

s = read_command("g.gisenv", flags='n')
return parse_key_val(s)

Neither the string 's' read from g.gisenv nor the dictionary returned by 'parse_key_val' is explicitly decoded to Unicode, so when it is finally implicitly cast to Unicode somewhere in Python's bowels, the default ascii codec is used, which leads to errors if $GISDBASE contains non ascii characters.

This a wider python (at least for < 3.x) problem, and fixing it for this specific issue won't probably solve all existing UnicodeDecodeErrors nor prevent new ones from surfacing, but at least any GUI component calling gisenv can probably be fixed at one location.

Any filenames/paths read into wxpython/pygrass should probably be explicitly decoded to Unicode, e.g. with the 'decode()' function defined at line 79 in script/core.py. Enforcing only ascii chars works for mapset and location names, but can't/shouldn't be the solution for path components not under GRASS's direct control.

[Bas Couwenberg]

As reported in ​Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

[neteler]

Ticket retargeted after milestone closed

[mlennert]

This seems to be fixed in trunk, but not in release70. However, I don't have the time right now to try and find out which of the fixes in trunk fixed this particular issue...

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after 7.0.3 milestone closed

[neteler]

Replying to sebastic:

As reported in ​Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

[mlennert]

Replying to neteler:

Replying to sebastic:

As reported in ​Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

r65787 ? It adds encoding and decoding to the parse_key_val which is used in gisenv(). I'm still too confused by encoding handling in Python to be able to say whether this is the "right" solution to the problem, though.

[mlennert]

Replying to mlennert:

Replying to neteler:

Replying to sebastic:

As reported in ​Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

r65787 ? It adds encoding and decoding to the parse_key_val which is used in gisenv().

Yes, by merging r65787 the bug seems to be fixed.

I'm still too confused by encoding handling in Python to be able to say whether this is the "right" solution to the problem, though.

Could some of the wxgui gurus look at this to see if this can just be backported or if another solution would be better ?

[Bas Couwenberg]

While backporting the fix to 7.0.x would be nice, it is no longer present in 7.2.0RC2 (nor RC1).