Opened 7 years ago

Closed 14 months ago

#1672 closed defect (fixed)

gcpmanager fails to list source mapsets if non-ascii characters in grass data dir path

Reported by: hamish Owned by: grass-dev@…
Priority: normal Milestone: 7.2.0
Component: wxGUI Version: svn-trunk
Keywords: georectifier, UnicodeDecodeError Cc: torsti.schulz@…
CPU: x86-64 Platform: Linux

Description

[Hi, one fwd'd from the Debian bug tracker, http://bugs.debian.org/676190 --H]

From: Torsti Schulz <torsti.schulz at iki dot fi>
To: Debian Bug Tracking System
Subject: grass-gui: gcpmanager fails to list source mapsets if non-ascii characters
 in grass data dir path
Date: Tue, 05 Jun 2012 14:35:48 +0300

Package: grass-gui
Version: 6.4.2-1
Severity: normal

Hi,

Using the wxpython GUI Georectify tool is impossible if the path to the grass
data directory (e.g. GRASSDATA) contains non ascii characters (e.g.
'/home/user/Työpöytä/GRASSDATA'). Selecting the source location works, but
the 'Select source mapset:' dropdown menu remains greyed out. The command
console shows the following error message, after selecting the source location:

***
Traceback (most recent call last):
  File
"/usr/lib/grass64/etc/wxpython/gui_modules/gcpmanager.py",
line 377, in OnLocation

tmplist = os.listdir(os.path.join(self.grassdatabase,
self.xylocation))
  File "/usr/lib/python2.7/posixpath.py", line 71, in join

path += '/' + b
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 15: ordinal
not in range(128)
***

Changing the grass data directory path to only contain ascii characters (e.g.
'/home/user/GRASSDATA') makes the Georectify tool usable. This seems to an
issue with how the python code handles path names and unicode characters.

Cheers,
Torsti



-- System Information:
Debian Release: wheezy/sid
  APT prefers testing
  APT policy: (700, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=fi_FI.utf8, LC_CTYPE=fi_FI.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages grass-gui depends on:
ii  gnome-terminal [x-terminal-emulator]  3.4.1.1-1
ii  grass-core                            6.4.2-1
ii  lesstif2                              1:0.95.2-1
ii  libc6                                 2.13-33
ii  libfreetype6                          2.4.9-1
ii  libgdal1                              1.9.0-2
ii  libgl1-mesa-glx [libgl1]              8.0.2-2
ii  libglu1-mesa [libglu1]                8.0.2-2
ii  libice6                               2:1.0.8-2
ii  libpng12-0                            1.2.49-1
ii  libsm6                                2:1.2.1-2
ii  libtiff5                              4.0.1-7
ii  libx11-6                              2:1.4.99.901-2
ii  libxext6                              2:1.3.1-2
ii  libxmu6                               2:1.1.1-1
ii  libxt6                                1:1.1.3-1
ii  python                                2.7.2-10
ii  python-opengl                         3.0.1-1
ii  python-wxgtk2.8                       2.8.12.1-11
ii  tcl8.5                                8.5.11-2
ii  tk8.5                                 8.5.11-2
ii  xterm [x-terminal-emulator]           278-1
ii  zlib1g                                1:1.2.7.dfsg-11

Versions of packages grass-gui recommends:
ii  ghostscript     9.05~dfsg-6
ii  python-imaging  1.1.7-4

grass-gui suggests no packages.

Change History (15)

comment:1 Changed 7 years ago by hamish

see also

and

Hamish

comment:2 Changed 7 years ago by torsti

Version: 6.4.2svn-trunk

This one is still around (r55643). It can be traced back to 'self.grassdatabase' in 'OnLocation' (gui/wxpython/gcp/manager.py, line 376), which gets its value at line 94:

self.grassdatabase = grass.gisenv()['GISDBASE']

'gisenv' is defined at line 720 in lib/python/script/core.py:

s = read_command("g.gisenv", flags='n')
return parse_key_val(s)

Neither the string 's' read from g.gisenv nor the dictionary returned by 'parse_key_val' is explicitly decoded to Unicode, so when it is finally implicitly cast to Unicode somewhere in Python's bowels, the default ascii codec is used, which leads to errors if $GISDBASE contains non ascii characters.

This a wider python (at least for < 3.x) problem, and fixing it for this specific issue won't probably solve all existing UnicodeDecodeErrors nor prevent new ones from surfacing, but at least any GUI component calling gisenv can probably be fixed at one location.

Any filenames/paths read into wxpython/pygrass should probably be explicitly decoded to Unicode, e.g. with the 'decode()' function defined at line 79 in script/core.py. Enforcing only ascii chars works for mapset and location names, but can't/shouldn't be the solution for path components not under GRASS's direct control.

comment:3 Changed 4 years ago by Bas Couwenberg

Milestone: 6.4.37.0.2

As reported in Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

comment:4 Changed 4 years ago by neteler

Milestone: 7.0.27.0.3

Ticket retargeted after milestone closed

comment:5 Changed 4 years ago by mlennert

This seems to be fixed in trunk, but not in release70. However, I don't have the time right now to try and find out which of the fixes in trunk fixed this particular issue...

comment:6 Changed 4 years ago by neteler

Milestone: 7.0.3

Ticket retargeted after milestone closed

comment:7 Changed 4 years ago by neteler

Milestone: 7.0.4

Ticket retargeted after 7.0.3 milestone closed

comment:8 in reply to:  3 ; Changed 4 years ago by neteler

Replying to sebastic:

As reported in Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

comment:9 in reply to:  8 ; Changed 4 years ago by mlennert

Replying to neteler:

Replying to sebastic:

As reported in Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

r65787 ? It adds encoding and decoding to the parse_key_val which is used in gisenv(). I'm still too confused by encoding handling in Python to be able to say whether this is the "right" solution to the problem, though.

comment:10 in reply to:  9 Changed 4 years ago by mlennert

Replying to mlennert:

Replying to neteler:

Replying to sebastic:

As reported in Debian Bug #676190, this issue is still present in GRASS 7.0.1. When non-ascii characters are in the grass data path, the 'Select source mapset:' dropdown is populated with empty options. When no non-ascii characters are the path the dropdown is populated with the expected mapsets.

Anybody able to identify the change we need to backport?

r65787 ? It adds encoding and decoding to the parse_key_val which is used in gisenv().

Yes, by merging r65787 the bug seems to be fixed.

I'm still too confused by encoding handling in Python to be able to say whether this is the "right" solution to the problem, though.

Could some of the wxgui gurus look at this to see if this can just be backported or if another solution would be better ?

comment:11 Changed 4 years ago by martinl

Milestone: 7.0.47.0.5

comment:12 Changed 3 years ago by neteler

Milestone: 7.0.57.0.6

comment:13 Changed 3 years ago by Bas Couwenberg

While backporting the fix to 7.0.x would be nice, it is no longer present in 7.2.0RC2 (nor RC1).

comment:14 Changed 21 months ago by neteler

Milestone: 7.0.67.0.7

comment:15 Changed 14 months ago by martinl

Milestone: 7.0.77.2.0
Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.