encoding issue when launching modules

[mlennert]

And another encoding issue:

In the GUI, going to the "Modules" tab and clicking on "Advanced search", the g.search.modules module window opens, but I get the following error message:

'ascii' codec can't encode character u'\xe0' in position 11: ordinal not in range(128)

When I then click 'OK' I can access the g.search.modules window and execute the command without any further issue.

[mlennert]

Apparently this is a larger issue:

i.segment --ui
Traceback (most recent call last):
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
    task.get_cmd(ignoreErrors=True, ignoreRequired=True))
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 248, in get_cmd
    errList = self.get_cmd_error()
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
    {'name': p['name'], 'desc': encode(desc)})
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/utils.py", line 217, in encode
    return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 29: ordinal not in range(128)

v.lrs.create --ui
Traceback (most recent call last):
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
    task.get_cmd(ignoreErrors=True, ignoreRequired=True))
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 248, in get_cmd
    errList = self.get_cmd_error()
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
    {'name': p['name'], 'desc': encode(desc)})
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/utils.py", line 217, in encode
    return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 36: ordinal not in range(128)

r.surf.idw --ui
Traceback (most recent call last):
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
    task.get_cmd(ignoreErrors=True, ignoreRequired=True))
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 248, in get_cmd
    errList = self.get_cmd_error()
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
    {'name': p['name'], 'desc': encode(desc)})
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/utils.py", line 217, in encode
    return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 35: ordinal not in range(128)

Interestingly, for all these modules no GUI window is created automatically if they are launched without '--ui' from the command line. I have the feeling this is new.

When I launch the modules from the GUI menu, I get a window with the error message, but clicking 'OK' lets me use the module normally.

[mlennert]

This is not an issue in release72, so changing milestone.

[mlennert]

I just tested on my office computer with a fresh svn checkout, and cannot reproduce. It must be something on my home computer... I'll try again tonight, but maybe this is just noise...

[mlennert]

I'm completely stumped on this one: my office and home machine both run Debian testing. I recompiled a fresh svn trunk checkout from scratch (r71433). I moved the .grass7 directories to avoid any cruft. locale settings are equal between the two. In neither do I have special characters anywhere in the gisenv paths.

Result: on the home machine I get the above errors, on the office machine I don't.

????

Setting LANGUAGE=C allows the module GUI to launch without an error on my home machine. The only difference I see is that on the office machine LANGUAGE=C and the launching the module does not give any messages in the terminal, while on the home machine, the same combination gives me:

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

Could that be the issue ?

[neteler]

Replying to mlennert:

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

Probably you are lacking the adwaita-gtk2-theme package? Here the resp. Fedora RPM:

rpm -qil adwaita-gtk2-theme
Name        : adwaita-gtk2-theme
Version     : 3.22.3
Release     : 2.fc26
...
URL         : http://git.gnome.org/browse/gnome-themes-standard
Summary     : Adwaita gtk2 theme
Description : The adwaita-gtk2-theme package contains a gtk2 theme for presenting widgets with a GNOME look and feel.

[mlennert]

Replying to neteler:

Replying to mlennert:

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

(forms.py:26034): Gtk-WARNING **: Unable to locate theme engine in module_path: "adwaita",

Probably you are lacking the adwaita-gtk2-theme package? Here the resp. Fedora RPM:

Thanks for the hint. In Debian, it's the gnome-themes-standard package.

Installing that gets rid of the Gtk-WARNINGs, but not of the bug trying to run the modules.

But I'm getting closer. In lib/python/script/utils.py we have the function encode, with the following code:

    if isinstance(string, bytes):
        return string
    if isinstance(string, unicode):
        enc = _get_encoding()
        return string.encode(enc)
    return bytes(string)

On the work computer the string gets recognized as unicode and therefore the code in the second if is run. On the home computer, the string is not recognized as unicode and the last line is run and causes the error.

However, if I modify the code as such:

    print type(string)
    if isinstance(string, bytes):
        print "isinstance bytes"
        return string
    if isinstance(string, unicode):
        print "isinstance unicode"
        enc = _get_encoding()
        return string.encode(enc)
    print "isinstance none"
    return bytes(string)

the output I get is

> v.to.rast --ui
<type 'str'>
isinstance bytes
<type 'str'>
isinstance bytes
<type 'unicode'>       <==================
isinstance none        <==================
Traceback (most recent call last):
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py", line 2995, in <module>
    task.get_cmd(ignoreErrors=True, ignoreRequired=True))
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 248, in get_cmd
    errList = self.get_cmd_error()
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/task.py", line 208, in get_cmd_error
    {'name': p['name'], 'desc': encode(desc)})
  File "/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/script/utils.py", line 221, in encode
    return bytes(string)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 36: ordinal not in range(128)

In other words, it is recognized as type unicode, but isinstance(string, unicode) returns false...

Actually, when type is 'str' it is recognized as unicode:

    print type(string)
    print isinstance(string, unicode)

gives

<type 'str'>
True
<type 'str'>
True
<type 'unicode'>
False

Any hints explaining why this is happening on one machine, but not the other ? Python versions are identical (Python 2.7.13).

[mlennert]

Replying to mlennert:

Actually, when type is 'str' it is recognized as unicode:

    print type(string)
    print isinstance(string, unicode)

gives

<type 'str'>
True
<type 'str'>
True
<type 'unicode'>
False

This is not the case on my office machine, where I get:

<type 'str'>
False
<type 'str'>
False
<type 'unicode'>
True

[mlennert]

Bingo ! I found the problem. At the top of lib/python/script/utils.py, there is this:

try:
    from builtins import unicode
except ImportError:
    # python3
    unicode = str

However, in order to be able to import builtins I have to install the package python-future which was not the case on my home computer. Does this mean we assume this to be a dependence ?

This said, if I just comment out this part of the code, I do not get the errors either, as apparently the decode function works as is with the builtin unicode type in python 2.

Anna, could you enlighten me why this import from future is necessary ? And if it is, we will have to make this an official dependence...

[mlennert]

Replying to mlennert:

Bingo ! I found the problem. At the top of lib/python/script/utils.py, there is this:

try:
    from builtins import unicode
except ImportError:
    # python3
    unicode = str

However, in order to be able to import builtins I have to install the package python-future which was not the case on my home computer. Does this mean we assume this to be a dependence ?

This said, if I just comment out this part of the code, I do not get the errors either, as apparently the decode function works as is with the builtin unicode type in python 2.

Anna, could you enlighten me why this import from future is necessary ? And if it is, we will have to make this an official dependence...

Sorry, wrong adressee: I just saw that Pietro introduced this just recently (r71394). Will contact him.

[mlennert]

Apparently fixed in r71435. I tested by removing the python-future package from my office computer and I get the GUI as expected. I'll test on my home computer ASAP before closing.

[neteler]

G7.2.svn tested ok on Linux (Fedora 26) with

 export LANG=de_DE.UTF-8
 export LANGUAGE=de_DE.UTF-8
 export LC_MESSAGES=de_DE.UTF-8

and

 export LANG=fr_FR.UTF-8
 export LANGUAGE=fr_FR.UTF-8
 export LC_MESSAGES=fr_FR.UTF-8

[mlennert]

Replying to neteler:

G7.2.svn tested ok on Linux (Fedora 26) with

 export LANG=de_DE.UTF-8
 export LANGUAGE=de_DE.UTF-8
 export LC_MESSAGES=de_DE.UTF-8

and

 export LANG=fr_FR.UTF-8
 export LANGUAGE=fr_FR.UTF-8
 export LC_MESSAGES=fr_FR.UTF-8

I think we can release 7.2.2RC2 now. With a big fat call for testing on the lists...

[mlennert]

Closing as fixed in trunk and 7.2.