UnicodeDecodeError in r.category in wxGUI when category labels contain special characters

[mlennert]

I have a map with the following categories:

> r.category liege_ucl
0	
10	eau
15	sol nu
20	bâti
21	route
30	terre labourée
35	herbage non naturel
40	zone ouverte (semi)naturelle 1
45	zone ouverte (semi)naturelle 2
48	zone ouverte (semi)naturelle 3
50	résineux
55	feuillus

In the GUI, r.category gives:

r.category map=liege_ucl@RW                                                     
0	
10	eau
15	sol nu
21	route
35	herbage non naturel
40	zone ouverte (semi)naturelle 1
45	zone ouverte (semi)naturelle 2
48	zone ouverte (semi)naturelle 3
55	feuillus

i.e. all categories with special characters in the labels are just ignored.

The traceback in the Console window:

Traceback (most recent call last):
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
  File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 4: ordinal
not in range(128)
Traceback (most recent call last):
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
  File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 15: ordinal
not in range(128)
Traceback (most recent call last):
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 472, in
OnCmdOutput

self.cmdOutput.AddStyledMessage(message, type)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 771, in
AddStyledMessage

self.AddTextWrapped(message, wrap=None)
  File
"/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-pc-
linux-gnu/gui/wxpython/gui_core/goutput.py", line 722, in
AddTextWrapped

self.AddText(txt)
  File "/usr/lib/python2.7/dist-
packages/wx-3.0-gtk2/wx/stc.py", line 2041, in AddText

return _stc.StyledTextCtrl_AddText(*args, **kwargs)
UnicodeDecodeError
:
'ascii' codec can't decode byte 0xc3 in position 4: ordinal
not in range(128)

[annakrat]

Hmm, I can't reproduce it on linux. Could you test if it is still the problem?

[marisn]

Works for me from CLI, GUI and Layer Manager Console tab:

r.category map=labels
1	arumi
2	vagas
3	blīvs tīrums
4	skrajš tīrums

GRASS versija: 7.3.svn
GRASS SVN revīzija: r71685M
Būvējuma datums: 2017-11-12
Būvēšanas platforma: x86_64-pc-linux-gnu
wxPython: 3.0.2.0

LANG=lv_LV
LC_CTYPE=lv_LV.UTF-8
LC_NUMERIC=C

[mlennert]

Replying to annakrat:

Hmm, I can't reproduce it on linux. Could you test if it is still the problem?

I have just checked. There is no issue if the GUI is defined in the system's locale. However, choosing English as language also resets all the locale settings and so when the GUI is in English, any accented characters cannot be displayed.

So the question becomes: should choice of the language in which the GUI appears influence the locale settings in this manner ? If you say yes, than we can close this.

[marisn]

Replying to mlennert:

So the question becomes: should choice of the language in which the GUI appears influence the locale settings in this manner ? If you say yes, than we can close this.

Yes, it should. There are messages also provided by non-GUI apps or GUI apps can be started from CLI thus bypassing wxgui language switching mechanisms.

Here is a test case triggering a failure without switching GUI language (NC basic; sorry for my Russian):

g.copy rast=landuse,landuse
r.category map=landuse separator=":" rules=- << EOF
1:развитая
2:сельское хозяйство
3:травянистый
4:кустарниковые
5:лесной
6:водный
7:осадок
EOF
r.category map=landuse

[mlennert]

Replying to marisn:

Replying to mlennert:

So the question becomes: should choice of the language in which the GUI appears influence the locale settings in this manner ? If you say yes, than we can close this.

Yes, it should. There are messages also provided by non-GUI apps or GUI apps can be started from CLI thus bypassing wxgui language switching mechanisms.

So, the GUI language should influence all the rest (which it doesn't by the way, since r.category works like a charm with special characters on the command line even when the GUI is set to 'en', i.e. to locale=C). Personally, I would prefer to be able to choose to show the GUI in another language without loosing half of the attribute info...

One related question is why 'en' should lead to locale=C. We could have the GUI in en_EN.UTF-8 or something like that.

Here is a test case triggering a failure without switching GUI language (NC basic; sorry for my Russian):

When you say 'without switching GUI language', do you mean that the GUI is set to Russian ?

g.copy rast=landuse,landuse
r.category map=landuse separator=":" rules=- << EOF
1:развитая
2:сельское хозяйство
3:травянистый
4:кустарниковые
5:лесной
6:водный
7:осадок
EOF
r.category map=landuse

I do not have any failure here, with the GUI language set to 'system' which is

locale
LANG=fr_BE
LANGUAGE=fr_BE
LC_CTYPE=fr_BE.UTF-8
LC_NUMERIC=C
LC_TIME=fr_BE.UTF-8
LC_COLLATE=fr_BE.UTF-8
LC_MONETARY=fr_BE.UTF-8
LC_MESSAGES=fr_BE.UTF-8
LC_PAPER=fr_BE.UTF-8
LC_NAME=fr_BE.UTF-8
LC_ADDRESS=fr_BE.UTF-8
LC_TELEPHONE=fr_BE.UTF-8
LC_MEASUREMENT=fr_BE.UTF-8
LC_IDENTIFICATION=fr_BE.UTF-8
LC_ALL=

However, when I switch the GUI language to 'en', I get:

r.category map=landuse@user1                                                    
(Sun Nov 12 18:44:33 2017) Command finished (0 sec)   

i.e. nothing...

[annakrat]

Replying to mlennert:

Replying to marisn:

Replying to mlennert:

So the question becomes: should choice of the language in which the GUI appears influence the locale settings in this manner ? If you say yes, than we can close this.

One related question is why 'en' should lead to locale=C. We could have the GUI in en_EN.UTF-8 or something like that.

So it seems the problem happens only for 'en'. I looked at it but I don't quite understand the problem, maybe what you suggest would work...

[marisn]

Closing this one as the bug is not in r.category but in switching to en language (#3441).