Opened 5 years ago

Closed 4 years ago

#3838 closed defect (fixed)

r74307 breaks GUI: UnicodeDecodeError

Reported by: marisn Owned by: grass-dev@…
Priority: normal Milestone: 7.8.3
Component: wxGUI Version: svn-trunk
Keywords: Cc:
CPU: Unspecified Platform: Linux

Description

Any attempts to run some GUI parts result in a partial or complete failure with CLI filling with UnicodeDecodeError messages. r74306 is fine, r74307 breaks everything.

GRASS GIS 7.7.svn (r74442M) locale: lv_LV.UTF-8 Python: 2.7.15 dev-python/wxpython-3.0.2.0:3.0 x11-libs/wxGTK-3.0.4-r1:3.0

Startup screen: 

Startē GRASS GIS...
/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 545, in DoGetBestSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc4 in position 27: invalid continuation byte
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 545, in DoGetBestSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 7: invalid continuation byte
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 556, in OnSize
    self._updateLabel()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/widgets.py", line 565, in _updateLabel
    GenStaticText.SetLabel(self, newLabel)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/lib/stattext.py", line 58, in SetLabel
    wx.PyControl.SetLabel(self, label)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 9207, in SetLabel
    return _core_.Window_SetLabel(*args, **kwargs)
  File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc5 in position 145: unexpected end of data

Launching <wxpython> GUI in the background, please wait...
/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/wxgui.py", line 172, in <module>
    sys.exit(main())
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/wxgui.py", line 159, in main
    app = GMApp(workspaceFile)
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/wxgui.py", line 53, in __init__
    wx.App.__init__(self, False)
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 8628, in __init__
    self._BootstrapApp()
  File "/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py", line 8196, in _BootstrapApp
    return _core_.PyApp__BootstrapApp(*args, **kwargs)
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/wxgui.py", line 106, in OnInit
    workspace=self.workspaceFile)
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/lmgr/frame.py", line 90, in __init__
    self.baseTitle = _("GRASS GIS %s Layer Manager") % grassVersion
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc4 in position 5: ordinal not in range(128)

g.region
/usr/lib64/python2.7/site-packages/wx-3.0-gtk2/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")
/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py:994: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if content in sections:
/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/forms.py:995: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  idx = sections.index(content)
Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/gselect.py", line 448, in OnPopup
    self.GetElementList(selected, exclude)
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/gselect.py", line 460, in GetElementList
    self._getElementList(self.type, self.mapsets, elements, exclude)
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-gnu/gui/wxpython/gui_core/gselect.py", line 544, in _getElementList
    _('Mapset') + ': ' + mapset, node=True, mapset=mapset)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 3: ordinal not in range(128)

Change History (16)

comment:1 by pmav99, 5 years ago

I can't reproduce this, but I am using wxPython 4: ​https://extras.wxpython.org/wxPython4/extras/linux/gtk2/ubuntu-18.04/

comment:2 by neteler, 5 years ago

I think we do not want to revert r74307 but rather bisect the problem with wxPython 3. Maybe in a docker environment?

comment:3 by pmav99, 5 years ago

Q: Is it necessary to support multiple wxPython versions? Is there any problem with picking either v3 or v4 and moving forward?

in reply to:  3 comment:4 by mmetz, 5 years ago

Replying to pmav99:

Q: Is it necessary to support multiple wxPython versions? Is there any problem with picking either v3 or v4 and moving forward?

On RHEL7, there is no wxPython4 available AFAICT, and RHEL7 will be an important production system until about 2024.

IMHO, GRASS 7.x must run on RHEL7 and other currently supported production systems. We can break this with GRASS 8.

in reply to:  3 ; comment:5 by marisn, 5 years ago

Replying to pmav99:

Q: Is it necessary to support multiple wxPython versions? Is there any problem with picking either v3 or v4 and moving forward?

There is no v4 in Gentoo and also Ubuntu LTS (only one year old) sits on v3. Due to required changes in various other packages using wxPython, v4 widespread adoption will take some time.

in reply to:  5 ; comment:6 by annakrat, 5 years ago

Replying to marisn:

Replying to pmav99:

Q: Is it necessary to support multiple wxPython versions? Is there any problem with picking either v3 or v4 and moving forward?

There is no v4 in Gentoo and also Ubuntu LTS (only one year old) sits on v3. Due to required changes in various other packages using wxPython, v4 widespread adoption will take some time.

Yes, we need to support both Python 2 and 3, and wxPython 3 and 4, at least for some time.

Regarding this ticket, I believe it's Python 2 vs 3 problem, not wx 3 vs 4.

in reply to:  6 comment:7 by pmav99, 5 years ago

Replying to annakrat:

Regarding this ticket, I believe it's Python 2 vs 3 problem, not wx 3 vs 4.

I could not reproduce this either on Python2 or Python3. That's why I suggested that it might be related to WX. I tried again on current trunk (both P2 and P3) and I still can't reproduce it. I will try to install wx3 on my virtualenv and report back.

Last edited 5 years ago by pmav99 (previous) (diff)

comment:8 by pmav99, 5 years ago

Just tried this with:

  • OS: Archlinux
  • Linux: 5.0.13
  • Python: 2.7.16
  • wxPython: 3.0.2.0.1
  • wxgtk3: 3.0.4-2

and I still can't reproduce the error. The GUI opens without any tracebacks.

I do get the same warning as @marisn: 

Launching <wxpython> GUI in the background, please wait...
10:37:23: Warning: Mismatch between the program and library build versions detected.
The library used 3.0 (wchar_t,compiler with C++ ABI 1013,wx containers,compatible with 2.8),
and wxPython used 3.0 (wchar_t,compiler with C++ ABI 1011,wx containers,compatible with 2.8).
/usr/lib/python2.7/site-packages/wx-3.0-gtk3/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")

but this is related to the version mismatch between the installed versions of wxgtk3 and wxpython, which should not be relevant to the issue at hand.

This might be related to GTK2 vs GKT3 after all. Those of you who can reproduce this, can you please post wxpython and GTK versions?

in reply to:  6 comment:9 by pmav99, 5 years ago

Replying to annakrat:

Yes, we need to support both Python 2 and 3, and wxPython 3 and 4, at least for some time.

In that case, CI should build GRASS for all supported combinations.

in reply to:  8 comment:10 by marisn, 5 years ago

Replying to pmav99:

and I still can't reproduce the error. The GUI opens without any tracebacks.

Please try to run GRASS in a following way:LC_ALL=lv_LV.UTF-8 LANG=lv_LV.UTF-8 LANGUAGE=lv_LV.UTF-8 grass77

comment:11 by pmav99, 5 years ago

Dear Maris

Could you please try the following patch: 

index 1183e93b..0bd515b2 100644
--- lib/python/__init__.py
+++ lib/python/__init__.py
@@ -1,6 +1,8 @@
 import gettext
 import os
 
+import six
+
 # Setup i18N
 #
 # Calling `gettext.install()` injects `_()` in the builtins namespace and
@@ -17,9 +19,14 @@ import os
 # - https://www.wefearchange.org/2012/06/the-right-way-to-internationalize-your.html
 #
 _LOCALE_DIR = os.path.join(os.getenv("GISBASE"), 'locale')
-gettext.install('grasslibs', _LOCALE_DIR)
-gettext.install('grassmods', _LOCALE_DIR)
-gettext.install('grasswxpy', _LOCALE_DIR)
+if six.PY2:
+    gettext.install('grasslibs', _LOCALE_DIR, unicode=True)
+    gettext.install('grassmods', _LOCALE_DIR, unicode=True)
+    gettext.install('grasswxpy', _LOCALE_DIR, unicode=True)
+else:
+    gettext.install('grasslibs', _LOCALE_DIR)
+    gettext.install('grassmods', _LOCALE_DIR)
+    gettext.install('grasswxpy', _LOCALE_DIR)
 
 
 __all__ = ["script", "temporal"]

in reply to:  11 comment:12 by marisn, 5 years ago

Replying to pmav99:

Dear Maris

Could you please try the following patch:

Yes, this patch solves most of problems. At least I can start GUI and do some simple things.

I still see a new error when I try to run some raster modules with GUI (d.rast; d.rgb; r.lake): 

'ascii' codec can't decode byte 0xc4 in position 3: ordinal not in range(128)

with following in the console: 

Traceback (most recent call last):
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 2499, in
OnUpdateDialog

self.parent.updateValuesHook()
  File "/home/maris/soft/grass_trunk/dist.x86_64-pc-linux-
gnu/gui/wxpython/gui_core/forms.py", line 770, in
updateValuesHook

ignoreErrors=True)]))
TypeError
:
'NoneType' object is not iterable

Still it is hard to say if it is related or a different set of problems. For sure before the r74307 they were absent.

comment:13 by pmav99, 5 years ago

I just tried the following using Python 2 both at trunk and at r74305. There were no tracebacks in either case. 

$ LC_ALL=fr_FR.UTF-8 ./run_grass.sh ~/gdata2/nc_spm_full_v2alpha/PERMANENT --text 
          __________  ___   __________    _______________
         / ____/ __ \/   | / ___/ ___/   / ____/  _/ ___/
        / / __/ /_/ / /| | \__ \\_  \   / / __ / / \__ \
       / /_/ / _, _/ ___ |___/ /__/ /  / /_/ // / ___/ /
       \____/_/ |_/_/  |_/____/____/   \____/___//____/

Bienvenue dans le SIG GRASS 7.7.svn (exported)
Page d'accueil du SIG GRASS :            https://grass.osgeo.org
Cette version fonctionne avec :          Z Shell (/usr/bin/zsh)
L'aide est disponible par la commande :  g.manual -i
Voir les termes de la licence avec :     g.version -c
Voir les termes de la licence avec :     g.version -x
Démarrer l'interface graphique avec :   g.gui wxpython
Lors prêt pour la fermeture entrer :    exit

$ d.mon -l
No monitors running

$ d.mon start=wx0
12:22:59: Warning: Mismatch between the program and library build versions detected.                                                                                                                               
The library used 3.0 (wchar_t,compiler with C++ ABI 1013,wx containers,compatible with 2.8),
and wxPython used 3.0 (wchar_t,compiler with C++ ABI 1011,wx containers,compatible with 2.8).
/usr/lib64/python2.7/site-packages/wx-3.0-gtk3/wx/_core.py:16629: UserWarning: wxPython/wxWidgets release number mismatch
  warnings.warn("wxPython/wxWidgets release number mismatch")

$ d.rast basin
 100%

Since the original traceback has been resolved, I would suggest opening a new ticket.

comment:14 by neteler, 4 years ago

Milestone: → 7.8.3

comment:15 by neteler, 4 years ago

Does this still occur in 7.8.git?

comment:16 by marisn, 4 years ago

Resolution: → fixed
Status: new → closed

You are right — this issue has been fixed at some point. Tested with current master.

Note: See TracTickets for help on using tickets.