wxGUI startup: 'ascii' codec can't decode byte 0xc3 in position 15: ordinal not in range(128)

[mlennert]

With a just freshly checked out trunk, and LANGUAGE=fr_BE, I get the following error when trying to launch the wxgui:

Impossible d'analyser le fichier XML des outils utilisateur. Chargement de l'arbre des modules par défaut.

Raison: 'ascii' codec can't decode byte 0xc3 in position 15: ordinal not in range(128)

Traceback (most recent call last):
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/lmgr/menudata.py", line 62, in __init__
    MenuTreeModelBuilder.__init__(self, filename, expandAddons=expandAddons)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/menutree.py", line 66, in __init__
    expAddons(xmlTree)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/toolboxes.py", line 346, in expandAddons
    _expandRuntimeModules(root, loadMetadata=True)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/toolboxes.py", line 596, in _expandRuntimeModules
    desc, keywords = _loadMetadata(name)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/toolboxes.py", line 631, in _loadMetadata
    e = _encode_string(e.value)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/toolboxes.py", line 114, in _encode_string
    return EncodeString(string)
  File "/data/home/mlennert/SRC/GRASS/grass_trunk/dist.x86_64-unknown-linux-gnu/gui/wxpython/core/gcmd.py", line 92, in EncodeString
    return string.encode(_enc)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 15: ordinal not in range(128)

We just cannot seem to get rid of these kinds of errors. I think this should be a top priority for work on the wxGUI in the near future as this is a severe usability issue for many people around the world.

[wenzeslaus]

Replying to mlennert:

With a just freshly checked out trunk

Do I understand correctly that this is not happening in 7.0 release branch?

We just cannot seem to get rid of these kinds of errors. I think this should be a top priority for work on the wxGUI in the near future as this is a severe usability issue for many people around the world.

It would be good if somebody who is actually running in this kind if environment would do these fixes.

[mlennert]

Replying to wenzeslaus:

Replying to mlennert:

With a just freshly checked out trunk

Do I understand correctly that this is not happening in 7.0 release branch?

Well, I just checked out release70 and there I get the same error (with the added info that this seems to be linked to the v.nnstat addon, but in the command line and it does not prevent the gui from opening:

v.nnstat: 'ascii' codec can't decode byte 0xc3 in position 15: ordinal not in range(128)

So maybe something in trunk that tries to handle addons differently ?

We just cannot seem to get rid of these kinds of errors. I think this should be a top priority for work on the wxGUI in the near future as this is a severe usability issue for many people around the world.

It would be good if somebody who is actually running in this kind if environment would do these fixes.

I'm willing to help within the limits of available time, but past discussions have left me in a state of perplexity concering these problems. AFAIU (and that's quite limited), there seem to be some fundamental problems with the handling of encoding issues in the GUI. Maris has tried to find some solutions and Glynn has weighed in with insights, but I don't think I quite grasp the whole thing.

It seems to me that we need to fundamentally rethink encoding handling, but that up to now encoding issues have been 'fixed' one by one without getting at the underlying weaknesses. But maybe I'm completely wrong in that understanding...

Moritz

[annakrat]

Please try r66036. Currently (quite recent change r64834) translatable strings return byte strings and we have to respect it in the code.

It's difficult to define any rules when things change all the time...

[mlennert]

Replying to annakrat:

Please try r66036.

Works, thanks !

Currently (quite recent change r64834) translatable strings return byte strings and we have to respect it in the code.

Yes. This change (and the ticket it relates to) clearly show one of the problems we currently have in grass development: with the growing importance of the wxGUI and of Python scripting, I have the feeling that choices at one point in the code have more and more impact on other parts of the code. Development at module level does not have this kind of impact and in my (perhaps erroneous) perception C-library changes were done by fewer people, possibly with better understanding of the consequences.

With Python opening development up to more people, and with the growing importance of the GUI, we might need a more centralised ruleset of how to do certain important things in order to ensure that we keep the code coherent.

It's difficult to define any rules when things change all the time...

That's why I've been pleading for a more fundamental reflection about how to approach encoding issues in the wxGUI in order to define one way to things once and for all without trying to solve this case-by-case. The trouble seems to be that most of us do not really fundamentally understand how string handling works and what different choices involve. But then again, maybe this is already solved... More importantly,I don't have the time at this stage to delve into the matter enough to be the motor of this, so I will shut up. ;-)

Moritz

[mlennert]

Can I close this ticket, as the specific bug is fixed, or does this need a backport ?

[annakrat]

Replying to mlennert:

Can I close this ticket, as the specific bug is fixed, or does this need a backport ?

Not sure yet, I have to test releasebranch, because there were some changes, reverts, and I can't do it right now.

[annakrat]

Replying to annakrat:

Replying to mlennert:

Can I close this ticket, as the specific bug is fixed, or does this need a backport ?

Not sure yet, I have to test releasebranch, because there were some changes, reverts, and I can't do it right now.

I fixed it there as well. I think we should test it on Windows before we close it.

[glynn]

Replying to mlennert:

It seems to me that we need to fundamentally rethink encoding handling, but that up to now encoding issues have been 'fixed' one by one without getting at the underlying weaknesses. But maybe I'm completely wrong in that understanding...

When issues arise due to unicode strings being passed to functions expecting byte string, the issue is normally fixed by adding an encode() call in the caller. But to catch other such cases, it would help to also modify the callee to explicitly check (e.g. via an assert statement) that the value is a byte string rather than a unicode string.

An explicit check will catch cases where the passed-in string happens to be convertible to ASCII. Relying upon Python raising a UnicodeError doesn't help much when most of the strings being passed around are convertible to ASCII.

Ideally, such checks should be added at the lowest point of the call chain, in order to catch all possible routes to the point where the error occurs.

[neteler]

Ticket retargeted after milestone closed

[Bas Couwenberg]

This issue has also been reported in ​Debian Bug #806257, that user also has a French locale but a different position in the error message:

'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

[mlennert]

Replying to sebastic:

This issue has also been reported in ​Debian Bug #806257, that user also has a French locale but a different position in the error message:

'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

I think this is more related to #2827 which has been fixed now.

[mlennert]

I don't get this error anymore, so for me this can be closed.

I haven't tested on Windows, though, but there have been no complaints, so let's close this and reopen if needed.

[mlennert]

Replying to mlennert:

Replying to sebastic:

This issue has also been reported in ​Debian Bug #806257, that user also has a French locale but a different position in the error message:

'ascii' codec can't decode byte 0xc3 in position 8: ordinal not in range(128)

I think this is more related to #2827 which has been fixed now.

Sorry, I meant #2826.