Opened 5 years ago

Closed 4 years ago

Last modified 4 years ago

#2634 closed defect (fixed)

'utf8' codec can't decode byte 0xf6 in position 8: invalid start byte

Reported by: hellik Owned by: grass-dev@…
Priority: critical Milestone: 7.0.1
Component: Default Version: svn-trunk
Keywords: utf, wxgui Cc:
CPU: x86-32 Platform: MSWindows 7

Description

taken from the ML

http://lists.osgeo.org/pipermail/grass-dev/2015-March/074529.html

GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 64858                                                       
Build date: 2015-03-15                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)
a ms windows error message window:

'utf8' codec can't decode byte 0xf6 in position 8: invalid start byte

<http://osgeo-org.1560.x6.nabble.com/file/n5193488/utf_codec_error_15032015_232313.png> 

pops up when invoking e.g.

d.vect
r.slope.aspect
...

and following error message:

Traceback (most recent call last):
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn\gui\wxpython\gu
i_core\forms.py", line 2054, in OnUpdateDialog

self.parent.updateValuesHook()
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn\gui\wxpython\gu
i_core\forms.py", line 628, in updateValuesHook

self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
TypeError

http://lists.osgeo.org/pipermail/grass-dev/2015-March/074540.html

interesting:

_no_ problems with commands like e.g.:

g.region
g.mapset
g.access
g.gisenv
g.extension

problems with commands like e.g.:

r.slope.aspect
d.vect
r.watershed
r.neigbhours
v.parallel
...

also tested on another machine:

System Info                                                                     
GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 64889                                                       
Build date: 2015-03-21                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-Vista-6.0.6002-SP2 (OSGeo4W)

the same error

Change History (24)

comment:1 Changed 4 years ago by hellik

Priority: majorcritical

comment:2 in reply to:  description ; Changed 4 years ago by hellik

Replying to hellik:

taken from the ML

http://lists.osgeo.org/pipermail/grass-dev/2015-March/074529.html

GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 64858                                                       
Build date: 2015-03-15                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)
a ms windows error message window:

'utf8' codec can't decode byte 0xf6 in position 8: invalid start byte

<http://osgeo-org.1560.x6.nabble.com/file/n5193488/utf_codec_error_15032015_232313.png> 

pops up when invoking e.g.

d.vect
r.slope.aspect
...

and following error message:

Traceback (most recent call last):
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn\gui\wxpython\gu
i_core\forms.py", line 2054, in OnUpdateDialog

self.parent.updateValuesHook()
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn\gui\wxpython\gu
i_core\forms.py", line 628, in updateValuesHook

self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
TypeError

http://lists.osgeo.org/pipermail/grass-dev/2015-March/074540.html

interesting:

_no_ problems with commands like e.g.:

g.region
g.mapset
g.access
g.gisenv
g.extension

problems with commands like e.g.:

r.slope.aspect
d.vect
r.watershed
r.neigbhours
v.parallel
...

also tested on another machine:

System Info                                                                     
GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 64889                                                       
Build date: 2015-03-21                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-Vista-6.0.6002-SP2 (OSGeo4W)

the same error

still valid for

GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 65460                                                       
Build date: 2015-06-14                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)  

makes winGRASS trunk quite unusable ... not there in winGRASS7.0.x ....

comment:3 in reply to:  2 Changed 4 years ago by hellik

Replying to hellik:

makes winGRASS trunk quite unusable ... not there in winGRASS7.0.x ....

tested on different german winVista/win7/win8-boxes ...

comment:4 Changed 4 years ago by hellik

Milestone: 7.1.07.0.1

comment:5 Changed 4 years ago by hellik

it's also now in the 7.0.1RC1, tested in the NC sample dataset

System Info                                                                     
GRASS Version: 7.0.1RC1                                                         
GRASS SVN Revision: 65472                                                       
Erstellungsdatum: 2015-06-15                                                    
Build Platform: i686-pc-mingw32                                                 
GDAL/OGR: 1.11.2                                                                
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)  

choosing d.rast from the wxgui

a MS error box pops up with

'utf8' codec can't decode byte 0xf6 in position 8: invalid start byte

followed by this message in the wxgui

Traceback (most recent call last):
  File "C:\OSGEO4~2\apps\grass\grass-7.0.1RC1\gui\wxpython\g
ui_core\forms.py", line 2023, in OnUpdateDialog

self.parent.updateValuesHook()
  File "C:\OSGEO4~2\apps\grass\grass-7.0.1RC1\gui\wxpython\g
ui_core\forms.py", line 628, in updateValuesHook

self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
TypeError

as mentioned in the original ticket above, e.g. g.region works without any problem

any ideas?

comment:6 in reply to:  5 ; Changed 4 years ago by hellik

Replying to hellik:

it's also now in the 7.0.1RC1, tested in the NC sample dataset

System Info                                                                     
GRASS Version: 7.0.1RC1                                                         
GRASS SVN Revision: 65472                                                       
Erstellungsdatum: 2015-06-15                                                    
Build Platform: i686-pc-mingw32                                                 
GDAL/OGR: 1.11.2                                                                
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)  

choosing d.rast from the wxgui

a MS error box pops up with

'utf8' codec can't decode byte 0xf6 in position 8: invalid start byte

followed by this message in the wxgui

Traceback (most recent call last):
  File "C:\OSGEO4~2\apps\grass\grass-7.0.1RC1\gui\wxpython\g
ui_core\forms.py", line 2023, in OnUpdateDialog

self.parent.updateValuesHook()
  File "C:\OSGEO4~2\apps\grass\grass-7.0.1RC1\gui\wxpython\g
ui_core\forms.py", line 628, in updateValuesHook

self.SetStatusText('
'.join(self.notebookpanel.createCmd(ignoreErrors = True)))
TypeError

as mentioned in the original ticket above, e.g. g.region works without any problem

any ideas?

language is set following:

GUI preferences -> language -> system

comment:7 in reply to:  6 ; Changed 4 years ago by hellik

Replying to hellik:

language is set following:

GUI preferences -> language -> system

ok, I switched the language to englisch:

GUI preferences -> language -> en

this error disappears.

maybe some error in the translation?

comment:8 in reply to:  7 Changed 4 years ago by hellik

Replying to hellik:

Replying to hellik:

language is set following:

GUI preferences -> language -> system

ok, I switched the language to englisch:

GUI preferences -> language -> en

this error disappears.

maybe some error in the translation?

system language is german

comment:9 in reply to:  7 ; Changed 4 years ago by marisn

Replying to hellik:

Replying to hellik:

language is set following:

GUI preferences -> language -> system

ok, I switched the language to englisch:

GUI preferences -> language -> en

this error disappears.

maybe some error in the translation?

Haven't tested, but it seems like the opposite. A string is translated but some python code expects it to be an ascii string. There have been similar issues already reported (see #2617 for longer discussion). Traceback indicates use of str method on unicode object in 'gui_core\forms.py' file. wxgui is full of such issues and in many places it is unknown if adding just a u' ' would not introduce problems elsewhere :( Backporting r64834 could help in some cases but an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

comment:10 in reply to:  9 ; Changed 4 years ago by martinl

Replying to marisn: adding just a u' ' would not introduce problems elsewhere :( Backporting r64834 could help in some cases but an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

it's already done in r65462

comment:11 in reply to:  10 ; Changed 4 years ago by martinl

Replying to martinl:

Replying to marisn: adding just a u' ' would not introduce problems elsewhere :( Backporting r64834 could help in some cases but an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

it's already done in r65462

Helmut, could you please test current 7.0.1svn if it still works (1)?

(1) http://wingrass.fsv.cvut.cz/grass70/

comment:12 in reply to:  11 ; Changed 4 years ago by hellik

Replying to martinl:

Replying to martinl:

Replying to marisn: adding just a u' ' would not introduce problems elsewhere :( Backporting r64834 could help in some cases but an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

it's already done in r65462

Helmut, could you please test current 7.0.1svn if it still works (1)?

(1) http://wingrass.fsv.cvut.cz/grass70/

tested with

GRASS Version: 7.0.1RC1                                                         
GRASS SVN Revision: 65472                                                       
Erstellungsdatum: 2015-06-15                                                    
Build Platform: i686-pc-mingw32                                                 
GDAL/OGR: 1.11.2                                                                
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W) 

manually and locally reverted r65462 => error disappears

problems with commands like e.g.:

r.slope.aspect
d.vect
r.watershed
r.neigbhours
v.parallel
...

all these modules are working without the reported error!

comment:13 in reply to:  12 Changed 4 years ago by hellik

Replying to hellik:

Replying to martinl:

Replying to martinl:

Replying to marisn: adding just a u' ' would not introduce problems elsewhere :( Backporting r64834 could help in some cases but an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

it's already done in r65462

Helmut, could you please test current 7.0.1svn if it still works (1)?

(1) http://wingrass.fsv.cvut.cz/grass70/

tested with

GRASS Version: 7.0.1RC1                                                         
GRASS SVN Revision: 65472                                                       
Erstellungsdatum: 2015-06-15                                                    
Build Platform: i686-pc-mingw32                                                 
GDAL/OGR: 1.11.2                                                                
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W) 

tested in german locale!

comment:14 in reply to:  12 ; Changed 4 years ago by martinl

Replying to hellik:

manually and locally reverted r65462 => error disappears

done in r65527.

comment:15 in reply to:  14 Changed 4 years ago by martinl

Replying to martinl:

Replying to hellik:

manually and locally reverted r65462 => error disappears

done in r65527.

I have reverted this commit also in r65528.

comment:16 in reply to:  14 Changed 4 years ago by martinl

Replying to martinl:

Replying to hellik:

manually and locally reverted r65462 => error disappears

done in r65527.

Please could you try the next build, no. >= 63 (1)

(1) http://wingrass.fsv.cvut.cz/grass70/

comment:17 Changed 4 years ago by annakrat

Have anyone tried to investigate where the utf8 error comes from? Perhaps r65462 is not the problem. I agree with reverting it for the release, but it should stay in trunk for more investigation.

comment:18 in reply to:  17 Changed 4 years ago by hellik

Replying to annakrat:

Have anyone tried to investigate where the utf8 error comes from? Perhaps r65462 is not the problem. I agree with reverting it for the release, but it should stay in trunk for more investigation.

g.gisenv set=DEBUG=3
C:\>r.slope.aspect
D1/3: G_set_program_name(): r.slope.aspect
D2/3: G_file_name(): path = C:\grassdata/gbifwgs84/chechdb
D3/3: G_recreate_command()
D3/3: win_spawn: args = C:\windows\system32\cmd.exe /c "C:\OSGEO4~2\bin\python.e
xe C:\OSGEO4~2\apps\grass\grass-7.1.svn/gui/wxpython/gui_core/forms.py r.slope.a
spect"
D1/3: G_set_program_name(): g.gisenv
D3/3: G_option_to_separator(): key = separator -> sep = '
'
D1/3: grass.script.core.start_command(): g.gisenv -n
D1/3: G_set_program_name(): g.gisenv
D3/3: G_option_to_separator(): key = separator -> sep = '
'
Traceback (most recent call last):
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn/gui/wxpython/gui_core/forms.py", li
ne 2545, in <module>
    Debug.msg(1, "forms.py opening form for: %s" % task.get_cmd(ignoreErrors=Tru
e))
  File "C:\OSGEO4~2\apps\grass\grass-7.1.svn\etc\python\grass\script\task.py", l
ine 235, in get_cmd
    cmd += [('%s=%s' % (p['name'], _('<required>'))).decode('utf-8')]
  File "C:\OSGEO4~2\apps\Python27\lib\encodings\utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf6 in position 14: invalid
start byte

C:\>

comment:19 Changed 4 years ago by annakrat

Please try this (with r64834 in place):

  • task.py

     
    232232                if p.get('default', '') != '':
    233233                    cmd += ['%s=%s' % (p['name'], p['default'])]
    234234                elif ignoreErrors and not suppress_required and not ignoreRequired:
    235                     cmd += [('%s=%s' % (p['name'], _('<required>'))).decode('utf-8')]
     235                    cmd += ['%s=%s' % (p['name'], _('<required>'))]
    236236            elif p.get('value', '') == '' and p.get('default', '') != '' and not ignoreDefault:
    237237                cmd += ['%s=%s' % (p['name'], p['default'])]
    238238            elif p.get('value', '') != '' and \

This was introduced in r61704 and seems to be not needed with r64834.

comment:20 in reply to:  19 ; Changed 4 years ago by hellik

Replying to annakrat:

Please try this (with r64834 in place):

  • task.py

     
    232232                if p.get('default', '') != '':
    233233                    cmd += ['%s=%s' % (p['name'], p['default'])]
    234234                elif ignoreErrors and not suppress_required and not ignoreRequired:
    235                     cmd += [('%s=%s' % (p['name'], _('<required>'))).decode('utf-8')]
     235                    cmd += ['%s=%s' % (p['name'], _('<required>'))]
    236236            elif p.get('value', '') == '' and p.get('default', '') != '' and not ignoreDefault:
    237237                cmd += ['%s=%s' % (p['name'], p['default'])]
    238238            elif p.get('value', '') != '' and \

This was introduced in r61704 and seems to be not needed with r64834.

tested locally with with r64834 in place: no error anymore.

so let's test with the next nightly build and go on to the next RC!

thanks!

comment:21 in reply to:  20 ; Changed 4 years ago by annakrat

Replying to hellik:

tested locally with with r64834 in place: no error anymore.

so let's test with the next nightly build and go on to the next RC!

thanks!

Committed to trunk in r65529 and backported to releasebranch in r65530.

comment:22 in reply to:  21 Changed 4 years ago by hellik

Replying to annakrat:

Replying to hellik:

tested locally with with r64834 in place: no error anymore.

so let's test with the next nightly build and go on to the next RC!

thanks!

Committed to trunk in r65529 and backported to releasebranch in r65530.

tested with

GRASS Version: 7.1.svn                                                          
GRASS SVN revision: 65532                                                       
Build date: 2015-07-02                                                          
Build platform: i686-pc-mingw32                                                 
GDAL: 1.11.2                                                                    
PROJ.4: 4.8.0                                                                   
GEOS: 3.4.2                                                                     
SQLite: 3.7.17                                                                  
Python: 2.7.4                                                                   
wxPython: 2.8.12.1                                                              
Platform: Windows-7-6.1.7601-SP1 (OSGeo4W)   

as it seems to work now, I'm closing ticket.

comment:23 Changed 4 years ago by hellik

Resolution: fixed
Status: newclosed

comment:24 in reply to:  10 Changed 4 years ago by glynn

Replying to martinl:

an overhaul of any string processing in wxgui is required with strict rules (that need to be created).

Adding support for Python 3 would probably help with that, as Python 3 doesn't perform implicit conversions between byte strings and Unicode strings (it raises a TypeError?).

Note: See TracTickets for help on using tickets.