wxGUI fails with Japanese locale --> mixed Python installs on Windows

[venkat]

I get following error with the wxGUI while running GRASS7-svn when selecting modules form the menu. on Windows-7 in Japanese locale. I guess it is related to utf-8 encoding.

---

Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 737, in
OnMenuCmd

cmd = self.GetMenuCmd(event)
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 722, in
GetMenuCmd

input = GUI().GetCommandInputMapParamKey(cmdlist[0])
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\gui_core\forms.py", line 2301, in
GetCommandInputMapParamKey

gtask.get_interface_description(cmd).decode(enc).encode('utf
-8')))
UnicodeDecodeError
:
'cp932' codec can't decode bytes in position 7074-7075:
illegal multibyte sequence

[neteler]

Two tickets related to cp932:

• ​http://trac.osgeo.org/osgeo4w/ticket/294#comment:4
• ​http://trac.osgeo.org/grass/ticket/1193#comment:2

[venkat]

This issue seems to have been fixed in GRASS-6.4 but I think still remains in GRASS-7.0.

[mlennert]

Could this be related to #2021 ?

[glynn]

Replying to mlennert:

Could this be related to #2021 ?

That seems unlikely, as this relates to Windows which doesn't use environment variables for locale settings.

[improgrammer]

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:

       enc = locale.getdefaultlocale()[1]
       if enc and enc.lower() == "cp932":
           p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
           tree = etree.fromstring(p.sub('encoding="utf-8"',
                                         get_interface_description(name).decode(enc).encode("utf-8")))
       else:
           tree = etree.fromstring(get_interface_description(name))
   

2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

[mlennert]

Replying to improgrammer:

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:

       enc = locale.getdefaultlocale()[1]
       if enc and enc.lower() == "cp932":
           p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
           tree = etree.fromstring(p.sub('encoding="utf-8"',
                                         get_interface_description(name).decode(enc).encode("utf-8")))
       else:
           tree = etree.fromstring(get_interface_description(name))
   

2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

Can someone else working in a Japanese Windows environment test this patch ?

[neteler]

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

[annakrat]

Replying to neteler:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

Done in r57501. Please test!

[neteler]

Replying to annakrat:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Done in r57501. Please test!

The related winGRASS 7 binary is now available here: ​http://wingrass.fsv.cvut.cz/grass70/

as well as in OSGEo4W.

[venkat]

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

Venka

[annakrat]

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

[mlennert]

Replying to annakrat:

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

Venka,

Have you been able to test these last changes ?

Moritz

[venkat]

import name MAXREPEAT error

[venkat]

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Thanks

Venka

[annakrat]

Replying to venkat:

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Please try the OSGeo4W installer, you shouldn't get this error there.

Anybody has an idea how to fix this annoying MAXREPEAT problem?

[neteler]

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

​http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

[martinl]

Replying to neteler:

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

​http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

[glynn]

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the "system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use ​Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

​DLL Search Order.

[martinl]

Replying to glynn:

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the

No, the problem still remains. Note that we have this problem only with standalone installer not with osgeo4w installer.

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use ​Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

Strangely C:\Program Files (x86)\GRASS GIS 7.0.svn\extrabin\python.exe launches python 2.7.3 (same as system python from C:\Python27\ArcGIS 10.2)! When launching C:\osgeo4w\bin\python.exe I got expected version of python, so 2.7.4 (current osgeo4w python version). This partly explains the fact that we have this problem only with standalone installer.

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

[mlennert]

I think that the Python launcher might be the best solution for this. See the experiments with different combinations of Python versions at [1].

[1] ​http://lists.osgeo.org/pipermail/grass-dev/2013-July/065197.html

[martinl]

Replying to martinl:

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

[mlennert]

Replying to martinl:

Replying to martinl:

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

Try it, but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

[hellik]

Replying to martinl:

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory.

that could be the clue. same directory could work.

I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

also no idea or memory here on my side. since I'm involved in the winGRASS, the separation in extrabin and extralib was always there.

[glynn]

Replying to martinl:

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

Add-on packages don't normally care about the minor version, so a package built for 2.7 should work with 2.7.<anything>.

This doesn't apply to sre_*.py; modules which are part of the base Python distribution don't expect to be used with anything other than the exact Python version of which they are part.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

That will be because the OSGeo4W version has its python27.dll in the same directory as python.exe. The directory containing the EXE comes first, before the system directories; %PATH% comes last.