wxGUI fails with Japanese locale --> mixed Python installs on Windows

[venkat]

I get following error with the wxGUI while running GRASS7-svn when selecting modules form the menu. on Windows-7 in Japanese locale. I guess it is related to utf-8 encoding.

---

Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 737, in
OnMenuCmd

cmd = self.GetMenuCmd(event)
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 722, in
GetMenuCmd

input = GUI().GetCommandInputMapParamKey(cmdlist[0])
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\gui_core\forms.py", line 2301, in
GetCommandInputMapParamKey

gtask.get_interface_description(cmd).decode(enc).encode('utf
-8')))
UnicodeDecodeError
:
'cp932' codec can't decode bytes in position 7074-7075:
illegal multibyte sequence

[neteler]

Two tickets related to cp932:

• ​http://trac.osgeo.org/osgeo4w/ticket/294#comment:4
• ​http://trac.osgeo.org/grass/ticket/1193#comment:2

[venkat]

This issue seems to have been fixed in GRASS-6.4 but I think still remains in GRASS-7.0.

[mlennert]

Could this be related to #2021 ?

[glynn]

Replying to mlennert:

Could this be related to #2021 ?

That seems unlikely, as this relates to Windows which doesn't use environment variables for locale settings.

[improgrammer]

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:

       enc = locale.getdefaultlocale()[1]
       if enc and enc.lower() == "cp932":
           p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
           tree = etree.fromstring(p.sub('encoding="utf-8"',
                                         get_interface_description(name).decode(enc).encode("utf-8")))
       else:
           tree = etree.fromstring(get_interface_description(name))
   

2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

[mlennert]

Replying to improgrammer:

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:

       enc = locale.getdefaultlocale()[1]
       if enc and enc.lower() == "cp932":
           p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
           tree = etree.fromstring(p.sub('encoding="utf-8"',
                                         get_interface_description(name).decode(enc).encode("utf-8")))
       else:
           tree = etree.fromstring(get_interface_description(name))
   

2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

Can someone else working in a Japanese Windows environment test this patch ?

[neteler]

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

[annakrat]

Replying to neteler:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

Done in r57501. Please test!

[neteler]

Replying to annakrat:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Done in r57501. Please test!

The related winGRASS 7 binary is now available here: ​http://wingrass.fsv.cvut.cz/grass70/

as well as in OSGEo4W.

[venkat]

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

Venka

[annakrat]

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

[mlennert]

Replying to annakrat:

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from ​http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

Venka,

Have you been able to test these last changes ?

Moritz

[venkat]

import name MAXREPEAT error

[venkat]

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Thanks

Venka

[annakrat]

Replying to venkat:

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Please try the OSGeo4W installer, you shouldn't get this error there.

Anybody has an idea how to fix this annoying MAXREPEAT problem?

[neteler]

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

​http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

[martinl]

Replying to neteler:

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

​http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

[glynn]

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the "system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use ​Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

​DLL Search Order.

[martinl]

Replying to glynn:

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the

No, the problem still remains. Note that we have this problem only with standalone installer not with osgeo4w installer.

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use ​Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

Strangely C:\Program Files (x86)\GRASS GIS 7.0.svn\extrabin\python.exe launches python 2.7.3 (same as system python from C:\Python27\ArcGIS 10.2)! When launching C:\osgeo4w\bin\python.exe I got expected version of python, so 2.7.4 (current osgeo4w python version). This partly explains the fact that we have this problem only with standalone installer.

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

[mlennert]

I think that the Python launcher might be the best solution for this. See the experiments with different combinations of Python versions at [1].

[1] ​http://lists.osgeo.org/pipermail/grass-dev/2013-July/065197.html

[martinl]

Replying to martinl:

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

[mlennert]

Replying to martinl:

Replying to martinl:

​DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

Try it, but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

[hellik]

Replying to martinl:

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory.

that could be the clue. same directory could work.

I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

also no idea or memory here on my side. since I'm involved in the winGRASS, the separation in extrabin and extralib was always there.

[glynn]

Replying to martinl:

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

Add-on packages don't normally care about the minor version, so a package built for 2.7 should work with 2.7.<anything>.

This doesn't apply to sre_*.py; modules which are part of the base Python distribution don't expect to be used with anything other than the exact Python version of which they are part.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

That will be because the OSGeo4W version has its python27.dll in the same directory as python.exe. The directory containing the EXE comes first, before the system directories; %PATH% comes last.

[hamish]

Replying to glynn:

Add-on packages don't normally care about the minor version, so a package built for 2.7 should work with 2.7.<anything>.

As seen in the pdf screenshot, the trouble comes with "import re", which I think(??) is a built-in.

We recently saw the same trouble here with 'import re' -> no MAXREPEAT and 6.4.3 on a Windows 7 laptop, where there was a slightly older python version of 2.7(.2?) installed to C:\Python, which I guessed came with a separate OSGeo4W install(?). For us it was a new laptop not doing anything else so we could uninstall the non-grass Python from the control panel and then all was good.

Are we prepending to the PYTHONPATH instead of adding the grass paths to the end of it? (hopefully the GRASS versions get picked up first then) AFAIK env.bat & friends in the stand-alone installer now explicitly set GRASS_PYTHON with with the full path to grass's python.exe.

Hamish

[glynn]

Replying to hamish:

As seen in the pdf screenshot, the trouble comes with "import re", which I think(??) is a built-in.

re.py and sre_*.py are part of the standard library. The error comes because MAXREPEAT is expected to be in the binary module "_sre", which is built into python27.dll. This is getting loaded from the Windows/System or SysWOW64 directory, hence the problem.

Are we prepending to the PYTHONPATH instead of adding the grass paths to the end of it? (hopefully the GRASS versions get picked up first then) AFAIK env.bat & friends in the stand-alone installer now explicitly set GRASS_PYTHON with with the full path to grass's python.exe.

I don't know whether imports from the standard library use PYTHONPATH or PYTHONHOME (and neither of those will affect binary modules built into python27.dll).

[martinl]

Replying to mlennert:

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17​, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

Try it,

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

[hellik]

Replying to martinl:

Try it,

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

I can't find any registry entry for calling python dlls here on my win7-box. isn't it an issue with file extension-association?

Helmut

[wenzeslaus]

Replying to hellik:

Replying to martinl:

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

I can't find any registry entry for calling python dlls here on my win7-box. isn't it an issue with file extension-association?

For the file association, we have even a separate ticket #2015. But the comments deal with other problems (actually, I don't know if they are related or not) and my original problem was never confirmed (although we had it one more than one computer).

[martinl]

Replying to hellik:

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

patch applied in all active branches, now GRASS should start without 'MAX_REPEAT' problem.

[annakrat]

Replying to martinl:

Replying to hellik:

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

patch applied in all active branches, now GRASS should start without 'MAX_REPEAT' problem.

Thanks a lot, there is a ticket already discussing this problem (#2039) so let's move the discussion there and leave this for Japanese locale.

Venka, can you please test it now again when the unrelated MAXREPEAT problem should be fixed?

[Venkat]

HDF5 and MrSid DLL missing

[Venkat]

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid DLL[1]. Could be better if gdal is compiled without HDF5 and MrSid support.

[1]​http://trac.osgeo.org/grass/attachment/ticket/1941/missing_dll.png

[wenzeslaus]

Replying to Venkat:

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid DLL. Could be better if gdal is compiled without HDF5 and MrSid support.

attachment:missing_dll.png​

Is this still a problem? If yes I suggest you to create a new ticket for this with the all details and close this ticket where all errors were solved or are in different tickets.

[Venkat]

Replying to wenzeslaus:

Replying to Venkat:

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid DLL. Could be better if gdal is compiled without HDF5 and MrSid support.

attachment:missing_dll.png​

Is this still a problem? If yes I suggest you to create a new ticket for this with the all details and close this ticket where all errors were solved or are in different tickets.

GRASS 7 starts after complaining about not finding HDF5 and MrSID dll's. The warnings are in the image below that was posted earlier. ​https://trac.osgeo.org/grass/attachment/ticket/1941/missing_dll.png

Best

Venka

[neteler]

The "Error 127: the specified procedure could not be found"

may have as reason DLL mixup. Please

• check if you have multiple versions of the same DLL in different folders,
• in case see: ​http://msdn.microsoft.com/en-us/library/ms682586%28VS.85%29.aspx
• see for a similar ticket: ​http://trac.osgeo.org/gdal/ticket/5110

[annakrat]

This ticket doesn't make any sense now. Please open a new ticket for the current problems if necessary. The original problem seems to be fixed.