Opened 7 years ago

Closed 5 years ago

#1941 closed defect (fixed)

wxGUI fails with Japanese locale --> mixed Python installs on Windows

Reported by: venkat Owned by: grass-dev@…
Priority: major Milestone: 7.0.0
Component: wxGUI Version: svn-trunk
Keywords: wingrass Cc:
CPU: x86-32 Platform: MSWindows 7

Description (last modified by hamish)

I get following error with the wxGUI while running GRASS7-svn when selecting modules form the menu. on Windows-7 in Japanese locale. I guess it is related to utf-8 encoding.


Traceback (most recent call last):
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 737, in
OnMenuCmd

cmd = self.GetMenuCmd(event)
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\lmgr\frame.py", line 722, in
GetMenuCmd

input = GUI().GetCommandInputMapParamKey(cmdlist[0])
  File "C:\Program Files\GRASS GIS
7.0.svn\etc\gui\wxpython\gui_core\forms.py", line 2301, in
GetCommandInputMapParamKey

gtask.get_interface_description(cmd).decode(enc).encode('utf
-8')))
UnicodeDecodeError
:
'cp932' codec can't decode bytes in position 7074-7075:
illegal multibyte sequence

Attachments (4)

patch_for_ticket_1941.patch (3.3 KB) - added by improgrammer 6 years ago.
wxgui_error.png (117.4 KB) - added by venkat 6 years ago.
screen_shot_2.pdf (90.3 KB) - added by venkat 6 years ago.
import name MAXREPEAT error
missing_dll.png (93.6 KB) - added by Venkat 6 years ago.
HDF5 and MrSid? DLL missing

Download all attachments as: .zip

Change History (41)

comment:1 Changed 7 years ago by martinl

Summary: wxGUI errorwxGUI fails with Japanese locale

comment:3 Changed 7 years ago by venkat

This issue seems to have been fixed in GRASS-6.4 but I think still remains in GRASS-7.0.

comment:4 Changed 6 years ago by mlennert

Could this be related to #2021 ?

comment:5 in reply to:  4 Changed 6 years ago by glynn

Replying to mlennert:

Could this be related to #2021 ?

That seems unlikely, as this relates to Windows which doesn't use environment variables for locale settings.

Changed 6 years ago by improgrammer

Attachment: patch_for_ticket_1941.patch added

comment:6 Changed 6 years ago by improgrammer

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

  1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:
        enc = locale.getdefaultlocale()[1]
        if enc and enc.lower() == "cp932":
            p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
            tree = etree.fromstring(p.sub('encoding="utf-8"',
                                          get_interface_description(name).decode(enc).encode("utf-8")))
        else:
            tree = etree.fromstring(get_interface_description(name))
    
  2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
  3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

comment:7 in reply to:  6 ; Changed 6 years ago by mlennert

Replying to improgrammer:

I encounter similar problem.

I suggest modifying the sementics of gtask.get_interface_description() to return xml in "utf-8" encoding. see attachment "patch_for_ticket_1941.patch".

Rational:

  1. The following code segment in task.py and forms.py is too limited because ("cp936","cp932", etc) need similar converting:
        enc = locale.getdefaultlocale()[1]
        if enc and enc.lower() == "cp932":
            p = re.compile('encoding="' + enc + '"', re.IGNORECASE)
            tree = etree.fromstring(p.sub('encoding="utf-8"',
                                          get_interface_description(name).decode(enc).encode("utf-8")))
        else:
            tree = etree.fromstring(get_interface_description(name))
    
  2. Any where the gtask.get_interface_description() is called, the "convert-to-utf8" is needed;
  3. Properly constructed interface description text is xml text with proper <?xml ... encoding="some-text-encoding"?> statement, extract text encoding schema from these xml text is more rational than extract enc from locale package.

Can someone else working in a Japanese Windows environment test this patch ?

comment:8 in reply to:  7 ; Changed 6 years ago by neteler

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

comment:9 in reply to:  8 ; Changed 6 years ago by annakrat

Replying to neteler:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Proposal: Since it so much easier to update winGRASS via installer, I would suggest that we submit the patch to SVN and test from there. In the worst case we revert after 24hs/whatever time.

Done in r57501. Please test!

comment:10 in reply to:  9 Changed 6 years ago by neteler

Replying to annakrat:

Replying to mlennert: ...

Can someone else working in a Japanese Windows environment test this patch ?

Done in r57501. Please test!

The related winGRASS 7 binary is now available here: http://wingrass.fsv.cvut.cz/grass70/

as well as in OSGEo4W.

Changed 6 years ago by venkat

Attachment: wxgui_error.png added

comment:11 Changed 6 years ago by venkat

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError?: No module named gis when I try to start the wxGUI. Details in the attachment.

Venka

comment:12 in reply to:  11 ; Changed 6 years ago by annakrat

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError?: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

comment:13 in reply to:  12 Changed 6 years ago by mlennert

Replying to annakrat:

Replying to venkat:

tried with WinGRASS-7.0.svn-r57518-685-Setup.exe from http://wingrass.fsv.cvut.cz/grass70/ but this gives me

ImportError?: No module named gis when I try to start the wxGUI. Details in the attachment.

You seem to have a problem with ctypes. I moved importing temporal gis library (which apparently uses ctypes) so that wxGUI should at least start.

Venka,

Have you been able to test these last changes ?

Moritz

Changed 6 years ago by venkat

Attachment: screen_shot_2.pdf added

import name MAXREPEAT error

comment:14 Changed 6 years ago by venkat

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Thanks

Venka

comment:15 in reply to:  14 ; Changed 6 years ago by annakrat

Replying to venkat:

Tried with WinGRASS-7.0.svn-r57622-697-Setup.exe and get errors as shown in the attachment.

Please try the OSGeo4W installer, you shouldn't get this error there.

Anybody has an idea how to fix this annoying MAXREPEAT problem?

comment:16 in reply to:  15 ; Changed 6 years ago by neteler

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

comment:17 in reply to:  16 ; Changed 6 years ago by martinl

Replying to neteler:

Replying to annakrat:

Anybody has an idea how to fix this annoying MAXREPEAT problem?

Did you try this?

http://grasswiki.osgeo.org/wiki/WinGRASS_errors#Error_message_.22ImportError:_cannot_import_name_MAXREPEAT.22

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

comment:18 in reply to:  17 ; Changed 6 years ago by glynn

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the "system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

DLL Search Order.

comment:19 in reply to:  18 ; Changed 6 years ago by martinl

Replying to glynn:

Replying to martinl:

Usually it's caused by Python installed together with Esri ArcGIS. The newest version of ArcGIS 10.2 still comes with Python 2.7.3. It would be good to find solution on our side without overwriting system-installed Python.

2.7.3 should be fine for GRASS, right? So the solution should be to detect that the

No, the problem still remains. Note that we have this problem only with standalone installer not with osgeo4w installer.

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

AFAICT, the issue is that the system python27.dll (from Windows/System32 or Windows/SysWOW64) is being used, but PYTHONHOME points to the bundled version of the Python standard library.

If you can start a Python shell using the GRASS Python, you can use Process Explorer to view the list of DLLs which are actually being used by that process. AFAIK, System32/SysWOW64 take precedence over everything except the directory containing the exe, and %PATH% comes last.

Strangely C:\Program Files (x86)\GRASS GIS 7.0.svn\extrabin\python.exe launches python 2.7.3 (same as system python from C:\Python27\ArcGIS 10.2)! When launching C:\osgeo4w\bin\python.exe I got expected version of python, so 2.7.4 (current osgeo4w python version). This partly explains the fact that we have this problem only with standalone installer.

DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

comment:20 Changed 6 years ago by mlennert

I think that the Python launcher might be the best solution for this. See the experiments with different combinations of Python versions at [1].

[1] http://lists.osgeo.org/pipermail/grass-dev/2013-July/065197.html

comment:21 in reply to:  19 ; Changed 6 years ago by martinl

Replying to martinl:

DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

comment:22 in reply to:  21 ; Changed 6 years ago by mlennert

Replying to martinl:

Replying to martinl:

DLL Search Order.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

Try it, but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

comment:23 in reply to:  21 Changed 6 years ago by hellik

Replying to martinl:

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory.

that could be the clue. same directory could work.

I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

also no idea or memory here on my side. since I'm involved in the winGRASS, the separation in extrabin and extralib was always there.

comment:24 in reply to:  19 ; Changed 6 years ago by glynn

Replying to martinl:

"system" Python is sufficient and not try to use the bundled Python (i.e. don't set PYTHONHOME, set GRASS_PYTHON to point to the system version, etc).

But then we will have problem with other python packages (GRASS dependecies) which are installed through osgeo4w environment as python-matplotlib or python-numpy, right? Version of "system" python will be most probably different from the python version available in osgeo4w environment.

Add-on packages don't normally care about the minor version, so a package built for 2.7 should work with 2.7.<anything>.

This doesn't apply to sre_*.py; modules which are part of the base Python distribution don't expect to be used with anything other than the exact Python version of which they are part.

Right, when I launch python.exe which comes with standalone installer it points to C:Windows\SysWOW64\python27.dll. When I launch python.exe from C:\OSGeo4W\bin it points to the right dll file, ie. C:\OSGeo4W\bin\python27.dll.

That will be because the OSGeo4W version has its python27.dll in the same directory as python.exe. The directory containing the EXE comes first, before the system directories; %PATH% comes last.

comment:25 Changed 6 years ago by hamish

Description: modified (diff)
Summary: wxGUI fails with Japanese localewxGUI fails with Japanese locale --> mixed Python installs on Windows

comment:26 in reply to:  24 ; Changed 6 years ago by hamish

Replying to glynn:

Add-on packages don't normally care about the minor version, so a package built for 2.7 should work with 2.7.<anything>.

As seen in the pdf screenshot, the trouble comes with "import re", which I think(??) is a built-in.

We recently saw the same trouble here with 'import re' -> no MAXREPEAT and 6.4.3 on a Windows 7 laptop, where there was a slightly older python version of 2.7(.2?) installed to C:\Python, which I guessed came with a separate OSGeo4W install(?). For us it was a new laptop not doing anything else so we could uninstall the non-grass Python from the control panel and then all was good.

Are we prepending to the PYTHONPATH instead of adding the grass paths to the end of it? (hopefully the GRASS versions get picked up first then) AFAIK env.bat & friends in the stand-alone installer now explicitly set GRASS_PYTHON with with the full path to grass's python.exe.

Hamish

comment:27 in reply to:  26 Changed 6 years ago by glynn

Replying to hamish:

As seen in the pdf screenshot, the trouble comes with "import re", which I think(??) is a built-in.

re.py and sre_*.py are part of the standard library. The error comes because MAXREPEAT is expected to be in the binary module "_sre", which is built into python27.dll. This is getting loaded from the Windows/System? or SysWOW64 directory, hence the problem.

Are we prepending to the PYTHONPATH instead of adding the grass paths to the end of it? (hopefully the GRASS versions get picked up first then) AFAIK env.bat & friends in the stand-alone installer now explicitly set GRASS_PYTHON with with the full path to grass's python.exe.

I don't know whether imports from the standard library use PYTHONPATH or PYTHONHOME (and neither of those will affect binary modules built into python27.dll).

comment:28 in reply to:  22 ; Changed 6 years ago by martinl

Replying to mlennert:

Ah, OK, it's probably caused by the fact that the standalone installer puts python27.dll to the different directory (ie. extralib) compared to python.exe (extrabin). Directory extralib is in the path source:grass/trunk/mswindows/env.bat#L17, but it's probably not enough. The problem could be solved by moving python27.dll to extrabin. Then we could probably move all DLL files to extrabin and get rid of this directory. It's comment practice to have DLL files together with EXE files in the same directory. I don't know why the standalone installer separates them to the two different directories, ie. extrabin and extralib.

Try it,

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

comment:29 in reply to:  28 ; Changed 6 years ago by hellik

Replying to martinl:

Try it,

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

I can't find any registry entry for calling python dlls here on my win7-box. isn't it an issue with file extension-association?

Helmut

comment:30 in reply to:  29 Changed 6 years ago by wenzeslaus

Replying to hellik:

Replying to martinl:

but ISTR that there is also an issue with the Windows registry, i.e. that the dlls called depend on registry entries (I think that this came up as a problem at least in the case of python scripts called from python scripts).

Right, it's open issue.

I can't find any registry entry for calling python dlls here on my win7-box. isn't it an issue with file extension-association?

For the file association, we have even a separate ticket #2015. But the comments deal with other problems (actually, I don't know if they are related or not) and my original problem was never confirmed (although we had it one more than one computer).

comment:31 in reply to:  29 ; Changed 6 years ago by martinl

Replying to hellik:

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

patch applied in all active branches, now GRASS should start without 'MAX_REPEAT' problem.

comment:32 in reply to:  31 Changed 6 years ago by annakrat

Replying to martinl:

Replying to hellik:

it helped, GRASS now starts, python scripts called from command line also starts. My suggestion is to apply patches r57639 and r57646 also in GRASS 6.x, ie. to remove extralib directory from standalone installer. Any objections?

+1 for backport

patch applied in all active branches, now GRASS should start without 'MAX_REPEAT' problem.

Thanks a lot, there is a ticket already discussing this problem (#2039) so let's move the discussion there and leave this for Japanese locale.

Venka, can you please test it now again when the unrelated MAXREPEAT problem should be fixed?

Changed 6 years ago by Venkat

Attachment: missing_dll.png added

HDF5 and MrSid? DLL missing

comment:33 Changed 6 years ago by Venkat

Priority: blockermajor

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid? DLL[1]. Could be better if gdal is compiled without HDF5 and MrSid? support.

[1]http://trac.osgeo.org/grass/attachment/ticket/1941/missing_dll.png

comment:34 in reply to:  33 ; Changed 6 years ago by wenzeslaus

Replying to Venkat:

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid DLL. Could be better if gdal is compiled without HDF5 and MrSid support.

attachment:missing_dll.png

Is this still a problem? If yes I suggest you to create a new ticket for this with the all details and close this ticket where all errors were solved or are in different tickets.

comment:35 in reply to:  34 ; Changed 6 years ago by Venkat

Replying to wenzeslaus:

Replying to Venkat:

MAXREPEAT issue is fixed and GRASS-7 starts after complaining about missing HDF5 and MrSid DLL. Could be better if gdal is compiled without HDF5 and MrSid support.

attachment:missing_dll.png

Is this still a problem? If yes I suggest you to create a new ticket for this with the all details and close this ticket where all errors were solved or are in different tickets.

GRASS 7 starts after complaining about not finding HDF5 and MrSID dll's. The warnings are in the image below that was posted earlier. https://trac.osgeo.org/grass/attachment/ticket/1941/missing_dll.png

Best

Venka

comment:36 in reply to:  35 Changed 6 years ago by neteler

The "Error 127: the specified procedure could not be found"

may have as reason DLL mixup. Please

comment:37 Changed 5 years ago by annakrat

Resolution: fixed
Status: newclosed

This ticket doesn't make any sense now. Please open a new ticket for the current problems if necessary. The original problem seems to be fixed.

Note: See TracTickets for help on using tickets.