Opened 13 years ago

Closed 7 years ago

# WinGRASS: $GISBASE/etc/gui/scripts/ require something like$(EXE) to run

Reported by: Owned by: hamish grass-dev@… blocker 7.0.0 wxGUI svn-trunk wingrass Unspecified MSWindows XP

### Description

Hi,

None of the GUI wrapper scripts called from the GUI menus work in WinGRASS due to lack of associated file extensions.

Hamish

### comment:10 follow-up:  11 Changed 12 years ago by cmbarton

So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line.

Michael

### comment:11 in reply to:  10 ; follow-up:  12 Changed 12 years ago by glynn

So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line.

On Windows, 6.4's g.parser explicitly invokes the script via $GRASS_SH. Obviously this won't work for Python scripts. FWIW, Python scripts work in Windows in 7.0, whether the script is in the path or not (once PYTHONPATH has been set; Init.bat doesn't set this). The biggest difference between the two is that in 7.0, the grass.script.parser() Python function uses "g.parser -s", which eliminates the need to re-execute the script. It may be worth back-porting this feature to 6.x (the existing behaviour is still available for use by shell scripts). ### comment:12 in reply to: 11 ; follow-up: 13 Changed 12 years ago by cmbarton Replying to glynn: Replying to cmbarton: So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line. On Windows, 6.4's g.parser explicitly invokes the script via$GRASS_SH. Obviously this won't work for Python scripts.

Does this mean that it is only expecting shell scripts in Windows? What happens on other platforms (since Python does run without problems in Linux and on Mac)?

FWIW, Python scripts work in Windows in 7.0, whether the script is in the path or not (once PYTHONPATH has been set; Init.bat doesn't set this).

The biggest difference between the two is that in 7.0, the grass.script.parser() Python function uses "g.parser -s", which eliminates the need to re-execute the script. It may be worth back-porting this feature to 6.x (the existing behaviour is still available for use by shell scripts).

Seems like that would be a very good idea, if it means that you can run something besides shell scripts in a Windows environment.

Michael

### comment:13 in reply to:  12 Changed 12 years ago by glynn

On Windows, 6.4's g.parser explicitly invokes the script via $GRASS_SH. Obviously this won't work for Python scripts. Does this mean that it is only expecting shell scripts in Windows? Yes. Windows doesn't understand shebang lines, so g.parser can't "run" the shell scripts as if they were executables; it has to use "$GRASS_SH <script>".

It might be possible to use "$GRASS_SH -c <script>", as MSys' bash understands shebang lines. OTOH, for "#!/usr/bin/env python", this will only work if there is an executable named "python" in the path (if it's called e.g. "python25" or it isn't in the path, you lose). Alternatively, the scripts could be installed with a ".sh" extension, relying upon the file associations to run such scripts via bash. Or you could make g.parser examine the first line of the script and choose a suitable interpreter (it isn't as simple as processing a shebang on Unix as we don't know where MSys' virtual filesystem is located and we can't assume that the "#!/usr/bin/env" trick will work with "native" interpreters). What happens on other platforms (since Python does run without problems in Linux and on Mac)? The code to re-invoke the script is conditionalised upon "#ifdef __MINGW32__". Unix (including MacOSX) simply execl()s the script directly, while the Windows version invokes it via$GRASS_SH.

### comment:14 follow-up:  17 Changed 12 years ago by hamish

Hello,

update for this bug report based on the latest native wingrass installer (r40049) on XP:

works:

• wxGUI File->"NVIZ (req's tcltk)"
• "nviz -q" from the wxGUI layer manager 'Cmd> ' prompt
• wxGUI Vector->Develop->"Digitize (req's tcltk)"

(excellent!)

does not work:

• wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.)
• wxGUI Vector->Develop->Change object types (Traceback error in command output tab ... Unable to fetch interface description for command 'v.type_wrapper.py'.)
• from the Cmd> prompt:
• g.change.gui (nothing happens),
• g.change.gui.py (Unable to fetch interface desc),
• g.change.gui.py --help (Command finished 0 sec),
• g.change.gui.py --interface-description (Command finished 0 sec)

any suggestions?

Hamish

### comment:15 follow-up:  16 Changed 12 years ago by cmbarton

Why can't windows use g.gui gui=[guiplatform] -nu?

In fact, even though both g.change.gui and v.type_wrapper are in my source, they are not getting into my OS X binary. Perhaps they are not getting into the Windows binary either?

There is a problem running python scripts in GRASS for Windows. Ironic in that one reason for Python is that it is cross-platform. From recent discussions on this (Glynn, Colin, me, others), it seems like the best solution will be to NOT package Python with GRASS (wxPython may be a different issue), but to have an installer that checks for a Python installation and if not present directs the user to python.org to install it. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

Michael

### comment:16 in reply to:  15 ; follow-ups:  19  20 Changed 12 years ago by cnielsen

Why can't windows use g.gui gui=[guiplatform] -nu?

This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem.

In fact, even though both g.change.gui and v.type_wrapper are in my source, they are not getting into my OS X binary. Perhaps they are not getting into the Windows binary either?

These are present in the windows installations in etc\gui\scripts. Well, in fact there is g.change.gui.py g.change.gui.sh g.change.gui.sh.bat but no g.change.gui

There is a problem running python scripts in GRASS for Windows. Ironic in that one reason for Python is that it is cross-platform. From recent discussions on this (Glynn, Colin, me, others), it seems like the best solution will be to NOT package Python with GRASS (wxPython may be a different issue), but to have an installer that checks for a Python installation and if not present directs the user to python.org to install it. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

I've been working on an update to the NSIS installer that will check to see if Python is installed via the registry, and if not auto-download and install the four necessary packages (python, wxpython, pywin32, and numpy). The installer already does something similar for the sample datasets.

However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not? Where do pywin32 and numpy fit in? Can build specific parts only be included in the installer and still use an external Python?

I can make the installer, but I need these details to know how to assemble it.

-Colin

### comment:17 in reply to:  14 ; follow-up:  18 Changed 12 years ago by hellik

does not work:

• wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.)
• wxGUI Vector->Develop->Change object types (Traceback error in command output tab ... Unable to fetch interface description for command 'v.type_wrapper.py'.)
• from the Cmd> prompt:
• g.change.gui (nothing happens),
• g.change.gui.py (Unable to fetch interface desc),
• g.change.gui.py --help (Command finished 0 sec),
• g.change.gui.py --interface-description (Command finished 0 sec)

a quick look at the manual:

there seems to be no g.change.gui(.py)

g.gui is working on windows

Helmut

### comment:18 in reply to:  17 ; follow-up:  21 Changed 12 years ago by hamish

does not work:

• wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.)

there seems to be no g.change.gui(.py)

sure there is. look at the title of this bug report.

The source is in gui/scripts/, it gets installed to $GISBASE/etc/gui/scripts/. All the wrapper scripts in there fail on wingrass. These wrapper scripts are needed to make command line based modules more GUI dialog friendly, or to extract particular targted functionality. the above quoted menu path shows where it is in the wx menu. g.gui is working on windows that is something different (a normal C module, not a python wrapper script). Hamish ### comment:19 in reply to: 16 ; follow-up: 24 Changed 12 years ago by hamish Replying to cmbarton: Why can't windows use g.gui gui=[guiplatform] -nu? Replying to cnielsen: This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem. • this is hardly as user friendly as a GUI which asks: "use Wx", "use Tcl", or "use text" & restart. these wrapper scripts are not designed for power users who are comfortable with the command line. • the default osgeo4w build does not give the user a msys prompt, although I guess you could try the Cmd> prompt if you knew the exact magic incantation. • v.type from a GUI is impossible and needs a wrapper. There is a problem running python scripts in GRASS for Windows. it's just a bug, and not a fundamental problem. mere details can and will be fixed. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts. we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason. Hamish ### comment:20 in reply to: 16 ; follow-up: 23 Changed 12 years ago by glynn Replying to cnielsen: However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not? The vdigit and nviz modules are compiled for a specific version of Python, and vdigit for a specific version of wxPython. But at present, neither of those modules compile on Windows. Where do pywin32 and numpy fit in? The GUI needs numpy and pywin32, but those packages only need to be compatible with the system Python, not with any specific version, nor with wxPython. Can build specific parts only be included in the installer and still use an external Python? If Python code imports a binary module, it must be running under the version of Python/wxPython for which the binary module was compiled. If we were to supply the vdigit and nviz modules in the Windows installer, the wxGUI would need to be run using the version of Python/wxPython for which those modules were built. ### comment:21 in reply to: 18 ; follow-ups: 22 28 Changed 12 years ago by glynn Replying to hamish: All the wrapper scripts in there fail on wingrass. My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True. [shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.] lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar. ### comment:22 in reply to: 21 ; follow-up: 27 Changed 12 years ago by cmbarton Replying to glynn: Replying to hamish: All the wrapper scripts in there fail on wingrass. My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True. On Mac we only need to supply the wxPython binaries to ensure that the correct version is run I think. But maybe william will know better. [shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.] lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar. ### comment:23 in reply to: 20 Changed 12 years ago by cmbarton Replying to glynn: Replying to cnielsen: However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not? The vdigit and nviz modules are compiled for a specific version of Python, and vdigit for a specific version of wxPython. But at present, neither of those modules compile on Windows. Where do pywin32 and numpy fit in? The GUI needs numpy and pywin32, but those packages only need to be compatible with the system Python, not with any specific version, nor with wxPython. Can build specific parts only be included in the installer and still use an external Python? If Python code imports a binary module, it must be running under the version of Python/wxPython for which the binary module was compiled. If we were to supply the vdigit and nviz modules in the Windows installer, the wxGUI would need to be run using the version of Python/wxPython for which those modules were built. Suggestion for building binaries: Build with a a Python version (say 2.5.x), supply the specific wxPython binaries, have the installer check for the Python version and install it if needed, have the grass startup script in Windows set a python path so that the needed version comes first for a grass session. Michael ### comment:24 in reply to: 19 ; follow-up: 25 Changed 12 years ago by cmbarton Replying to hamish: Replying to cmbarton: Why can't windows use g.gui gui=[guiplatform] -nu? Replying to cnielsen: This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem. • this is hardly as user friendly as a GUI which asks: "use Wx", "use Tcl", or "use text" & restart. these wrapper scripts are not designed for power users who are comfortable with the command line. • the default osgeo4w build does not give the user a msys prompt, although I guess you could try the Cmd> prompt if you knew the exact magic incantation. I agree. Given that this is aimed at users who are not so command-sophisticated and certainly using the GUI menus, maybe this is better done by a small, quick method inside the GUI code. It should work for all platforms that way. • v.type from a GUI is impossible and needs a wrapper. Yes. This needs a script. Or v.type needs to be rewritten a little so that it parses intelligently. There is a problem running python scripts in GRASS for Windows. it's just a bug, and not a fundamental problem. mere details can and will be fixed. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts. we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason. IIRC, Glynn said that it is necessary to alter the Windows registry for it to work. That is why doing a regular Python installation is preferable. Currently, no Python script works. Michael Hamish ### comment:25 in reply to: 24 ; follow-up: 26 Changed 12 years ago by hamish Replying to cmbarton: maybe this is better done by a small, quick method inside the GUI code. It should work for all platforms that way. while that may work for expediency, the best solution is to actually tackle the bug, figure it out, and solve the general case as it will cause more problems once we are more dependent on python scripts in MS Windows. • v.type from a GUI is impossible and needs a wrapper. Yes. This needs a script. Or v.type needs to be rewritten a little so that it parses intelligently. the problem is due to the limitations of G_parser() and the module GUI generation code, not specifically v.type. Personally I think it is counter productive to force the modules to work around those limitations, as opposed to addressing the fundamental problem. (the same goes for forcing module options into the Required tab which don't really need to be there, just for the exposure) I'm not sure how well v.clean will work from the GUI launchers. this is something to work on in trunk of course, not 6.x. in 6 we have to rely on these wrapper scripts.. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts. we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason. IIRC, Glynn said that it is necessary to alter the Windows registry for it to work. see comment:4,5 above. We can do that automatically from a dos batch file. That is why doing a regular Python installation is preferable. we need to check the registry and set something if not already set. if already set we should not stomp on what the user has already set up. This doesn't require our own python, just something that works and is >2.4 or so. The wx + C++ modules on the other hand are the trickier problem. Currently, no Python script works. :-( note that there are shell versions of those two scripts in the same dir which are used by gis.m. Perhaps we can use those and delay dealing with the .py problem until the next release.. they should work like any other of the shell script modules in 6.x, just in a slightly different dir. They could be installed into the main shell script dir as an ugly work around. Hamish ### comment:26 in reply to: 25 Changed 12 years ago by mmetz Replying to hamish: I'm not sure how well v.clean will work from the GUI launchers. See also ticket #646 ### comment:27 in reply to: 22 Changed 12 years ago by glynn Replying to cmbarton: All the wrapper scripts in there fail on wingrass. My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True. On Mac we only need to supply the wxPython binaries to ensure that the correct version is run I think. Most of the wrapper scripts in gui/scripts don't use wx. When running commands from the GUI, it shouldn't make any difference whether those commands are shell scripts, Python scripts, batch files, or binaries. ### comment:28 in reply to: 21 Changed 12 years ago by hamish Replying to hamish: All the wrapper scripts in [$ETC/gui/scripts/] fail on wingrass.

My first guess would be that something isn't setting shell=True in the Popen constructor.

On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True.

[shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.]

lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar.

it seems to, in gui/wxpython/gui_modules/gcmd.py line ~ 434-441. and the regular shell script modules from $ETC/scripts/ all work fine AFAIK. Hamish ### comment:29 follow-up: 30 Changed 12 years ago by hamish for 6.4.0 I suggest to just copy the two problematic wrapper scripts +.bat (if applicable) into$GISBASE/scripts/ as part of the wingrass packaging scripts, just to make it work*. we can worry about fixing the underlying cause later on and ship a cleaner solution with 6.4.1.

[*] assuming doing that does work ... probably needs to patch the menu xml as well.

[] why did the 6.4.1 milestone go away? there were several bugs targeted for it.

Hamish

### comment:30 in reply to:  29 Changed 12 years ago by martinl

[] why did the 6.4.1 milestone go away? there were several bugs targeted for it.

### comment:31 follow-up:  32 Changed 12 years ago by hamish

should be fixed with r40260,61.

wxGUI on WinGrass wanted .bat in $GISBASE/bin/ while GIS.m wants them in$GISBASE/etc/scripts/. So now the Makefile installs them to both places.

Also the wxGUI menu xml needed to point to the .sh version to work.

this stuff is still a mystery:

set .py="%GRASSDIR%\extrabin\python.exe"
assoc .py=Python.File
ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*"
assoc .pyw=Python.NoConFile
ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*"

but now only an issue for grass 7 + wingrass.

(hmmm, actually I don't see where wxgui.py adds $(ETC)/gui/scripts/ with sys.path.append()? and yet somehow it still works on linux without specifying a path name!?) keeping bug open pending testing with the next wingrass build. Hamish ### comment:32 in reply to: 31 Changed 12 years ago by glynn Replying to hamish: this stuff is still a mystery: assoc .py=Python.File ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*" assoc .pyw=Python.NoConFile ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*" GRASS shouldn't be doing this. Changing these settings may break other applications. ### comment:33 Changed 12 years ago by hamish Resolution: → fixed new → closed ### comment:34 follow-ups: 35 39 Changed 10 years ago by mmetz Milestone: 6.4.0 → 7.0.0 fixed closed → reopened 6.4.0 RCs → svn-trunk Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? Example: Settings -> Install extensions from add-ons fails. Markus M ### comment:35 in reply to: 34 ; follow-up: 36 Changed 10 years ago by martinl Replying to mmetz: Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? Example: Settings -> Install extensions from add-ons fails. This should be fixed in trunk. Probably we could introduce bat-files for python scripts as in G6 for bash-scripts. I tried bin\g.manual.bat @"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %* from CLI g.manual command not found g.manual.bat line 1: @%GRASS_PYTHON%: command not found Any idea? ### comment:36 in reply to: 35 ; follow-up: 37 Changed 10 years ago by hellik Replying to martinl: Replying to mmetz: Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? Example: Settings -> Install extensions from add-ons fails. This should be fixed in trunk. Probably we could introduce bat-files for python scripts as in G6 for bash-scripts. I tried bin\g.manual.bat @"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %* from CLI g.manual command not found g.manual.bat line 1: @%GRASS_PYTHON%: command not found Any idea? I've tried some time ago g.manual.py.bat, as the bat-filename, it worked. Helmut ### comment:37 in reply to: 36 ; follow-up: 38 Changed 10 years ago by martinl Replying to hellik: I've tried some time ago g.manual.py.bat, as the bat-filename, it worked. in G7? There are no bat-files. ### comment:38 in reply to: 37 Changed 10 years ago by hellik Replying to martinl: Replying to hellik: I've tried some time ago g.manual.py.bat, as the bat-filename, it worked. in G7? There are no bat-files. manually added for testing some scripts, it's maybe somewhere in another ticket, maybe I remember where... Helmut ### comment:39 in reply to: 34 ; follow-ups: 40 41 Changed 10 years ago by glynn Replying to mmetz: Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? The system must be able to "execute" Python scripts. On Unix, this means that a Python interpreter named "python" must be present in$PATH. On Windows, it means that a Python interpreter must be associated with the .py extension.

Please do not try to implement half-baked "fixes" such as requiring .bat files or requiring special treatment for executing scripts. Both Unix and Windows have standard mechanisms to support execution of scripts; those mechanisms (and only those mechanisms) should be used.

### comment:40 in reply to:  39 Changed 10 years ago by mmetz

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement?

The system must be able to "execute" Python scripts.

On Unix, this means that a Python interpreter named "python" must be present in $PATH. On Windows, it means that a Python interpreter must be associated with the .py extension. That means that some python 2.x must be installed system-wide. The wingrass installer does not install python (%GISBASE%\extrabin\python.exe) system-wide and does not associate it with the .py extension (such as not to break any existing system-wide python-installation). Strangely, if I click on Cancel on the Windows window with the error message that a py file can not be opened, it is opened successfully and the corresponding GUI dialog appears. Obviously, that error message appears again when running the script (click on Run), and the script completes successfully after canceling the windows message. BTW, this applies to the latest GRASS7 wingrass installer, r50073. Markus M ### comment:41 in reply to: 39 ; follow-up: 42 Changed 10 years ago by martinl Replying to glynn: The system must be able to "execute" Python scripts. On Unix, this means that a Python interpreter named "python" must be present in$PATH. On Windows, it means that a Python interpreter must be associated with the .py extension.

Please do not try to implement half-baked "fixes" such as requiring .bat files or requiring special treatment for executing scripts. Both Unix and Windows have standard mechanisms to support execution of scripts; those mechanisms (and only those mechanisms) should be used.

just to sure what we want...

python scripts should be executable from

1. cmd
2. msys
3. wxGUI

without need to define extension, eg.

g.manual

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

### comment:42 in reply to:  41 ; follow-up:  43 Changed 10 years ago by glynn

just to sure what we want...

python scripts should be executable from

1. cmd
2. msys
3. wxGUI

In 7.0, MSys isn't required, and supporting it shouldn't be GRASS' responsibility (I don't know of any other Windows application which goes out of its way to support being used from MSys). If it is supported, it shouldn't be at the expense of native operation.

without need to define extension, eg.

g.manual

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

No, Windows supports this. Installing Python associates the .py extension with the Python interpreter. Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

MSys doesn't use extensions, it uses the shebang. Moreover, it generally does a poor job of interacting with native Windows programs. It was created to support Unix-style build systems (i.e. configure scripts and Unix Makefiles), and tends to be something of a walled garden. But unlike Cygwin, it isn't consistent. Whereas Cygwin uses Unix semantics (Unix-style pathnames, shebang) both in the shell and in executables, MSys uses Unix semantics in the shell but the executables use MSVCRT, which uses Windows semantics. It tries to automatically convert filenames, but only when passed as command-line arguments (and in a few specific environment variables, e.g. PATH). Passing filenames via files, pipes, other environment variables, etc requires conversions to be performed manually.

### comment:43 in reply to:  42 ; follow-ups:  44  50 Changed 9 years ago by mmetz

just to sure what we want...

python scripts should be executable from

1. cmd
2. msys
3. wxGUI

In 7.0, MSys isn't required, and supporting it shouldn't be GRASS' responsibility (I don't know of any other Windows application which goes out of its way to support being used from MSys). If it is supported, it shouldn't be at the expense of native operation.

without need to define extension, eg.

g.manual

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

No, Windows supports this. Installing Python associates the .py extension with the Python interpreter.

WinGRASS does not install python, nor numpy, scipy, or wxpython. These are all bundled with wingrass but not installed the Windows way. Therefore the system does not know that there is something like python.exe and the system does not know what to do with .py files. IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

WinGRASS is currently compiled against osgeo4w python and related packages, i.e. msys packages. It looks like that either wingrass installes python system-wide, the Windows way, or wingrass does not require a system-wide python and uses its own python, but then some .bat wrapper like

@"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %*

(see martinl) would be needed. Is there another solution for the case that wingrass should work without a system-wide python?

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

BTW, there is another strange side-effect. Addon scripts for GRASS 7 do not execute. Because of a some ugly hack it is possible to get the GUI for an addon script (--interface-description works), but clicking on run returns an error that the module is not found. This happens on some, not all windows systems.

Markus M

### comment:44 in reply to:  43 ; follow-up:  45 Changed 9 years ago by hellik

IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

how do other software deal with this python issue?

just a little overview here on my win7-box. python is shipped/bundled a few times:

C:\Python26\ArcGIS10.0 C:\Program Files (x86)\GRASS GIS 6.4.3svn\Python27 C:\Program Files (x86)\Inkscape\python C:\Program Files (x86)\LibreOffice? 3.5\program\python-core-2.6.1 C:\Program Files\GIMP 2\Python C:\OSGeo4W\apps\Python27

Helmut

### comment:45 in reply to:  44 ; follow-up:  46 Changed 9 years ago by mmetz

IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

how do other software deal with this python issue?

Maybe there are different approaches.

just a little overview here on my win7-box. python is shipped/bundled a few times:

C:\Python26\ArcGIS10.0 C:\Program Files (x86)\GRASS GIS 6.4.3svn\Python27 C:\Program Files (x86)\Inkscape\python C:\Program Files (x86)\LibreOffice? 3.5\program\python-core-2.6.1 C:\Program Files\GIMP 2\Python C:\OSGeo4W\apps\Python27

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

Markus M

### comment:46 in reply to:  45 ; follow-up:  47 Changed 9 years ago by hellik

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

C:\Users\xxxx>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

and a screenshot of standard software related to python added.

Helmut

python software

### comment:47 in reply to:  46 ; follow-up:  48 Changed 9 years ago by mmetz

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

C:\Users\xxxx>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

and a screenshot of standard software related to python added.

I was rather interested about how many different python versions are in the list of installed software as known by the system. The list .py* file extensions does not give the full path to the python.exe being used, i.e. which one of the 6 python.exe's on your system will be used.

Markus M

### comment:48 in reply to:  47 Changed 9 years ago by hellik

I was rather interested about how many different python versions are in the list of installed software as known by the system. The list .py* file extensions does not give the full path to the python.exe being used, i.e. which one of the 6 python.exe's on your system will be used.

it's not so easy to find out this in the strange windows world ;o), but here we go (it's in the registry):

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\py_auto_file]
@=""

[HKEY_CLASSES_ROOT\py_auto_file\shell]

[HKEY_CLASSES_ROOT\py_auto_file\shell\open]

[HKEY_CLASSES_ROOT\py_auto_file\shell\open\command]
@="\"C:\\Program Files (x86)\\GRASS 6.5.SVN\\extrabin\\python.exe\" \"%1\""
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.CompiledFile]
@="Compiled Python File"

[HKEY_CLASSES_ROOT\Python.CompiledFile\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\pyc.ico"

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell\open]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\python.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.CompiledFile\shellex]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.File]
@="Python File"

[HKEY_CLASSES_ROOT\Python.File\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\py.ico"

[HKEY_CLASSES_ROOT\Python.File\shell]

[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE]

[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"C:\\Python26\\ArcGIS10.0\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""

[HKEY_CLASSES_ROOT\Python.File\shell\open]

[HKEY_CLASSES_ROOT\Python.File\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\python.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.File\shellex]

[HKEY_CLASSES_ROOT\Python.File\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"
Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.NoConFile]
@="Python File (no console)"

[HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\py.ico"

[HKEY_CLASSES_ROOT\Python.NoConFile\shell]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"C:\\Python26\\ArcGIS10.0\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\open]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.NoConFile\shellex]

[HKEY_CLASSES_ROOT\Python.NoConFile\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"

Helmut

### comment:50 in reply to:  43 ; follow-up:  51 Changed 9 years ago by glynn

WinGRASS does not install python, nor numpy, scipy, or wxpython. These are all bundled with wingrass but not installed the Windows way. Therefore the system does not know that there is something like python.exe and the system does not know what to do with .py files. IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

Correct. WinGRASS should assume Python, the same way that it does on Linux. On Linux, we don't install Python, we just assume that "#!/usr/bin/env python" will invoke a suitable Python interpreter. Windows doesn't use a shebang, it uses extensions, so on Windows, the assumption should be that the .py extension is associated with a suitable Python interpreter.

WinGRASS is currently compiled against osgeo4w python and related packages,

WinGRASS isn't "compiled" against any version of Python.

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

If the .py extension hasn't been associated with a Python interpreter, executing Python scripts won't work, with or with the .py extension.

If we don't want to spend the rest of our lives adding one hack after another to deal with Python scripts on Windows, "executing" a Python script (via CreateProcess(), system(), subprocess.Popen() or whatever) has to just work, which requires that the .py extension is associated with a working Python interpreter.

### comment:51 in reply to:  50 ; follow-up:  53 Changed 9 years ago by mmetz

WinGRASS is currently compiled against osgeo4w python and related packages,

WinGRASS isn't "compiled" against any version of Python.

OK, but GRASS is compiled "with" a particular Python version, for ctypes. And apparently wxversion is needed which is only available in a devel package (Linux).

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

If the .py extension hasn't been associated with a Python interpreter, executing Python scripts won't work, with or with the .py extension.

If we don't want to spend the rest of our lives adding one hack after another to deal with Python scripts on Windows, "executing" a Python script (via CreateProcess(), system(), subprocess.Popen() or whatever) has to just work, which requires that the .py extension is associated with a working Python interpreter.

Should the existence of a working Python interpreter thus be the responsibility of the user? Should we therefore remove the bundled python from the wingrass installer? Because we do not want to interfere with an existing Python installation, we either do not include an official Python installer, or we do, but check first if any Python is already installed on the system.

According to Helmut's report 48 on the registry settings, I can not see any trace of the Python versions coming with Inkscape and LibreOffice, but only of the Python version coming with ArcGIS.

To phrase it provocatively, do we want to go the ESRI way or do we want to go the Inkscape and LibreOffice way?

Markus M

### comment:52 Changed 9 years ago by cmbarton

What has worked for the Mac is to bundle wxPython (that is where we've run into problems with versions) and use the system Python. We compile so that it works with both 2.6 and 2.7. I don't know if that would work for Windows or not, but maybe.

Michael

### comment:53 in reply to:  51 Changed 9 years ago by glynn

WinGRASS isn't "compiled" against any version of Python.

OK, but GRASS is compiled "with" a particular Python version, for ctypes.

The ctypesgen program which generates the ctypes wrappers is a Python script. The generated ctypes wrappers don't depend upon the version of which Python was used to run ctypesgen.

And apparently wxversion is needed which is only available in a devel package (Linux).

That would be a bug in the particular distribution. The Windows installers for wxPython include wxversion, and any sane Linux package for wxPython would do likewise. By separating wxversion from the main wxPython package, the distribution is effectively insisting that any wxPython-based code is customised for that specific distribution (i.e. is written for, and doesn't attempt to check for, that distribution's "official" wxPython version).

Should the existence of a working Python interpreter thus be the responsibility of the user? Should we therefore remove the bundled python from the wingrass installer? Because we do not want to interfere with an existing Python installation, we either do not include an official Python installer, or we do, but check first if any Python is already installed on the system.

The latter is preferable, but more work. I.e. if Python is not installed, we should attempt to install it (via official installers) and required packages. If Python is installed, we should check that it's a compatible version, and attempt to install any required packages (wxPython, numpy, etc) for that specific version.

According to Helmut's report 48 on the registry settings, I can not see any trace of the Python versions coming with Inkscape and LibreOffice, but only of the Python version coming with ArcGIS.

GUI "applications" don't require an installed Python. Typically, the Python interpreter is embedded in the application's main executable. Python scripts are only ever run from within that application. Often, scripts which are part of the application package wouldn't run elsewhere because they depend upon modules which are compiled into the application's executable.

This doesn't work when we're trying to provide a command-line and/or scripting environment. Ultimately, people should be able to write scripts and run them e.g. from an icon on the desktop or from some other program which knows nothing about either GRASS or Python.

(This depends in part upon fixing the installation process to make GRASS "sessions" an optional feature. I can't remember the last time I actually "started GRASS"; I just set up all of the environment variables in ~/.bash_profile so that GRASS commands work everywhere).

To phrase it provocatively, do we want to go the ESRI way or do we want to go the Inkscape and LibreOffice way?

ESRI way.

### comment:55 in reply to:  description ; follow-up:  57 Changed 7 years ago by hellik

Hi,

None of the GUI wrapper scripts called from the GUI menus work in WinGRASS due to lack of associated file extensions.

these are the shell and python scripts installed in $GISBASE/etc/gui/scripts/ explained here: see also bug #234. Hamish suggest to close the ticket as scripts are now working in winGRASS. at least downgrading priority if ticket should stay open. ### comment:56 follow-up: 58 Changed 7 years ago by martinl I wonder why this rule is not working$(DSTDIR)/%.bat: $(MODULE_TOPDIR)/scripts/windows_launch.bat sed -e "s#SCRIPT_NAME#$(%)#" -e "s#SCRIPT_DIR#$(DSTDIR)#"$(MODULE_TOPDIR)/scripts/windows_launch.bat > $@ unix2dos$@

it gives eg.

@"%GRASS_PYTHON%" "/c/osgeo4w/usr/src/grass_trunk/dist.i686-pc-mingw32/gui/scripts/.py" %*

should have been

@"%GRASS_PYTHON%" "/c/osgeo4w/usr/src/grass_trunk/dist.i686-pc-mingw32/gui/scripts/d.rast3d.py" %*

Similar variables like $< or$@ works as expected.

### comment:57 in reply to:  55 ; follow-up:  60 Changed 7 years ago by martinl

suggest to close the ticket as scripts are now working in winGRASS. at least downgrading priority if ticket should stay open.

before that we need to fix makefiles to produce also bat files from g.gui.* modules and GUI scripts (see my comment above).

### comment:58 in reply to:  56 ; follow-up:  59 Changed 7 years ago by glynn

I wonder why this rule is not working

$(DSTDIR)/%.bat:$(MODULE_TOPDIR)/scripts/windows_launch.bat
sed -e "s#SCRIPT_NAME#$(%)#" -e "s#SCRIPT_DIR#$(DSTDIR)#" $(MODULE_TOPDIR)/scripts/windows_launch.bat >$@
unix2dos $@ Similar variables like$< or $@ works as expected. What is "$(%)" supposed to be? The stem of a pattern rule (the part which matches the "%" in the target) is available via "$*". ### comment:59 in reply to: 58 ; follow-up: 61 Changed 7 years ago by martinl Replying to glynn: What is "$(%)" supposed to be? The stem of a pattern rule (the part which matches the "%" in the target) is available via "\$*".

right, thanks, it helped, see r62955-6.

### comment:60 in reply to:  57 ; follow-up:  62 Changed 7 years ago by martinl

suggest to close the ticket as scripts are now working in winGRASS. at least downgrading priority if ticket should stay open.

before that we need to fix makefiles to produce also bat files from g.gui.* modules and GUI scripts (see my comment above).

done in r62956, but it's not working - attempt to launch g.gui.animation from cmdline:

C:\>Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\OSGEO4~1\apps\Python27\lib\multiprocessing\forking.py", line 381, in
main
File "C:\OSGEO4~1\apps\Python27\lib\pickle.py", line 1378, in load
File "C:\OSGEO4~1\apps\Python27\lib\pickle.py", line 858, in load
dispatch[key](self)
File "C:\OSGEO4~1\apps\Python27\lib\pickle.py", line 1090, in load_global
klass = self.find_class(module, name)
File "C:\OSGEO4~1\apps\Python27\lib\pickle.py", line 1124, in find_class
__import__(module)
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\temporal\__init__.
py", line 28, in <module>
from temporal_algebra import *
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\temporal\temporal_
algebra.py", line 450, in <module>
import grass.pygrass.modules as pymod
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\modules\__
init__.py", line 2, in <module>
from grass.pygrass.modules.interface import Module, ParallelModuleQueue
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\modules\in
terface\__init__.py", line 9, in <module>
from grass.pygrass.modules.interface import module
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\modules\in
terface\module.py", line 297, in <module>
class Module(object):
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\modules\in
terface\module.py", line 700, in Module
@mdebug(1, extra=_get_bash)
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\modules\in
terface\module.py", line 36, in mdebug
msgr = get_msgr()
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\messages\_
_init__.py", line 352, in get_msgr
_instance[0] = Messenger(*args, **kwargs)
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\messages\_
_init__.py", line 175, in __init__
self.start_server()
File "C:\OSGEO4~1\apps\grass\grass-7.1.svn\etc\python\grass\pygrass\messages\_
_init__.py", line 185, in start_server
self.server.start()
File "C:\OSGEO4~1\apps\Python27\lib\multiprocessing\process.py", line 130, in
start
self._popen = Popen(self)
File "C:\OSGEO4~1\apps\Python27\lib\multiprocessing\forking.py", line 258, in
__init__
cmd = get_command_line() + [rhandle]
File "C:\OSGEO4~1\apps\Python27\lib\multiprocessing\forking.py", line 358, in
get_command_line
is not going to be frozen to produce a Windows executable.''')
RuntimeError:
Attempt to start a new process before the current process
has finished its bootstrapping phase.

### comment:61 in reply to:  59 ; follow-up:  63 Changed 7 years ago by martinl

right, thanks, it helped, see r62955-6.

r62055 produces for GUI scripts (d.rast3d and d.wms) the BAT files. But when adding a new 3D raster map to the GUI it seems that instead of BAT file is launched PY file. I removed PY from PATHEXT in r63160, but it had no effect. Is there any way how to force launching BAT wrappers instead of PY scripts? Or it must be placed in different directories?

### comment:62 in reply to:  60 Changed 7 years ago by martinl

done in r62956, but it's not working - attempt to launch g.gui.animation from cmdline:

this bug seems to be fixed in r63208

### comment:63 in reply to:  61 ; follow-up:  64 Changed 7 years ago by martinl

r62055 produces for GUI scripts (d.rast3d and d.wms) the BAT files. But when adding a new 3D raster map to the GUI it seems that instead of BAT file is launched PY file. I removed PY from PATHEXT in r63160, but it had no effect. Is there any way how to force launching BAT wrappers instead of PY scripts? Or it must be placed in different directories?

any idea how to solve this issue?