WinGRASS: $GISBASE/etc/gui/scripts/ require something like $(EXE) to run

[hamish]

Hi,

None of the GUI wrapper scripts called from the GUI menus work in WinGRASS due to lack of associated file extensions.

these are the shell and python scripts installed in $GISBASE/etc/gui/scripts/

explained here:

 http://thread.gmane.org/gmane.comp.gis.grass.devel/32133

see also bug #234.

Hamish

[hamish]

I may have solved the problem of the GUI scripts not running on WinGRASS with r37085, 86, 87. They were missing .bat wrappers. leaving bug open as I haven't properly tested it.

TODO

• grass 7: (gui/scripts/Makefile) fix (wildcard *.*) and then use with create.bat:
• wxpython/scripts/Makefile: do p.* need .bat files too?
• similar thing needed to get g.gui launcher to work on WinGRASS?

Hamish

[hamish]

still doesn't work for v.type_wrapper.py and g.change.gui.py using latest native wingrass installer (6.4.0.svn 2009-05-02).

in wx it fails looking for the --interface-description. in tcl it just gives a red "X" in the output window, no error message.

maybe need something like:

 guiscritpspath = os.path.join(globalvar.ETCDIR, 'gui','scripts')
 sys.path.append(guiscriptspath)

??

(still doesn't work, similar error if called with --ui from the msys terminal prompt)

Hamish

[hamish]

This still doesn't work for v.type_wrapper.py and g.change.gui.py using latest native WinGrass installer (Colin's 6.4.0.svn r38537 2009-07-29).

wxGUI Vector -> Develop -> Convert object types menu item gives a windows dialog:

 Windows cannot open this file:
 File: v.type_wrapper.py
To open this file, Windows needs to know what program created it. ...
[*] Use the web service to find the approp prog
[ ] Select the program from a list

        [ok] [cancel]

and if you press cancel the wxGui layer manager output tab comes back with a Traceback error ending in

...
IOError
:
Unable to fetch interface description for command
'v.type_wrapper.py'.

echo $PATHEXT in the associated Msys terminal ends in ';.PY', so at least that part of lib/init/init.bat or init.sh is getting run.

neither $PATH nor $PYTHONPATH includes $GISBASE/etc/gui/scripts/ see this trac ticket log for some sys.path.append(guiscriptspath) ideas.

also I'd note that NVIZ still doesn't start from the wxGUI: File -> NVIZ (req tcltk) brings up the NVIZ launch dialog and lets you select a map but when you hit [Run] it just returns after 0sec. Same if you try from the wxGui Cmd> prompt. But if you type 'nviz -q' at the associated Msys terminal it brings it up ok.

??

Hamish

[hamish]

(working with Colin's native WinGrass installer r39271)

%PATHEXT% is used to indicate that .py files are executable and usable without their extension. But this only seems to work from the DOS box version of WinGrass, not from the MSys xterm (no big surprise there).

Running a python script from the DOS prompt it fails though because it doesn't know what program to associate .PY with.

what is apparently missing is:

set .py="%GISBASE%/extrabin\python.exe"

but setting that in the DOS box doesn't associate it properly.

(aside: strangely dir "%GISBASE%/extrabin\python.exe" succeeds while dir "%GISBASE%/extrabin/python.exe" fails at the DOS prompt)

some random google hit shows it might need assoc + ftype as well, so I add this all together in C:\GRASS-6-SVN\grass64.bat just after the "set PYTHONHOME":

set .py="%GRASSDIR%\extrabin\python.exe"
assoc .py=Python.File
ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*"
assoc .pyw=Python.NoConFile
ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*"

and then "cd C:\GRASS-6-SVN\etc\gui\scripts" and v.type_wrapper<enter> finally launches python, but gives a "unknown option -e" error and shows the python.exe usage text.

should set PYTHONPATH="%GRASSDIR%\etc\python" be set in C:\GRASS-6-SVN\grass64.bat as well?

?, Hamish

[glynn]

Replying to hamish:

%PATHEXT% is used to indicate that .py files are executable and usable without their extension. But this only seems to work from the DOS box version of WinGrass, not from the MSys xterm (no big surprise there).

Right. MSys' bash emulates Unix execution semantics.

Ultimately, WinGRASS is supposed to be "GRASS for Windows", not "GRASS for MSys".

Running a python script from the DOS prompt it fails though because it doesn't know what program to associate .PY with. what is apparently missing is:

set .py="%GISBASE%/extrabin\python.exe"

I've never seen this syntax. The normal way to associate applications with extensions is via assoc and ftype (but AFAICT, those change the system-wide settings, which may have be overriden by per-user settings).

some random google hit shows it might need assoc + ftype as well,

It's more likely to be "instead of". "set" sets environment variables, which have nothing to do with file associations.

so I add this all together in C:\GRASS-6-SVN\grass64.bat just after the "set PYTHONHOME":

set .py="%GRASSDIR%\extrabin\python.exe"
assoc .py=Python.File
ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*"
assoc .pyw=Python.NoConFile
ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*"

Note that this will change the registry keys, which will permanently affect the handling of all .py files on the system, not just those which are part of GRASS, and not just for the lifetime of the command prompt.

Bundling a "local" copy of Python (with a matching copy of wxPython) may be a reasonable approach for getting the wxGUI working, but any standalone scripts will use the system Python installation, and GRASS shouldn't be interfering with that (applications which automatically "steal" file associations are generally considered to be malware).

and then "cd C:\GRASS-6-SVN\etc\gui\scripts" and v.type_wrapper<enter> finally launches python, but gives a "unknown option -e" error and shows the python.exe usage text. should set PYTHONPATH="%GRASSDIR%\etc\python" be set in C:\GRASS-6-SVN\grass64.bat as well?

Probably not. This should only be set when invoking a Python script using the bundled copy of Python. Scripts run from the command prompt will use the system's Python, so you shouldn't be modifying PYTHONPATH in this case.

[hamish]

Replying to hamish:

Running a python script from the DOS prompt it fails though because it doesn't know what program to associate .PY with. what is apparently missing is:

  set .py="%GISBASE%/extrabin\python.exe"

Replying to glynn:

I've never seen this syntax.

Nevermind, further down in the thread that suggested that is the small caveat that belongs to some NT4 3rd party replacement for cmd.exe (gee, thanks for the tip then).

The normal way to associate applications with extensions is via assoc and ftype (but AFAICT, those change the system-wide settings, which may have be overriden by per-user settings).

...

Note that this will change the registry keys, which will permanently affect the handling of all .py files on the system, not just those which are part of GRASS, and not just for the lifetime of the command prompt.

... leading to more like bug #570

So lacking a way to locally set the extension association, our only choice is to ditch PATHEXT and use wrapper scripts which ensure that the GRASS_PYTHON enviro var. is respected. Workable from the GUI, but running GRASS python modules from the command line will be at the mercy of whatever version of python the user has installed. (a python wrapper script which launched another (but this time known) version of python to run a script is probably asking for trouble)

Bundling a "local" copy of Python (with a matching copy of wxPython) may be a reasonable approach for getting the wxGUI working,

(and even with doing that we still aren't close to mastering it)

but any standalone scripts will use the system Python installation, and GRASS shouldn't be interfering with that

is there anything left which won't use the python set by the GRASS_PYTHON enviro variable? (from #570 apparently the answer is yes)

(applications which automatically "steal" file associations are generally considered to be malware).

... Windows would seem to have a lot of malware then ... at least the DLL version clobbering in favour of the last-to-run installer is less prevalent these days.

Hamish

[cmbarton]

Given all this, is there a way to run a Python script in GRASS for Windows now (i.e., by including some code in the script or its batch file) or will it have to wait until the GRASS code is changed?

Michael

[glynn]

Replying to hamish:

The normal way to associate applications with extensions is via assoc and ftype (but AFAICT, those change the system-wide settings, which may have be overriden by per-user settings).

...

Note that this will change the registry keys, which will permanently affect the handling of all .py files on the system, not just those which are part of GRASS, and not just for the lifetime of the command prompt.

... leading to more like bug #570

I thought that #570 related specifically to the GUI.

Note that the GUI is a bit more problematic because it requires a version of Python with wx 2.8, and relying upon the system Python having this is a bit optimistic (but not entirely unrealistic; wxPython has stock binary packages to match the stock Python 2.4, 2.5 and 2.6, and supports concurrent installation of multiple versions of wxPython).

So lacking a way to locally set the extension association, our only choice is to ditch PATHEXT and use wrapper scripts which ensure that the GRASS_PYTHON enviro var. is respected.

The obvious choice is to simply use the system Python, just like we do on Unix.

It's not as if the user is overwhelmed by a choice of 20 different third-party Python packages. There are standard Windows (and MacOSX) binary packages available from the python.org website (without any of the licensing issues associated with relying upon ActiveState, like was the case for Tcl/Tk?).

If the user wants to associate the .py extension with something which isn't compatible[*] with the stock Python, that's their problem. It's not like we support Unix systems where /bin/sh is csh, either.

[*] It doesn't have to actually be the stock Python, but it does need to provide the same language with the same modules. Scripts don't need binary compatibility.

but any standalone scripts will use the system Python installation, and GRASS shouldn't be interfering with that

is there anything left which won't use the python set by the GRASS_PYTHON enviro variable? (from #570 apparently the answer is yes)

Running a .py script from the command line or via the Windows shell (i.e. via ShellExecute, which includes subprocess.Popen(..., shell = True)) won't use GRASS_PYTHON, nor should it. Even within a GRASS session, the command line should be able to run non-GRASS commands, including those written in Python.

[hamish]

Replying to cmbarton:

Given all this, is there a way to run a Python script in GRASS for Windows now (i.e., by including some code in the script or its batch file) or will it have to wait until the GRASS code is changed?

I think the answer is yes, there is a way. This bug report is mostly worried about add-on scripts in $GISBASE/etc/gui/scripts being called from the GUI. You will have to experiment from the command line to test local g.parser scripts or those in GRASS_ADDON_DIR; please let us know results!

Hamish

[cmbarton]

So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line.

Michael

[glynn]

Replying to cmbarton:

So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line.

On Windows, 6.4's g.parser explicitly invokes the script via $GRASS_SH. Obviously this won't work for Python scripts.

FWIW, Python scripts work in Windows in 7.0, whether the script is in the path or not (once PYTHONPATH has been set; Init.bat doesn't set this).

The biggest difference between the two is that in 7.0, the grass.script.parser() Python function uses "g.parser -s", which eliminates the need to re-execute the script. It may be worth back-porting this feature to 6.x (the existing behaviour is still available for use by shell scripts).

[cmbarton]

Replying to glynn:

Replying to cmbarton:

So far nothing works. We did up the Python script the way it is suggested in the WIKI and it works fine on Linux and Mac. It either does nothing or hangs before launching on Windows. The farthest we've gotten is to get the error that the script does not generate an XML interface-description. My interpretation is that python parser work on Linux and Mac but not on Windows from the command line.

On Windows, 6.4's g.parser explicitly invokes the script via $GRASS_SH. Obviously this won't work for Python scripts.

Does this mean that it is only expecting shell scripts in Windows? What happens on other platforms (since Python does run without problems in Linux and on Mac)?

FWIW, Python scripts work in Windows in 7.0, whether the script is in the path or not (once PYTHONPATH has been set; Init.bat doesn't set this). The biggest difference between the two is that in 7.0, the grass.script.parser() Python function uses "g.parser -s", which eliminates the need to re-execute the script. It may be worth back-porting this feature to 6.x (the existing behaviour is still available for use by shell scripts).

Seems like that would be a very good idea, if it means that you can run something besides shell scripts in a Windows environment.

Michael

[glynn]

Replying to cmbarton:

On Windows, 6.4's g.parser explicitly invokes the script via $GRASS_SH. Obviously this won't work for Python scripts.

Does this mean that it is only expecting shell scripts in Windows?

Yes. Windows doesn't understand shebang lines, so g.parser can't "run" the shell scripts as if they were executables; it has to use "$GRASS_SH <script>".

It might be possible to use "$GRASS_SH -c <script>", as MSys' bash understands shebang lines. OTOH, for "#!/usr/bin/env python", this will only work if there is an executable named "python" in the path (if it's called e.g. "python25" or it isn't in the path, you lose).

Alternatively, the scripts could be installed with a ".sh" extension, relying upon the file associations to run such scripts via bash.

Or you could make g.parser examine the first line of the script and choose a suitable interpreter (it isn't as simple as processing a shebang on Unix as we don't know where MSys' virtual filesystem is located and we can't assume that the "#!/usr/bin/env" trick will work with "native" interpreters).

What happens on other platforms (since Python does run without problems in Linux and on Mac)?

The code to re-invoke the script is conditionalised upon "#ifdef __MINGW32__". Unix (including MacOSX) simply execl()s the script directly, while the Windows version invokes it via $GRASS_SH.

[hamish]

Hello,

update for this bug report based on the latest native wingrass installer (r40049) on XP:

works:

• wxGUI File->"NVIZ (req's tcltk)"
• "nviz -q" from the wxGUI layer manager 'Cmd> ' prompt
• wxGUI Vector->Develop->"Digitize (req's tcltk)"

(excellent!)

does not work:

• wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.)
• wxGUI Vector->Develop->Change object types (Traceback error in command output tab ... Unable to fetch interface description for command 'v.type_wrapper.py'.)
• from the Cmd> prompt:
  • g.change.gui (nothing happens),
  • g.change.gui.py (Unable to fetch interface desc),
  • g.change.gui.py --help (Command finished 0 sec),
  • g.change.gui.py --interface-description (Command finished 0 sec)

any suggestions?

Hamish

[cmbarton]

Why can't windows use g.gui gui=[guiplatform] -nu?

In fact, even though both g.change.gui and v.type_wrapper are in my source, they are not getting into my OS X binary. Perhaps they are not getting into the Windows binary either?

There is a problem running python scripts in GRASS for Windows. Ironic in that one reason for Python is that it is cross-platform. From recent discussions on this (Glynn, Colin, me, others), it seems like the best solution will be to NOT package Python with GRASS (wxPython may be a different issue), but to have an installer that checks for a Python installation and if not present directs the user to python.org to install it. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

Michael

[cnielsen]

Replying to cmbarton:

Why can't windows use g.gui gui=[guiplatform] -nu?

This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem.

In fact, even though both g.change.gui and v.type_wrapper are in my source, they are not getting into my OS X binary. Perhaps they are not getting into the Windows binary either?

These are present in the windows installations in etc\gui\scripts. Well, in fact there is g.change.gui.py g.change.gui.sh g.change.gui.sh.bat but no g.change.gui

There is a problem running python scripts in GRASS for Windows. Ironic in that one reason for Python is that it is cross-platform. From recent discussions on this (Glynn, Colin, me, others), it seems like the best solution will be to NOT package Python with GRASS (wxPython may be a different issue), but to have an installer that checks for a Python installation and if not present directs the user to python.org to install it. A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

I've been working on an update to the NSIS installer that will check to see if Python is installed via the registry, and if not auto-download and install the four necessary packages (python, wxpython, pywin32, and numpy). The installer already does something similar for the sample datasets.

However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not? Where do pywin32 and numpy fit in? Can build specific parts only be included in the installer and still use an external Python?

I can make the installer, but I need these details to know how to assemble it.

-Colin

[hellik]

Replying to hamish:

does not work: - wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.) - wxGUI Vector->Develop->Change object types (Traceback error in command output tab ... Unable to fetch interface description for command 'v.type_wrapper.py'.) - from the Cmd> prompt: * g.change.gui (nothing happens), * g.change.gui.py (Unable to fetch interface desc), * g.change.gui.py --help (Command finished 0 sec), * g.change.gui.py --interface-description (Command finished 0 sec)

a quick look at the manual:

http://grass.osgeo.org/grass64/manuals/html64_user/general.html

there seems to be no g.change.gui(.py)

g.gui is working on windows

Helmut

[hamish]

Replying to hamish:

does not work: - wxGUI Config->Working Environ->Change Default GUI (Traceback error in command output tab ... Unable to fetch interface description for command 'g.change.gui.py'.)

Replying to hellik:

there seems to be no g.change.gui(.py)

sure there is. look at the title of this bug report.

The source is in gui/scripts/, it gets installed to $GISBASE/etc/gui/scripts/. All the wrapper scripts in there fail on wingrass. These wrapper scripts are needed to make command line based modules more GUI dialog friendly, or to extract particular targted functionality.

 https://trac.osgeo.org/grass/browser/grass/branches/releasebranch_6_4/gui/scripts/

the above quoted menu path shows where it is in the wx menu.

g.gui is working on windows

that is something different (a normal C module, not a python wrapper script).

Hamish

[hamish]

Replying to cmbarton:

Why can't windows use g.gui gui=[guiplatform] -nu?

Replying to cnielsen:

This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem.

• this is hardly as user friendly as a GUI which asks: "use Wx", "use Tcl", or "use text" & restart. these wrapper scripts are not designed for power users who are comfortable with the command line.
• the default osgeo4w build does not give the user a msys prompt, although I guess you could try the Cmd> prompt if you knew the exact magic incantation.
• v.type from a GUI is impossible and needs a wrapper.

There is a problem running python scripts in GRASS for Windows.

it's just a bug, and not a fundamental problem. mere details can and will be fixed.

A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason.

Hamish

[glynn]

Replying to cnielsen:

However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not?

The vdigit and nviz modules are compiled for a specific version of Python, and vdigit for a specific version of wxPython. But at present, neither of those modules compile on Windows.

Where do pywin32 and numpy fit in?

The GUI needs numpy and pywin32, but those packages only need to be compatible with the system Python, not with any specific version, nor with wxPython.

Can build specific parts only be included in the installer and still use an external Python?

If Python code imports a binary module, it must be running under the version of Python/wxPython for which the binary module was compiled. If we were to supply the vdigit and nviz modules in the Windows installer, the wxGUI would need to be run using the version of Python/wxPython for which those modules were built.

[glynn]

Replying to hamish:

All the wrapper scripts in there fail on wingrass.

My first guess would be that something isn't setting shell=True in the Popen constructor.

On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True.

[shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.]

lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar.

[cmbarton]

Replying to glynn:

Replying to hamish:

All the wrapper scripts in there fail on wingrass.

My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True.

On Mac we only need to supply the wxPython binaries to ensure that the correct version is run I think. But maybe william will know better.

[shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.] lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar.

[cmbarton]

Replying to glynn:

Replying to cnielsen:

However, if the gui or other modules needs a build specific version then this could get complicated... Is there a clear idea of which parts require build specific and which do not?

The vdigit and nviz modules are compiled for a specific version of Python, and vdigit for a specific version of wxPython. But at present, neither of those modules compile on Windows.

Where do pywin32 and numpy fit in?

The GUI needs numpy and pywin32, but those packages only need to be compatible with the system Python, not with any specific version, nor with wxPython.

Can build specific parts only be included in the installer and still use an external Python?

If Python code imports a binary module, it must be running under the version of Python/wxPython for which the binary module was compiled. If we were to supply the vdigit and nviz modules in the Windows installer, the wxGUI would need to be run using the version of Python/wxPython for which those modules were built.

Suggestion for building binaries: Build with a a Python version (say 2.5.x), supply the specific wxPython binaries, have the installer check for the Python version and install it if needed, have the grass startup script in Windows set a python path so that the needed version comes first for a grass session.

Michael

[cmbarton]

Replying to hamish:

Replying to cmbarton:

Why can't windows use g.gui gui=[guiplatform] -nu?

Replying to cnielsen:

This command does work in the msys command prompt (native installer r40049), with this followed by g.gui &, you can switch back and forth with no problem.

* this is hardly as user friendly as a GUI which asks: "use Wx", "use Tcl", or "use text" & restart. these wrapper scripts are not designed for power users who are comfortable with the command line. * the default osgeo4w build does not give the user a msys prompt, although I guess you could try the Cmd> prompt if you knew the exact magic incantation.

I agree. Given that this is aimed at users who are not so command-sophisticated and certainly using the GUI menus, maybe this is better done by a small, quick method inside the GUI code. It should work for all platforms that way.

* v.type from a GUI is impossible and needs a wrapper.

Yes. This needs a script. Or v.type needs to be rewritten a little so that it parses intelligently.

There is a problem running python scripts in GRASS for Windows.

it's just a bug, and not a fundamental problem. mere details can and will be fixed.

A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason.

IIRC, Glynn said that it is necessary to alter the Windows registry for it to work. That is why doing a regular Python installation is preferable. Currently, no Python script works.

Michael

Hamish

[hamish]

Replying to cmbarton:

maybe this is better done by a small, quick method inside the GUI code. It should work for all platforms that way.

while that may work for expediency, the best solution is to actually tackle the bug, figure it out, and solve the general case as it will cause more problems once we are more dependent on python scripts in MS Windows.

* v.type from a GUI is impossible and needs a wrapper.

Yes. This needs a script. Or v.type needs to be rewritten a little so that it parses intelligently.

the problem is due to the limitations of G_parser() and the module GUI generation code, not specifically v.type. Personally I think it is counter productive to force the modules to work around those limitations, as opposed to addressing the fundamental problem. (the same goes for forcing module options into the Required tab which don't really need to be there, just for the exposure)

I'm not sure how well v.clean will work from the GUI launchers.

this is something to work on in trunk of course, not 6.x. in 6 we have to rely on these wrapper scripts..

A regular Python installation will properly put Python in the registry so that it can be called to execute scripts.

we should have the equivalent already set up in the msinstaller startup batch files and scripts to associate .py with a suitable python.exe. check this bug's previous comments & grep the msinstaller scripts for more. but that isn't working for some reason.

IIRC, Glynn said that it is necessary to alter the Windows registry for it to work.

see comment:4,5 above. We can do that automatically from a dos batch file.

That is why doing a regular Python installation is preferable.

we need to check the registry and set something if not already set. if already set we should not stomp on what the user has already set up. This doesn't require our own python, just something that works and is >2.4 or so. The wx + C++ modules on the other hand are the trickier problem.

Currently, no Python script works.

:-(

note that there are shell versions of those two scripts in the same dir which are used by gis.m. Perhaps we can use those and delay dealing with the .py problem until the next release.. they should work like any other of the shell script modules in 6.x, just in a slightly different dir. They could be installed into the main shell script dir as an ugly work around.

Hamish

[mmetz]

Replying to hamish:

I'm not sure how well v.clean will work from the GUI launchers.

See also ticket #646

[glynn]

Replying to cmbarton:

All the wrapper scripts in there fail on wingrass.

My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True.

On Mac we only need to supply the wxPython binaries to ensure that the correct version is run I think.

Most of the wrapper scripts in gui/scripts don't use wx. When running commands from the GUI, it shouldn't make any difference whether those commands are shell scripts, Python scripts, batch files, or binaries.

[hamish]

Replying to hamish:

All the wrapper scripts in [$ETC/gui/scripts/] fail on wingrass.

Replying to glynn:

My first guess would be that something isn't setting shell=True in the Popen constructor. On Windows, shell=False (the default) only works for .exe files. If you want to be able to "run" .py, .bat etc files, you have to set shell=True. [shell=True uses cmd /c ..., while shell=False calls CreateProcess() directly.] lib/python/core.py has its own versions of Popen() and call() which default to shell=True on Windows (Unix should always use shell=False). The wx GUI should be doing something similar.

it seems to, in gui/wxpython/gui_modules/gcmd.py line ~ 434-441. and the regular shell script modules from $ETC/scripts/ all work fine AFAIK.

Hamish

[hamish]

for 6.4.0 I suggest to just copy the two problematic wrapper scripts +.bat (if applicable) into $GISBASE/scripts/ as part of the wingrass packaging scripts, just to make it work*. we can worry about fixing the underlying cause later on and ship a cleaner solution with 6.4.1**.

[*] assuming doing that does work ... probably needs to patch the menu xml as well.

[**] why did the 6.4.1 milestone go away? there were several bugs targeted for it.

Hamish

[martinl]

Replying to hamish:

[**] why did the 6.4.1 milestone go away? there were several bugs targeted for it.

Re-added.

[hamish]

should be fixed with r40260,61.

wxGUI on WinGrass wanted .bat in $GISBASE/bin/ while GIS.m wants them in $GISBASE/etc/scripts/. So now the Makefile installs them to both places.

Also the wxGUI menu xml needed to point to the .sh version to work.

this stuff is still a mystery:

set .py="%GRASSDIR%\extrabin\python.exe"
assoc .py=Python.File
ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*"
assoc .pyw=Python.NoConFile
ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*"

but now only an issue for grass 7 + wingrass.

(hmmm, actually I don't see where wxgui.py adds $(ETC)/gui/scripts/ with sys.path.append()? and yet somehow it still works on linux without specifying a path name!?)

keeping bug open pending testing with the next wingrass build.

Hamish

[glynn]

Replying to hamish:

this stuff is still a mystery:

assoc .py=Python.File
ftype Python.File="%GRASSDIR%\extrabin\python.exe" "%1" "%*"
assoc .pyw=Python.NoConFile
ftype Python.NoConFile="%GRASSDIR%\extrabin\pythonw.exe" "%1" "%*"

GRASS shouldn't be doing this. Changing these settings may break other applications.

[mmetz]

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement?

Example: Settings -> Install extensions from add-ons fails.

Markus M

[martinl]

Replying to mmetz:

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? Example: Settings -> Install extensions from add-ons fails.

This should be fixed in trunk. Probably we could introduce bat-files for python scripts as in G6 for bash-scripts. I tried bin\g.manual.bat

@"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %*

from CLI

g.manual
command not found

g.manual.bat
line 1: @%GRASS_PYTHON%: command not found

Any idea?

[hellik]

Replying to martinl:

Replying to mmetz:

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement? Example: Settings -> Install extensions from add-ons fails.

This should be fixed in trunk. Probably we could introduce bat-files for python scripts as in G6 for bash-scripts. I tried bin\g.manual.bat {{{ @"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %* }}} from CLI {{{ g.manual command not found g.manual.bat line 1: @%GRASS_PYTHON%: command not found }}} Any idea?

I've tried some time ago

g.manual.py.bat,

as the bat-filename, it worked.

Helmut

[martinl]

Replying to hellik:

I've tried some time ago g.manual.py.bat, as the bat-filename, it worked.

in G7? There are no bat-files.

[hellik]

Replying to martinl:

Replying to hellik:

I've tried some time ago g.manual.py.bat, as the bat-filename, it worked.

in G7? There are no bat-files.

manually added for testing some scripts, it's maybe somewhere in another ticket, maybe I remember where...

Helmut

[glynn]

Replying to mmetz:

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement?

The system must be able to "execute" Python scripts.

On Unix, this means that a Python interpreter named "python" must be present in $PATH. On Windows, it means that a Python interpreter must be associated with the .py extension.

Please do not try to implement half-baked "fixes" such as requiring .bat files or requiring special treatment for executing scripts. Both Unix and Windows have standard mechanisms to support execution of scripts; those mechanisms (and only those mechanisms) should be used.

[mmetz]

Replying to glynn:

Replying to mmetz:

Reopening the ticket because calling python scripts from /scripts with wxGUI does not work with the error message that the system does not know what to do with py files. This applies to systems without a separate system-wide python 2.x installation. Or is a separate python 2.x installation now a requirement?

The system must be able to "execute" Python scripts. On Unix, this means that a Python interpreter named "python" must be present in $PATH. On Windows, it means that a Python interpreter must be associated with the .py extension.

That means that some python 2.x must be installed system-wide. The wingrass installer does not install python (%GISBASE%\extrabin\python.exe) system-wide and does not associate it with the .py extension (such as not to break any existing system-wide python-installation).

Strangely, if I click on Cancel on the Windows window with the error message that a py file can not be opened, it is opened successfully and the corresponding GUI dialog appears. Obviously, that error message appears again when running the script (click on Run), and the script completes successfully after canceling the windows message.

BTW, this applies to the latest GRASS7 wingrass installer, r50073.

Markus M

[martinl]

Replying to glynn:

The system must be able to "execute" Python scripts. On Unix, this means that a Python interpreter named "python" must be present in $PATH. On Windows, it means that a Python interpreter must be associated with the .py extension. Please do not try to implement half-baked "fixes" such as requiring .bat files or requiring special treatment for executing scripts. Both Unix and Windows have standard mechanisms to support execution of scripts; those mechanisms (and only those mechanisms) should be used.

just to sure what we want...

python scripts should be executable from

1. cmd
2. msys
3. wxGUI

without need to define extension, eg.

g.manual

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

[glynn]

Replying to martinl:

just to sure what we want... python scripts should be executable from 1. cmd 2. msys 3. wxGUI

In 7.0, MSys isn't required, and supporting it shouldn't be GRASS' responsibility (I don't know of any other Windows application which goes out of its way to support being used from MSys). If it is supported, it shouldn't be at the expense of native operation.

without need to define extension, eg.

g.manual

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

No, Windows supports this. Installing Python associates the .py extension with the Python interpreter. Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

MSys doesn't use extensions, it uses the shebang. Moreover, it generally does a poor job of interacting with native Windows programs. It was created to support Unix-style build systems (i.e. configure scripts and Unix Makefiles), and tends to be something of a walled garden. But unlike Cygwin, it isn't consistent. Whereas Cygwin uses Unix semantics (Unix-style pathnames, shebang) both in the shell and in executables, MSys uses Unix semantics in the shell but the executables use MSVCRT, which uses Windows semantics. It tries to automatically convert filenames, but only when passed as command-line arguments (and in a few specific environment variables, e.g. PATH). Passing filenames via files, pipes, other environment variables, etc requires conversions to be performed manually.

[mmetz]

Replying to glynn:

Replying to martinl:

just to sure what we want... python scripts should be executable from 1. cmd 2. msys 3. wxGUI

In 7.0, MSys isn't required, and supporting it shouldn't be GRASS' responsibility (I don't know of any other Windows application which goes out of its way to support being used from MSys). If it is supported, it shouldn't be at the expense of native operation.

without need to define extension, eg.

{{{ g.manual }}}

should be possible to execute somehow. But AFAIU, in this case Windows doesn't know that python should be called, right?

No, Windows supports this. Installing Python associates the .py extension with the Python interpreter.

WinGRASS does not install python, nor numpy, scipy, or wxpython. These are all bundled with wingrass but not installed the Windows way. Therefore the system does not know that there is something like python.exe and the system does not know what to do with .py files. IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

WinGRASS is currently compiled against osgeo4w python and related packages, i.e. msys packages. It looks like that either wingrass installes python system-wide, the Windows way, or wingrass does not require a system-wide python and uses its own python, but then some .bat wrapper like

@"%GRASS_PYTHON%" "%GISBASE%/scripts/g.manual.py" %*

(see martinl) would be needed. Is there another solution for the case that wingrass should work without a system-wide python?

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

BTW, there is another strange side-effect. Addon scripts for GRASS 7 do not execute. Because of a some ugly hack it is possible to get the GUI for an addon script (--interface-description works), but clicking on run returns an error that the module is not found. This happens on some, not all windows systems.

Markus M

[hellik]

Replying to mmetz:

IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

how do other software deal with this python issue?

just a little overview here on my win7-box. python is shipped/bundled a few times:

C:\Python26\ArcGIS10.0 C:\Program Files (x86)\GRASS GIS 6.4.3svn\Python27 C:\Program Files (x86)\Inkscape\python C:\Program Files (x86)\LibreOffice? 3.5\program\python-core-2.6.1 C:\Program Files\GIMP 2\Python C:\OSGeo4W\apps\Python27

Helmut

[mmetz]

Replying to hellik:

Replying to mmetz:

IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

how do other software deal with this python issue?

Maybe there are different approaches.

just a little overview here on my win7-box. python is shipped/bundled a few times: C:\Python26\ArcGIS10.0 C:\Program Files (x86)\GRASS GIS 6.4.3svn\Python27 C:\Program Files (x86)\Inkscape\python C:\Program Files (x86)\LibreOffice? 3.5\program\python-core-2.6.1 C:\Program Files\GIMP 2\Python C:\OSGeo4W\apps\Python27

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

Markus M

[hellik]

Replying to mmetz:

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

C:\Users\xxxx>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

and a screenshot of standard software related to python added.

Helmut

[hellik]

python software

[mmetz]

Replying to hellik:

Replying to mmetz:

Which ones are known to the system? You can check in Systemsteuerung -> Software or similar. Maybe only the one of ArcGIS10?

C:\Users\xxxx>echo %PATHEXT%
.COM;.EXE;.BAT;.CMD;.VBS;.VBE;.JS;.JSE;.WSF;.WSH;.MSC

and a screenshot of standard software related to python added.

I was rather interested about how many different python versions are in the list of installed software as known by the system. The list .py* file extensions does not give the full path to the python.exe being used, i.e. which one of the 6 python.exe's on your system will be used.

Markus M

[hellik]

Replying to mmetz:

I was rather interested about how many different python versions are in the list of installed software as known by the system. The list .py* file extensions does not give the full path to the python.exe being used, i.e. which one of the 6 python.exe's on your system will be used.

it's not so easy to find out this in the strange windows world ;o), but here we go (it's in the registry):

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\py_auto_file]
@=""

[HKEY_CLASSES_ROOT\py_auto_file\shell]

[HKEY_CLASSES_ROOT\py_auto_file\shell\open]

[HKEY_CLASSES_ROOT\py_auto_file\shell\open\command]
@="\"C:\\Program Files (x86)\\GRASS 6.5.SVN\\extrabin\\python.exe\" \"%1\""

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.CompiledFile]
@="Compiled Python File"

[HKEY_CLASSES_ROOT\Python.CompiledFile\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\pyc.ico"

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell\open]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\python.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.CompiledFile\shellex]

[HKEY_CLASSES_ROOT\Python.CompiledFile\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.File]
@="Python File"

[HKEY_CLASSES_ROOT\Python.File\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\py.ico"

[HKEY_CLASSES_ROOT\Python.File\shell]

[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE]

[HKEY_CLASSES_ROOT\Python.File\shell\Edit with IDLE\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"C:\\Python26\\ArcGIS10.0\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""

[HKEY_CLASSES_ROOT\Python.File\shell\open]

[HKEY_CLASSES_ROOT\Python.File\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\python.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.File\shellex]

[HKEY_CLASSES_ROOT\Python.File\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Python.NoConFile]
@="Python File (no console)"

[HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon]
@="C:\\Python26\\ArcGIS10.0\\DLLs\\py.ico"

[HKEY_CLASSES_ROOT\Python.NoConFile\shell]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\Edit with IDLE\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"C:\\Python26\\ArcGIS10.0\\Lib\\idlelib\\idle.pyw\" -n -e \"%1\""

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\open]

[HKEY_CLASSES_ROOT\Python.NoConFile\shell\open\command]
@="\"C:\\Python26\\ArcGIS10.0\\pythonw.exe\" \"%1\" %*"

[HKEY_CLASSES_ROOT\Python.NoConFile\shellex]

[HKEY_CLASSES_ROOT\Python.NoConFile\shellex\DropHandler]
@="{60254CA5-953B-11CF-8C96-00AA00B8708C}"

Helmut

[hellik]

related  http://trac.osgeo.org/grass/ticket/1282

[glynn]

Replying to mmetz:

WinGRASS does not install python, nor numpy, scipy, or wxpython. These are all bundled with wingrass but not installed the Windows way. Therefore the system does not know that there is something like python.exe and the system does not know what to do with .py files. IIRC, a decision was made some time ago to not install python the Windows way together with wingrass because that could interfere with already installed python versions.

Correct. WinGRASS should assume Python, the same way that it does on Linux. On Linux, we don't install Python, we just assume that "#!/usr/bin/env python" will invoke a suitable Python interpreter. Windows doesn't use a shebang, it uses extensions, so on Windows, the assumption should be that the .py extension is associated with a suitable Python interpreter.

WinGRASS is currently compiled against osgeo4w python and related packages,

WinGRASS isn't "compiled" against any version of Python.

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

If the .py extension hasn't been associated with a Python interpreter, executing Python scripts won't work, with or with the .py extension.

If we don't want to spend the rest of our lives adding one hack after another to deal with Python scripts on Windows, "executing" a Python script (via CreateProcess(), system(), subprocess.Popen() or whatever) has to just work, which requires that the .py extension is associated with a working Python interpreter.

[mmetz]

Replying to glynn:

Replying to mmetz:

WinGRASS is currently compiled against osgeo4w python and related packages,

WinGRASS isn't "compiled" against any version of Python.

OK, but GRASS is compiled "with" a particular Python version, for ctypes. And apparently wxversion is needed which is only available in a devel package (Linux).

Adding .PY to the PATHEXT environment variable allows Python scripts to be run without the extension (in the same way as .bat, .cmd, etc). That will work for anything which uses the shell (system(), Python's subprocess module, batch files, etc).

Even if the system knows nothing about any python.exe?

If the .py extension hasn't been associated with a Python interpreter, executing Python scripts won't work, with or with the .py extension. If we don't want to spend the rest of our lives adding one hack after another to deal with Python scripts on Windows, "executing" a Python script (via CreateProcess(), system(), subprocess.Popen() or whatever) has to just work, which requires that the .py extension is associated with a working Python interpreter.

Should the existence of a working Python interpreter thus be the responsibility of the user? Should we therefore remove the bundled python from the wingrass installer? Because we do not want to interfere with an existing Python installation, we either do not include an official Python installer, or we do, but check first if any Python is already installed on the system.

According to Helmut's report 48 on the registry settings, I can not see any trace of the Python versions coming with Inkscape and LibreOffice, but only of the Python version coming with ArcGIS.

To phrase it provocatively, do we want to go the ESRI way or do we want to go the Inkscape and LibreOffice way?

Markus M

[cmbarton]

What has worked for the Mac is to bundle wxPython (that is where we've run into problems with versions) and use the system Python. We compile so that it works with both 2.6 and 2.7. I don't know if that would work for Windows or not, but maybe.

Michael

[glynn]

Replying to mmetz:

WinGRASS isn't "compiled" against any version of Python.

OK, but GRASS is compiled "with" a particular Python version, for ctypes.

The ctypesgen program which generates the ctypes wrappers is a Python script. The generated ctypes wrappers don't depend upon the version of which Python was used to run ctypesgen.

And apparently wxversion is needed which is only available in a devel package (Linux).

That would be a bug in the particular distribution. The Windows installers for wxPython include wxversion, and any sane Linux package for wxPython would do likewise. By separating wxversion from the main wxPython package, the distribution is effectively insisting that any wxPython-based code is customised for that specific distribution (i.e. is written for, and doesn't attempt to check for, that distribution's "official" wxPython version).

Should the existence of a working Python interpreter thus be the responsibility of the user? Should we therefore remove the bundled python from the wingrass installer? Because we do not want to interfere with an existing Python installation, we either do not include an official Python installer, or we do, but check first if any Python is already installed on the system.

The latter is preferable, but more work. I.e. if Python is not installed, we should attempt to install it (via official installers) and required packages. If Python is installed, we should check that it's a compatible version, and attempt to install any required packages (wxPython, numpy, etc) for that specific version.

According to Helmut's report 48 on the registry settings, I can not see any trace of the Python versions coming with Inkscape and LibreOffice, but only of the Python version coming with ArcGIS.

GUI "applications" don't require an installed Python. Typically, the Python interpreter is embedded in the application's main executable. Python scripts are only ever run from within that application. Often, scripts which are part of the application package wouldn't run elsewhere because they depend upon modules which are compiled into the application's executable.

This doesn't work when we're trying to provide a command-line and/or scripting environment. Ultimately, people should be able to write scripts and run them e.g. from an icon on the desktop or from some other program which knows nothing about either GRASS or Python.

(This depends in part upon fixing the installation process to make GRASS "sessions" an optional feature. I can't remember the last time I actually "started GRASS"; I just set up all of the environment variables in ~/.bash_profile so that GRASS commands work everywhere).

To phrase it provocatively, do we want to go the ESRI way or do we want to go the Inkscape and LibreOffice way?

ESRI way.