Opened 10 years ago
Last modified 5 years ago
#2424 new defect
PyGRASS does not work when GRASS is invoked from outside
Reported by: | wenzeslaus | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | 7.2.4 |
Component: | Python ctypes | Version: | svn-trunk |
Keywords: | installation, pygrass, temporal, scripts | Cc: | |
CPU: | Unspecified | Platform: | MSWindows 8 |
Description
When one want to work with GRASS without starting it explicitly, it is possible to use grass.script
but it is not possible to use grass.pygrass
, grass.temporal
and grass.lib
.
This might have been discussed on mailing already, anyway the reason is that the dynamic libraries are not accessible to the current process.
Output on Linux:
Traceback (most recent call last): File "./run_grass_from_outside.py", line 62, in <module> from grass.pygrass import vector File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/pygrass/__init__.py", line 7, in <module> import grass.lib.gis as _libgis File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/gis.py", line 23, in <module> _libs["grass_gis.7.1.svn"] = load_library("grass_gis.7.1.svn") File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py", line 55, in load_library return self.load(path) File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py", line 71, in load raise ImportError,e ImportError: libgrass_datetime.7.1.svn.so: cannot open shared object file: No such file or directory
Output on MS Windows:
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 523, in runfile execfile(filename, namespace) File "C:\Users\akratoc\test_grass_standalone.py", line 59, in <module> from grass.pygrass import vector File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\pygrass\__init__.py", line 7, in <module> import grass.lib.gis as _libgis File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\gis.py", line 23, in <module> _libs["grass_gis.7.0.0beta3"] = load_library("grass_gis.7.0.0beta3") File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 55, in load_library return self.load(path) File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 221, in load return _WindowsLibrary(path) File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 207, in _init_ self.cdll = ctypes.cdll.LoadLibrary(path) File "C:\Anaconda\lib\ctypes\__init__.py", line 443, in LoadLibrary return self._dlltype(name) File "C:\Anaconda\lib\ctypes\__init__.py", line 365, in _init_ self._handle = _dlopen(self._name, mode) WindowsError: [Error 193] %1 is not a valid Win32 application
It is unfortunate that you even cannot import grass.pygrass.modules
:
from grass.pygrass import modules
For Linux, we have to rely on decision of packagers and a user being able to manage ~/.bashrc
or environmental variables of the given process.
For MS Windows, the solution would be to allow user to add the variables to the environment. It seems to me that it is "enough" to add path with dynamic libraries (lib
and extrabin
) and GRASS installation directory (the one with the grass*.bat
file) to PATH
and Python packages (etc/python
) to PYTHONPATH
.
The solution is of course invasive but there is no other way. A lot of applications just do the same without worrying about consequences. We can try what Git for MS Windows is doing: giving a choice how git commands and other tools should be spread into the system (modify or not modify the PATH
).
The question of course is what should be the default.
Other question is whether to add to the beginning or the end of PATH
and PYTHONPATH
variables.
Also, there must be a warning (in MS Windows installer) that no other version of GRASS GIS will work.
The same might apply to other OSGeo or related software. This is unfortunately unknown. I guess that this option cannot be available in OSGeo4W installer.
I have no idea about Mac OS X. Which solution would be appropriate there?
Do you have some other ideas how to solve or workaround this issue? Would it be possible to provide a script/functionality in GRASS GIS which would put the things to environmental variables and take it away if requested (I guess that the removal is necessary for standard installation too)? This could be much more flexible. This might be even applicable to all platforms although for MS Windows it would be quite different from Linux.
Related:
- #2333 (choose python interpreter during the GRASS installation on windows)
Change History (18)
follow-up: 2 comment:1 by , 10 years ago
follow-up: 4 comment:2 by , 10 years ago
Replying to neteler:
On Linux, I had success with this startup:
export LD_LIBRARY_PATH=$(grass70 --config path)/lib python start_grass7_create_new_location_ADVANCED.py
Sure, that's the problem. For Linux user it is incredibly simple, for MS Windows one, incredibly hard.
comment:3 by , 10 years ago
Another case when I'm hitting this issue is in running tests. There is a script which runs the tests. I import form gunittest
in this test and I don't really need much but in __init__.py
I import some things to provide and easy access, so whatever I want to import, __init__.py gets executed, because pygrass.modules
are used this executes the pygrass
main __init__.py which imports some PyGRASS things which depend on ctypes
interface and when dynamic libraries are not available, this obviously fails. This is an example, how simple you may need dynamic libraries. Temporal framework Python API is probably the same case.
In some cases you can get around this by putting try excepts to the library __init__.py
, as I did for gunittest.
Another way how to improve the situation is to use empty __init__.py
files. We are not doing it now, we are doing almost the opposite and it of course has a big advantage as hiding the inner structure of the package. On the other hand, from grass.script
or grass.pygrass
you usually need just a specific part. There is no consensus but there is a lot of people thinking that empty __init__.py
files are the best. You anyway don't gain much by hiding the inner structure of the package unless you forbid the specific imports in your API doc and we are not doing this. This is of course something we should sort out before the release (sorry about that).
follow-up: 5 comment:4 by , 10 years ago
Replying to wenzeslaus:
Another case when I'm hitting this issue is in running tests. There is a script which runs the tests. I import form gunittest in this test and I don't really need much but in init.py I import some things to provide and easy access, so whatever I want to import, init.py gets executed, because pygrass.modules are used this executes the pygrass main init.py which imports some PyGRASS things which depend on ctypes interface and when dynamic libraries are not available, this obviously fails. This is an example, how simple you may need dynamic libraries. Temporal framework Python API is probably the same case.
Since the pygrass.modules does not depend from the ctypes, I think we could move the modules stuff outside pygrass to remove the depedencies to ctypes...
The main problem is that I put in pygrass.init:
import grass.lib.gis as _libgis _libgis.G_gisinit('')
Because all the pygrass sub-modules: gis, raster, vector and functions must have G_gisinit, If you think that move the pygrass.modules outside pyrass is not a good idea, I can move the G_gisinit from the main init of pygrass to the sub-modules.
Another way how to improve the situation is to use empty init.py files.
Generally I'm not against to an empty or almost empty init file. So If you agree on that, I could change it.
This it will change the way as we import stuff, so it wont be possible to import:
from grass import pygrass # and then access to the sub-modules pygrass.modules.Module
For me it is not a problem, generally I import the class that I need on the top, like:
from grass.pygrass.modules import Module
In any case I think that the pygrass.modules.init should contain at least:
from grass.pygrass.modules.interface.module import Module, ParallelModuleQueue
just to make the use of these classes more convenient and to "hide" the internal structure:
modules/ interface/ module.py ...
Otherwise it became too verbose.
comment:5 by , 10 years ago
Replying to zarch:
Since the pygrass.modules does not depend from the ctypes, I think we could move the modules stuff outside pygrass to remove the depedencies to ctypes...
The main problem is that I put in
pygrass.__init__
:
import grass.lib.gis as _libgis _libgis.G_gisinit('')
Because all the pygrass sub-modules: gis, raster, vector and functions must have
G_gisinit
, If you think that move the pygrass.modules outside pyrass is not a good idea, I can move theG_gisinit
from the main__init__
of pygrass to the sub-modules.
I cannot comment on G_gisinit
much but the error is that the DLL is not found, so I though that it is before the call.
Another way how to improve the situation is to use empty
__init__.py
files.Generally I'm not against to an empty or almost empty init file. So If you agree on that, I could change it.
This it will change the way as we import stuff, so it wont be possible to import:
from grass import pygrass # and then access to the sub-modules pygrass.modules.Module
For me it is not a problem, generally I import the class that I need on the top, like:
from grass.pygrass.modules import Module
This is what I do too, the same for grass.script
. But grass.script
is an example where some things were used for compatibility and changes in internal structure are expected in the future I would say. But generally, yes, it seems that it is better to have empty __init__.py
files.
I'm not sure if it is better to move pygrass.modules
outside of pygrass
or not. I was not thinking about it much. However, it seems that pygrass.modules
is really something different from the library interface. The only connection is that it is new and Pythonic and it was implemented together. The ctypes
dependency is quite a big difference. I'm not sure if there will be something in the future which would introduce ctypes
to modules. Perhaps yes, for example some command line parsing interface using GRASS parser (based on library rather then on g.parser
). What would be the name of a standalone package? Just grass.modules
?
In any case I think that the
pygrass.modules.__init__
should contain at least:
from grass.pygrass.modules.interface.module import Module, ParallelModuleQueue
just to make the use of these classes more convenient and to "hide" the internal structure:
Otherwise it became too verbose.
I agree, this is really internal and __init__.py
should be used to hide it.
comment:6 by , 9 years ago
Milestone: | 7.0.0 → 7.1.0 |
---|
More general but related:
We may want to follow QGIS 3.0 ticket PyQGIS bootstrap is complicated which suggests adding QGIS Python packages to standard Python packages path to avoid setting PYTHONPATH
and perhaps other things:
https://github.com/qgis/qgis3.0_api/issues/9
Also note that now besides the wiki page, we have also a pretty good documentation of the start process.
Additionally, besides the way suggested by Markus:
export LD_LIBRARY_PATH=$(grass70 --config path)/lib python python_script_with_pygrass_or_ctypes.py
we have now in trunk (after #2579):
grass ~/grassdata/nc_spm/test/ --exec python_script_with_pygrass_or_ctypes.py
which ensures that the scripts runs in exactly same environment as if it would be executed in GRASS GIS session.
comment:9 by , 8 years ago
The fact that temporal API also doesn't work probably has increasing importance (GRASS-dev: Working with TGIS without starting GRASS explicitly). However, unlike PyGRASS it should be possible to effectively replace most of the API calls by relevant temporal (t.*) modules which wrap them.
comment:10 by , 7 years ago
Milestone: | 7.2.1 → 7.2.2 |
---|
follow-up: 13 comment:12 by , 7 years ago
Please see
- Python: GRASS GIS 7 with an external library: grass-session:
- New easy-to-use CLI and API for GRASS GIS
comment:13 by , 7 years ago
These both help, but just to clarify, here we are concerned with only the part of the Python API which uses ctypes
in the background, i.e. not grass.script
which works just fine, but with grass.temporal
and a significant part of grass.pygrass
.
Replying to neteler:
- Python: GRASS GIS 7 with an external library: grass-session:
This makes using grass.script
much easier, but it is not possible to use anything which calls ctypes (grass.lib
), i.e. grass.temporal
and grass.pygrass
, unless the dynamic libraries are loaded in some special way (just changing LD_LIBRARY_PATH
was not enough so far). (From grass.pygrass
I mean the part using ctypes
, otherwise see the comment:3 to comment:6 here.)
- New easy-to-use CLI and API for GRASS GIS
In past your would have to do this (see comment:6 for details):
export LD_LIBRARY_PATH=$(grass70 --config path)/lib python script_with_ctypes.py
Since 7.2 (after #2579, i.e. r65252), you can (see again comment:6 for details):
grass ~/grassdata/nc_spm/test/ --exec script_with_ctypes.py
In regard to this ticket, the proposed CLI would work similarly to current --exec
. Same goes to the proposed Python API and the current APIs.
Nevertheless, --exec
and the proposed CLI interface reduce the need to load the libraries, for example, because the recommended way of using them is to write a GRASS module and then call it using --exec
- this way the GRASS-specific part of the program is isolated and at the same time usable as a separate GRASS module.
comment:15 by , 7 years ago
Milestone: | → 7.2.4 |
---|
follow-up: 18 comment:17 by , 5 years ago
Here is a way to load PyGRASS libs in external Python:
https://github.com/ninsbl/grass-session/blob/grass_versions/grass_session/session.py#L305
Not sure if this is what you are looking for? Have not tested on Windows either...
comment:18 by , 5 years ago
Replying to martinl:
Still relevant?
Yes. As far as I know only thing which changed is having additional workaround (besides explicit LD_LIBRARY_PATH
or --exec
and that is installing grass-session/grass_session
which is technically a 3rd party package in that sense that it is not part of GRASS GIS.
Replying to sbl:
Here is a way to load PyGRASS libs in external Python:
https://github.com/ninsbl/grass-session/blob/grass_versions/grass_session/session.py#L305
Not sure if this is what you are looking for?
Yes, ctypes.CDLL(lib, mode=1)
looks like a way to address the issue. It doesn't seem right to me that code to make GRASS GIS API work is outside of GRASS GIS. Any ideas or opinions on having grass-session functionality in GRASS GIS? Perhaps even avoiding the need for grass-session?
Have not tested on Windows either...
Yes, that's a big concern here because LD_LIBRARY_PATH
-like solution is not practical there (and setting GRASSBIN
is not either) and then there is the issue of multiple Pythons and where or how to install grass-session (which is actually not limited to Windows).
On Linux, I had success with this startup: