PyGRASS does not work when GRASS is invoked from outside

[wenzeslaus]

When one want to ​work with GRASS without starting it explicitly, it is possible to use grass.script but it is not possible to use grass.pygrass, grass.temporal and grass.lib.

This might have been discussed on mailing already, anyway the reason is that the dynamic libraries are not accessible to the current process.

Output on Linux:

Traceback (most recent call last):
  File "./run_grass_from_outside.py", line 62, in <module>
    from grass.pygrass import vector
  File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/pygrass/__init__.py", line 7, in <module>
    import grass.lib.gis as _libgis
  File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/gis.py", line 23, in <module>
    _libs["grass_gis.7.1.svn"] = load_library("grass_gis.7.1.svn")
  File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py", line 55, in load_library
    return self.load(path)
  File "/opt/src/grass-trunk/dist.x86_64-unknown-linux-gnu/etc/python/grass/lib/ctypes_loader.py", line 71, in load
    raise ImportError,e
ImportError: libgrass_datetime.7.1.svn.so: cannot open shared object file: No such file or directory

Output on MS Windows:

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:\Anaconda\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 523, in runfile
   execfile(filename, namespace)
 File "C:\Users\akratoc\test_grass_standalone.py", line 59, in <module>
   from grass.pygrass import vector
 File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\pygrass\__init__.py", line 7, in <module>
   import grass.lib.gis as _libgis
 File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\gis.py", line 23, in <module>
   _libs["grass_gis.7.0.0beta3"] = load_library("grass_gis.7.0.0beta3")
 File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 55, in load_library
   return self.load(path)
 File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 221, in load
   return _WindowsLibrary(path)
 File "C:\Program Files (x86)\GRASS GIS 7.0.0beta3\etc\python\grass\lib\ctypes_loader.py", line 207, in _init_
   self.cdll = ctypes.cdll.LoadLibrary(path)
 File "C:\Anaconda\lib\ctypes\__init__.py", line 443, in LoadLibrary
   return self._dlltype(name)
 File "C:\Anaconda\lib\ctypes\__init__.py", line 365, in _init_
   self._handle = _dlopen(self._name, mode)
WindowsError: [Error 193] %1 is not a valid Win32 application

It is unfortunate that you even cannot import grass.pygrass.modules:

from grass.pygrass import modules

For Linux, we have to rely on decision of packagers and a user being able to manage ~/.bashrc or environmental variables of the given process.

For MS Windows, the solution would be to allow user to add the variables to the environment. It seems to me that it is "enough" to add path with dynamic libraries (lib and extrabin) and GRASS installation directory (the one with the grass*.bat file) to PATH and Python packages (etc/python) to PYTHONPATH.

The solution is of course invasive but there is no other way. A lot of applications just do the same without worrying about consequences. We can try what Git for MS Windows is doing: giving a choice how git commands and other tools should be spread into the system (modify or not modify the PATH).

The question of course is what should be the default.

Other question is whether to add to the beginning or the end of PATH and PYTHONPATH variables.

Also, there must be a warning (in MS Windows installer) that no other version of GRASS GIS will work.

The same might apply to other OSGeo or related software. This is unfortunately unknown. I guess that this option cannot be available in OSGeo4W installer.

I have no idea about Mac OS X. Which solution would be appropriate there?

Do you have some other ideas how to solve or workaround this issue? Would it be possible to provide a script/functionality in GRASS GIS which would put the things to environmental variables and take it away if requested (I guess that the removal is necessary for standard installation too)? This could be much more flexible. This might be even applicable to all platforms although for MS Windows it would be quite different from Linux.

Related:

• #2333 (choose python interpreter during the GRASS installation on windows)

[neteler]

On Linux, I had success with this startup:

export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python start_grass7_create_new_location_ADVANCED.py

[wenzeslaus]

Replying to neteler:

On Linux, I had success with this startup:

export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python start_grass7_create_new_location_ADVANCED.py

Sure, that's the problem. For Linux user it is incredibly simple, for MS Windows one, incredibly hard.

[wenzeslaus]

Another case when I'm hitting this issue is in running tests. There is a script which runs the tests. I import form gunittest in this test and I don't really need much but in __init__.py I import some things to provide and easy access, so whatever I want to import, __init__.py​ gets executed, because pygrass.modules are used this executes the pygrass main __init__.py​ which imports some PyGRASS things which depend on ctypes interface and when dynamic libraries are not available, this obviously fails. This is an example, how simple you may need dynamic libraries. Temporal framework Python API is probably the same case.

In some cases you can get around this by putting try excepts to the library __init__.py, as I did for gunittest​.

Another way how to improve the situation is to use empty __init__.py files. We are not doing it now, we are doing almost the opposite and it of course has a big advantage as hiding the inner structure of the package. On the other hand, from grass.script or grass.pygrass you usually need just a specific part. There is no consensus but there is a lot of people thinking that empty __init__.py files are the best. You anyway don't gain much by hiding the inner structure of the package unless you forbid the specific imports in your API doc and we are not doing this. This is of course something we should sort out before the release (sorry about that).

[zarch]

Replying to wenzeslaus:

Another case when I'm hitting this issue is in running tests. There is a script which runs the tests. I import form gunittest in this test and I don't really need much but in init.py I import some things to provide and easy access, so whatever I want to import, init.py gets executed, because pygrass.modules are used this executes the pygrass main init.py which imports some PyGRASS things which depend on ctypes interface and when dynamic libraries are not available, this obviously fails. This is an example, how simple you may need dynamic libraries. Temporal framework Python API is probably the same case.

Since the pygrass.modules does not depend from the ctypes, I think we could move the modules stuff outside pygrass to remove the depedencies to ctypes...

The main problem is that I put in pygrass.init:

import grass.lib.gis as _libgis
_libgis.G_gisinit('')

Because all the pygrass sub-modules: gis, raster, vector and functions must have G_gisinit, If you think that move the pygrass.modules outside pyrass is not a good idea, I can move the G_gisinit from the main init of pygrass to the sub-modules.

Another way how to improve the situation is to use empty init.py files.

Generally I'm not against to an empty or almost empty init file. So If you agree on that, I could change it.

This it will change the way as we import stuff, so it wont be possible to import:

from grass import pygrass

# and then access to the sub-modules
pygrass.modules.Module

For me it is not a problem, generally I import the class that I need on the top, like:

from grass.pygrass.modules import Module

In any case I think that the pygrass.modules.init should contain at least:

from grass.pygrass.modules.interface.module import Module, ParallelModuleQueue

just to make the use of these classes more convenient and to "hide" the internal structure:

modules/
  interface/
    module.py
    ...

Otherwise it became too verbose.

[wenzeslaus]

Replying to zarch:

Since the pygrass.modules does not depend from the ctypes, I think we could move the modules stuff outside pygrass to remove the depedencies to ctypes...

The main problem is that I put in pygrass.__init__:

import grass.lib.gis as _libgis
_libgis.G_gisinit('')

Because all the pygrass sub-modules: gis, raster, vector and functions must have G_gisinit, If you think that move the pygrass.modules outside pyrass is not a good idea, I can move the G_gisinit from the main __init__ of pygrass to the sub-modules.

I cannot comment on G_gisinit much but the error is that the DLL is not found, so I though that it is before the call.

Another way how to improve the situation is to use empty __init__.py files.

Generally I'm not against to an empty or almost empty init file. So If you agree on that, I could change it.

This it will change the way as we import stuff, so it wont be possible to import:

from grass import pygrass

# and then access to the sub-modules
pygrass.modules.Module

For me it is not a problem, generally I import the class that I need on the top, like:

from grass.pygrass.modules import Module

This is what I do too, the same for grass.script. But grass.script is an example where some things were used for compatibility and changes in internal structure are expected in the future I would say. But generally, yes, it seems that it is better to have empty __init__.py files.

I'm not sure if it is better to move pygrass.modules outside of pygrass or not. I was not thinking about it much. However, it seems that pygrass.modules is really something different from the library interface. The only connection is that it is new and Pythonic and it was implemented together. The ctypes dependency is quite a big difference. I'm not sure if there will be something in the future which would introduce ctypes to modules. Perhaps yes, for example some command line parsing interface using GRASS parser (based on library rather then on g.parser). What would be the name of a standalone package? Just grass.modules?

In any case I think that the pygrass.modules.__init__ should contain at least:

from grass.pygrass.modules.interface.module import Module, ParallelModuleQueue

just to make the use of these classes more convenient and to "hide" the internal structure:

Otherwise it became too verbose.

I agree, this is really internal and __init__.py should be used to hide it.

[wenzeslaus]

More general but related:

We may want to follow QGIS 3.0 ticket PyQGIS bootstrap is complicated which suggests adding QGIS Python packages to standard Python packages path to avoid setting PYTHONPATH and perhaps other things:

​https://github.com/qgis/qgis3.0_api/issues/9

Also note that now besides the ​wiki page, we have also a pretty good documentation of the start process.

Additionally, besides the way suggested by Markus:

export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python python_script_with_pygrass_or_ctypes.py

we have now in trunk (after #2579):

grass ~/grassdata/nc_spm/test/ --exec python_script_with_pygrass_or_ctypes.py

which ensures that the scripts runs in exactly same environment as if it would be executed in GRASS GIS session.

[neteler]

Milestone renamed

[neteler]

Ticket retargeted after milestone closed

[wenzeslaus]

The fact that temporal API also doesn't work probably has increasing importance (​GRASS-dev: Working with TGIS without starting GRASS explicitly). However, unlike PyGRASS it should be possible to effectively replace most of the API calls by relevant temporal (t.*) modules which wrap them.

[neteler]

Ticket retargeted after milestone closed

[neteler]

Please see

• Python: GRASS GIS 7 with an external library: grass-session:
  • ​https://grasswiki.osgeo.org/wiki/Working_with_GRASS_without_starting_it_explicitly#Python:_GRASS_GIS_7_with_an_external_library:_grass-session
• New easy-to-use CLI and API for GRASS GIS
  • ​https://trac.osgeo.org/grass/wiki/GSoC/2018#Neweasy-to-useCLIandAPIforGRASSGIS

[wenzeslaus]

These both help, but just to clarify, here we are concerned with only the part of the Python API which uses ctypes in the background, i.e. not grass.script which works just fine, but with grass.temporal and a significant part of grass.pygrass.

Replying to neteler:

• Python: GRASS GIS 7 with an external library: grass-session:

This makes using grass.script much easier, but it is not possible to use anything which calls ctypes (grass.lib), i.e. grass.temporal and grass.pygrass, unless the dynamic libraries are loaded in some special way (just changing LD_LIBRARY_PATH was not enough so far). (From grass.pygrass I mean the part using ctypes, otherwise see the comment:3 to comment:6 here.)

• New easy-to-use CLI and API for GRASS GIS

In past your would have to do this (see comment:6 for details):

export LD_LIBRARY_PATH=$(grass70 --config path)/lib
python script_with_ctypes.py

Since 7.2 (after #2579, i.e. r65252), you can (see again comment:6 for details):

grass ~/grassdata/nc_spm/test/ --exec script_with_ctypes.py

In regard to this ticket, the proposed CLI would work similarly to current --exec. Same goes to the proposed Python API and the current APIs.

Nevertheless, --exec and the proposed CLI interface reduce the need to load the libraries, for example, because the recommended way of using them is to write a GRASS module and then call it using --exec - this way the GRASS-specific part of the program is isolated and at the same time usable as a separate GRASS module.

[martinl]

Ticket retargeted after milestone closed

[martinl]

Still relevant?

[sbl]

Here is a way to load PyGRASS libs in external Python:

​https://github.com/ninsbl/grass-session/blob/grass_versions/grass_session/session.py#L305

Not sure if this is what you are looking for? Have not tested on Windows either...

[wenzeslaus]

Replying to martinl:

Still relevant?

Yes. As far as I know only thing which changed is having additional workaround (besides explicit LD_LIBRARY_PATH or --exec and that is installing grass-session/grass_session which is technically a 3rd party package in that sense that it is not part of GRASS GIS.

Replying to sbl:

Here is a way to load PyGRASS libs in external Python:

​https://github.com/ninsbl/grass-session/blob/grass_versions/grass_session/session.py#L305

Not sure if this is what you are looking for?

Yes, ctypes.CDLL(lib, mode=1) looks like a way to address the issue. It doesn't seem right to me that code to make GRASS GIS API work is outside of GRASS GIS. Any ideas or opinions on having grass-session functionality in GRASS GIS? Perhaps even avoiding the need for grass-session?

Have not tested on Windows either...

Yes, that's a big concern here because LD_LIBRARY_PATH-like solution is not practical there (and setting GRASSBIN is not either) and then there is the issue of multiple Pythons and where or how to install grass-session (which is actually not limited to Windows).