Create a system for building and running GRASS modules written in Python

[wenzeslaus]

The r.modis Makefiles changed ​several times in last two years and still there are problems in running it (#2097 and it does not work for me now).

The main r.modis Makefile is:

MODULE_TOPDIR =../..

PGM = r.modis

SUBDIRS = \
        r.modis.download \
        r.modis.import \
		libmodis

include $(MODULE_TOPDIR)/include/Make/Dir.make

default: parsubdirs htmldir

install: installsubdirs

The critical r.modis Python package Makefile says:

MODULE_TOPDIR = ../../..

include $(MODULE_TOPDIR)/include/Make/Other.make
include $(MODULE_TOPDIR)/include/Make/Python.make

MODULES = downmodis rmodislib convertmodis parsemodis

ETCDIR = $(ETC)/r.modis

PYFILES := $(patsubst %,$(ETCDIR)/%.py,$(MODULES))
PYCFILES := $(patsubst %,$(ETCDIR)/%.pyc,$(MODULES))

default: $(PYFILES) $(PYCFILES)

$(ETCDIR):
	$(MKDIR) $@

$(ETCDIR)/%: % | $(ETCDIR)
	$(INSTALL_DATA) $< $@

install:
	cp -r $(ETCDIR) $(INST_DIR)

I was trying to understand and make it work but I don't get any results because I don't understand and I don't know where we actually want to place the Python scripts and Python packages in distribution and in addons if the modules are from addons.

Can somebody design the Makefile and/or Makefile rules?

Than we need runtime support for loading the Python packages.

One possibility is to put all packages needed by different modules into one directory. Then it is enough just to import the package in the module.

For example for core (non-addons) functionality:

.../grass/
.../rmodis/

The other is to have one directory for each module (or module group) and put there their package or packages. Typically there is only one, so:

.../rmodis/rmodis/

or maybe

.../rmodis/rmodis/
.../rmodis/3rdpartylib/

Probably in both cases we need some function which will determine where the package is and will add the path (sys.path) correctly. This function can be in grass.utils, so it will be not necessary to repeat the same code in all modules and then change it when there is an error in it. It could be something like:

def add_pythonlib_to_path(name):
    libpath = None
    # this is specified by Makefile
    # maybe this needs to be different for core modules
    # and for addons
    if os.path.isdir(os.path.join(os.getenv('GISBASE'), 'etc', name)):
        libpath = os.path.join(os.getenv('GISBASE'), 'etc', name)
    elif os.getenv('GRASS_ADDON_BASE') and \
            os.path.isdir(os.path.join(os.getenv('GRASS_ADDON_BASE'), 'etc',
                                       name)):
        libpath = os.path.join(os.getenv('GRASS_ADDON_BASE'), 'etc', name)
    # this is the directory name
    # when we run the script from source
    elif os.path.join(os.path.dirname(__file__), '..', name):
        libpath = os.path.join(os.path.dirname(__file__), '..')
    # the file with script is somewhere but package is `../package`
    # maybe this should be removed because it is strange case
    # maybe replace with current directory but that's strange too
    elif os.path.isdir(os.path.join('..', name)):
        libpath = os.path.join('..', name)
    else:
        gcore.fatal(_("Python library '%s' not found. Probably it was not"
                      "intalled correctly.") % name)

    sys.path.append(libpath)

add_pythonlib_to_path('rmodis')

I think that the Makefiles for module group (i.e. directory with GRASS modules written in Python and their Python packages) should work for both addons and when just placed in the source code.

I think that the requirement is that the module should run without running make. The documentation would be broken but at least it would work.

With this we need the naming convention for packages and module group directory.

[zarch]

Hi Vaclav,

I can not help to fix the Makefile, It is mysterious to me too!

Probably in both cases we need some function which will determine where the package is and will add the path (sys.path) correctly. This function can be in grass.utils, so it will be not necessary to repeat the same code in all modules and then change it when there is an error in it. It could be something like:

[cut]

I did, and is already in the trunk, maybe we should put in some other place, at the moment is in ​pygrass/functions.py you can find an example of use in ​i.segment.hierarchical

[wenzeslaus]

Replying to zarch:

Hi Vaclav,

I can not help to fix the Makefile, It is mysterious to me too!

Probably in both cases we need some function which will determine where the package is and will add the path (sys.path) correctly. This function can be in grass.utils, so it will be not necessary to repeat the same code in all modules and then change it when there is an error in it. It could be something like:

[cut]

I did, and is already in the trunk, maybe we should put in some other place, at the moment is in ​pygrass/functions.py you can find an example of use in ​i.segment.hierarchical

That's great, thanks for writing it. This is very general function and is for supporting scripts, so it should go to grass.script (lib/python/script), probably to utils. It can be actually in the same category as parser function, I just don't have any nice name for it (scripts would be nice but the whole package is script already, let's reconsider? ;-).

i.segment.hierarchical reminds me that you can have also single GRASS module written in Python with one or more Python files (Python modules). This is probably clear and easier but there should be sample Makefile also for this case.

I will try to use the function. But still, Makefile is the problem in the first place.

[wenzeslaus]

In r58208, I used get_lib_path function to find the r.modis library/package. This actually fixes r.modis.import for me on Ubuntu. The r.modis.download started successfully before and starts also after the change.

However, the Makefile is still a issue at least for addons. How the directories are set and what the get_lib_path function tries has unclear system. There is a module group name r.modis (name of directory in source code) and than there is a library (which is supposed to be a Python package?) name libmodis. However, libmodis is renamed to r.modis when installed and libraries (or whatever) from one module group are grouped to the same directory named by module group. So, the result is r.modis/r.modis directory in GRASS_ADDON_BASE (e.g., /home/john/.grass7/addons/r.modis/r.modis).

Grouping the libraries makes sense to avoid potential conflicts. For few libraries it is not necessary but I can imagine a groups of modules in addons which includes some 3rd party library into the distribution. Then grouping is advantage. The renaming seems strange to me. Is there some reason for it?

I'm not sure where the other than Python files related to the module or module group should go. This is the case of resources such as XMLs or images. I'm not sure if there is some mechanism for it now.

The terminology started to be a real issue here, I'm not sure how to speak about things. Addon, extension, package, library, GRASS module, Python module, module group, module package...

[wenzeslaus]

According to r59110 it seems that the "module groups" are not supported even for C when speaking about modules in addons.

[neteler]

Ticket retargeted after milestone closed

[neteler]

Ticket retargeted after milestone closed

[neteler]

Maybe this Makefile simplification could help: ​https://github.com/OSGeo/grass-addons/pull/45

[neteler]

Ticket retargeted after milestone closed