GRASS ctypes exception handling

[huhabla]

This enhancement request is based on the GRASS GIS developers mailing list discussion in [1].

The basic idea is to catch fatal error calls in Python when using the ctypes GRASS library wrapper. Catching the exit call in case of a fatal error is needed to gently exit the calling Python module. So open file descriptors or database connections can be closed safely, unfinished imports/exports or temporary data can be removed correctly, region/mapset/location state of the current session can be reconstructed (calling GRASS modules not library functions).

Glynn suggested a solution to this question:

Soeren: Is it possible to raise a Python exception instead of calling exit in case of a fatal error when using ctypes wrapped GRASS library functions?

Glynn: Yes, but you would have to wrap each function individually.

Glynn suggested this code in the gis library to allow Python Exception calls in case of a fatal error:

static jmp_buf jbuf;
	
static void error_handler(void *arg)
{
    longjmp(jbuf, 1);
}
	
int call_with_catch(void (*func)(void *), void *arg)
{
    if (setjmp(jbuf) == 0) {
        G_add_error_handler(error_handler, NULL);
        (*func)(arg);
        G_remove_error_handler(error_handler, NULL);
        return 0;
    }
    else {
        G_remove_error_handler(error_handler, NULL);
        return -1;
    }
}

Trying to implement this conception, i struggled with converting multiple function arguments using void pointer handling in ctypes. The code below works only for specific functions. This file is called catch.c located in the lib/gis directory:

#include <setjmp.h>
#include <grass/gis.h>

static jmp_buf jbuf;

static void error_handler (void *arg)
{
 longjmp (jbuf, 1);
}

int G_call_with_catch (int (*func) (const char*, const char*), const char *name, const char *mapset, int *state)
{
 if (setjmp (jbuf) == 0)
   {
     int ret;
     G_add_error_handler (error_handler, NULL);
     ret = (*func) (name, mapset);
     G_remove_error_handler (error_handler, NULL);
     *state = 0;
     return ret;

   }
 else
   {
     G_remove_error_handler (error_handler, NULL);
     *state = -1;
     return 0;
   }
}

The entry in include/defs/gis.h:

int G_call_with_catch (int (*func) (const char*, const char*), const char *, const char *, int *);

The Python code to catch the fatal error call of Rast_open_old:

import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

ropen = CFUNCTYPE(c_int, c_char_p, c_char_p)(raster.Rast_open_old)

state = c_int()

fd = gis.G_call_with_catch(ropen, "raster_float", "PERMANENT", byref(state))

if state.value != 0:
       raise Exception("Error")

The problem is that the wrapped library functions have all kind/types of return values and arguments. Trying to catch this in Python using ctypes is far beyond my capabilities and IMHO tricky.

My suggestion is to generate a wrapper around each function which may call fatal error, using the setjmp/longjmp approach from Glynn. Example:

The raster open function

int Rast_open_old(char *name, char* mapset);

Will be wrapped by this function

/** This function will call Rast_open_old() and catch the exit call
 *  in case a fatal error occurs.
 *
 *  \param state This variable is set to 0 on success and -1 in case of a fatal error
 *  \param message This variable must be large enough to store the fatal error message
 */
int Rast_open_old_noexit(char *name, char *mapset, int *state, char *message){
/* doing setjmp stuff here, setting and unsetting error handler, ... */
}

Python code using this wrapper may look like this

import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

name = "elevation"
mapset = "PERMANENT"
state = c_int()
message = 2048 * c_char

fd = raster.Rast_open_old_noexit(name, mapset, byref(state), byref(message))
if state != 0:
    raise Exception("Fatal error message: %s"%message)

Such wrapping functions can be generated automatically in a pre-compile process in each library directory. Each function name will be extended with a _noexit prefix and two new variables will be added: state and message. A simple Python script can generate the wrapper and includes files.

Any suggestions are welcome.

Soeren

[1] ​http://comments.gmane.org/gmane.comp.gis.grass.devel/47721

[glynn]

Replying to huhabla: One way to get around the type issue is to define a call_with_catch function which is callable from Python, as described under ​Extending and Embedding the Python Interpreter. The function would have the same signature as Python's apply() function, but would catch fatal errors and convert them to Python exceptions.

If you only need coarse-grained fatal-error handling (e.g. calling exit handlers), then you don't need to wrap every function, just the script's main() function. Actually, you don't even need that; you can use the ctypes wrapper for G_add_error_handler(), just as a C module would.

[glynn]

Replying to glynn:

If you only need coarse-grained fatal-error handling (e.g. calling exit handlers),

And, in fact, that's all that should be attempted. Fine-grained error handling is largely pointless given that you cannot safely call GRASS library functions once a fatal error has been triggered (library functions are not required to ensure that internal data structures are in a consistent state before generating a fatal error).

[huhabla]

Replying to glynn:

Replying to glynn:

If you only need coarse-grained fatal-error handling (e.g. calling exit handlers),

And, in fact, that's all that should be attempted. Fine-grained error handling is largely pointless given that you cannot safely call GRASS library functions once a fatal error has been triggered (library functions are not required to ensure that internal data structures are in a consistent state before generating a fatal error).

Ok, i will try to set Python error handler to clean up in the modules and python libraries. I still struggle with the void pointer handling and Python object casting. A simple example:

import grass.lib.gis as gis
import grass.lib.raster as raster
from ctypes import *

class FatalErrorException(Exception):
        pass

def error_routine(message, flag):
    raise FatalErrorException(message)

def cleanup_handler(name):
    print "Removing %s"%(name)

ERROR_ROUTINE = CFUNCTYPE(c_int, c_char_p, c_int)(error_routine)
CLEANUP_HANDLER = CFUNCTYPE(c_void_p, c_void_p)(cleanup_handler)

def set_raise_on_library_error_routine(raise_exp = True):
    if raise_exp:
        gis.G_set_error_routine(ERROR_ROUTINE)
    else:
        gis.G_unset_error_routine()

def main():
    gis.G_gisinit("test")

    set_raise_on_library_error_routine(True)
    name = "elevation_not_exists"
    mapset = "PERMANENT"

    gis.G_add_error_handler(CLEANUP_HANDLER, "%s@%s"%(name, mapset))
    fd = raster.Rast_open_old(name, mapset)


if __name__ == "__main__":
    main()

Running this module will result in:

GRASS 7.0.svn (Test):~/src > python test.py 
Traceback (most recent call last):
  File "_ctypes/callbacks.c", line 313, in 'calling callback function'
  File "test.py", line 9, in error_routine
    raise FatalErrorException(message)
__main__.FatalErrorException: Raster map <elevation_not_exists@PERMANENT> not found
Removing 10575748

Raising a FatalErrorException is a bit pointless, but it works as expected. The main problem is that i have no clue how to cast the Python string from void pointer in the clean up handler, or how to cast any other Python object i need to clean up?

[glynn]

Replying to huhabla:

A simple example:

CLEANUP_HANDLER = CFUNCTYPE(c_void_p, c_void_p)(cleanup_handler)

I think that this should be

CLEANUP_HANDLER = CFUNCTYPE(None, c_void_p)(cleanup_handler)

The return type should be "void", not "void *".

Raising a FatalErrorException is a bit pointless, but it works as expected. The main problem is that i have no clue how to cast the Python string from void pointer in the clean up handler, or how to cast any other Python object i need to clean up?

I think that the handler should just call sys.exit(), so that any exit handlers registered with atexit.register() are called.

[wenzeslaus]

What is the current status and opinion on this? Does RPC approach (with Python multiprocessing) from #2134 solves this issue or does it at least say how this should be implemented? One cleanup handler with FatalErrorException or wrapper for each function exposed by ctypes sound interesting (and also some C and C++ applications might appreciate some set_error_routine but we already had this discussion). According to commits related to #2134, I would say that this is wontfix, so I'm closing this ticket (reopen if I'm mistaken).