GRASS ctypes fail with Python 3.7.6+

[mmetz]

GRASS ctypes generated with the GRASS internal ​ctypes generator are no longer accepted by Python 3.7.6+, see also ​ctypesgen issue 77. Tested with Debian testing (Python 3.7.6) and Alpine edge (Python 3.8.1). Both distro versions are development versions, but they use released Python versions.

The errors are that a union is no longer accepted as a function argument, affecting e.g. the ​class string, and that strings consisting of of any number of characters terminated by the '\0' character are sometimes truncated to the first character.

Unfortunately, the main maintainer of ctypesgen does not test ​python versions > 3.6.

We need to fix the GRASS-internal ctypes generator to work with Python < 3.7.6 as well as Python >= 3.7.6.

[mmetz]

The main change is that for Python 3.7.6+ ​unions and structures with bit-fields should always be passed to functions by pointer.

The GRASS ctypes generator uses a custom ​class String which is a union, which needs to be passed to functions as a pointer for Python 3.7.6+.

Instead of using the custom class String, we could use the ​fundamental data type c_char_p. This works with Python 3.7.6 and Python 3.7.5, but not with Python 3.8.1. In some cases Python 3.8.1 complains about a TypeError, e.g. for C macros like

#define GRASS_VERSION_STRING   "@(#) 7.9.dev (2020)"

Maybe these C macros must be of ctype POINTER(c_char) which does not make sense because such a pointer would/should return only the first character pointed to by POINTER(c_char).

The important change in handling ctypes has been introduced with Python 3.7.6. The different behaviour of Python 3.7.6 and Python 3.8.1 regarding ctypes, particularly pointers to a '\0' terminated array of characters is right now a mystery.

I have a patch for the GRASS ctypes generator that works with Python 3.7.5 and 3.7.6, i.e. before and after the important change in ctypes handling has been introduced in Python, and another patch for the GRASS ctypes generator that works with Python 3.8.1, but not with Python 3.7.x. Therefore I regard my patches as dirty hacks and am looking forward to any advice on how to correctly generate fully compatible ctypes. Or even better, ​ctypesgen fixes these problems soon and we can use an updated version of ctypesgen.

[neteler]

The ctypesgen developer suggests a modification to our approach, see

​https://github.com/davidjamesca/ctypesgen/issues/77#issuecomment-593023266

[neteler]

Replying to mmetz:

... This works with Python 3.7.6 and Python 3.7.5, but not with Python 3.8.1. In some cases Python 3.8.1 complains about a TypeError, e.g. for C macros like

#define GRASS_VERSION_STRING   "@(#) 7.9.dev (2020)"

After the merge of ​https://github.com/OSGeo/grass/pull/325 there is only this occurrence left:

include/version.h.in
1:#define GRASS_VERSION_STRING   "@(#) @GRASS_VERSION_NUMBER@ (@GRASS_VERSION_DATE@)"

while in G78 (PR325 hasn't been backported yet) it is:

include/gis.h
42:#define GIS_H_VERSION GRASS_VERSION_STRING

include/version.h.in
1:#define GRASS_VERSION_STRING   "@(#) @GRASS_VERSION_NUMBER@ (@GRASS_VERSION_DATE@)"

Maybe GRASS_VERSION_STRING isn't needed at all?

[neteler]

The "@(#)" removal has been addressed in ​https://github.com/OSGeo/grass/pull/528

[martinl]

Can we close the ticket?

[neteler]

Yes, closing (feel free to reopen if needed)