Opened 11 months ago

Closed 7 months ago

#4018 closed defect (fixed)

GRASS ctypes fail with Python 3.7.6+

Reported by: mmetz Owned by: grass-dev@…
Priority: critical Milestone: 7.8.3
Component: Python ctypes Version: git-releasebranch78
Keywords: ctypes Cc:
CPU: All Platform: All

Description

GRASS ctypes generated with the GRASS internal ctypes generator are no longer accepted by Python 3.7.6+, see also ctypesgen issue 77. Tested with Debian testing (Python 3.7.6) and Alpine edge (Python 3.8.1). Both distro versions are development versions, but they use released Python versions.

The errors are that a union is no longer accepted as a function argument, affecting e.g. the class string, and that strings consisting of of any number of characters terminated by the '\0' character are sometimes truncated to the first character.

Unfortunately, the main maintainer of ctypesgen does not test python versions > 3.6.

We need to fix the GRASS-internal ctypes generator to work with Python < 3.7.6 as well as Python >= 3.7.6.

Change History (6)

comment:1 Changed 11 months ago by mmetz

The main change is that for Python 3.7.6+ unions and structures with bit-fields should always be passed to functions by pointer.

The GRASS ctypes generator uses a custom class String which is a union, which needs to be passed to functions as a pointer for Python 3.7.6+.

Instead of using the custom class String, we could use the fundamental data type c_char_p. This works with Python 3.7.6 and Python 3.7.5, but not with Python 3.8.1. In some cases Python 3.8.1 complains about a TypeError, e.g. for C macros like

#define GRASS_VERSION_STRING   "@(#) 7.9.dev (2020)"

Maybe these C macros must be of ctype POINTER(c_char) which does not make sense because such a pointer would/should return only the first character pointed to by POINTER(c_char).

The important change in handling ctypes has been introduced with Python 3.7.6. The different behaviour of Python 3.7.6 and Python 3.8.1 regarding ctypes, particularly pointers to a '\0' terminated array of characters is right now a mystery.

I have a patch for the GRASS ctypes generator that works with Python 3.7.5 and 3.7.6, i.e. before and after the important change in ctypes handling has been introduced in Python, and another patch for the GRASS ctypes generator that works with Python 3.8.1, but not with Python 3.7.x. Therefore I regard my patches as dirty hacks and am looking forward to any advice on how to correctly generate fully compatible ctypes. Or even better, ctypesgen fixes these problems soon and we can use an updated version of ctypesgen.

comment:2 Changed 9 months ago by neteler

The ctypesgen developer suggests a modification to our approach, see

https://github.com/davidjamesca/ctypesgen/issues/77#issuecomment-593023266

comment:3 in reply to:  1 Changed 8 months ago by neteler

Replying to mmetz:

... This works with Python 3.7.6 and Python 3.7.5, but not with Python 3.8.1. In some cases Python 3.8.1 complains about a TypeError, e.g. for C macros like

#define GRASS_VERSION_STRING   "@(#) 7.9.dev (2020)"

After the merge of https://github.com/OSGeo/grass/pull/325 there is only this occurrence left:

include/version.h.in
1:#define GRASS_VERSION_STRING   "@(#) @GRASS_VERSION_NUMBER@ (@GRASS_VERSION_DATE@)"

while in G78 (PR325 hasn't been backported yet) it is:

include/gis.h
42:#define GIS_H_VERSION GRASS_VERSION_STRING

include/version.h.in
1:#define GRASS_VERSION_STRING   "@(#) @GRASS_VERSION_NUMBER@ (@GRASS_VERSION_DATE@)"

Maybe GRASS_VERSION_STRING isn't needed at all?

comment:4 Changed 7 months ago by neteler

The "@(#)" removal has been addressed in https://github.com/OSGeo/grass/pull/528

comment:5 Changed 7 months ago by martinl

Can we close the ticket?

comment:6 Changed 7 months ago by neteler

Resolution: fixed
Status: newclosed

Yes, closing (feel free to reopen if needed)

Note: See TracTickets for help on using tickets.