ctypes compilation error with Python-3.7

[neteler]

I checked compilation of trunk with Python3 (using virtualenv) and get a compile error in ctypes:

(grasspy3) [mneteler@oboe ctypes ]$ cd /home/mneteler/software/grass77/lib/python/ctypes/

(grasspy3) [mneteler@oboe ctypes ]$ make
make /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib
make[1]: Entering directory '/home/mneteler/software/grass77/lib/python/ctypes'
make[1]: '/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib' is up to date.
make[1]: Leaving directory '/home/mneteler/software/grass77/lib/python/ctypes'
make OBJ.x86_64-pc-linux-gnu/date.py OBJ.x86_64-pc-linux-gnu/gis.py OBJ.x86_64-pc-linux-gnu/raster.py OBJ.x86_64-pc-linux-gnu/gmath.py OBJ.x86_64-pc-linux-gnu/proj.py OBJ.x86_64-pc-linux-gnu/imagery.py OBJ.x86_64-pc-linux-gnu/vector.py OBJ.x86_64-pc-linux-gnu/rtree.py OBJ.x86_64-pc-linux-gnu/display.py OBJ.x86_64-pc-linux-gnu/stats.py OBJ.x86_64-pc-linux-gnu/dbmi.py OBJ.x86_64-pc-linux-gnu/raster3d.py OBJ.x86_64-pc-linux-gnu/arraystats.py OBJ.x86_64-pc-linux-gnu/cluster.py OBJ.x86_64-pc-linux-gnu/vedit.py OBJ.x86_64-pc-linux-gnu/segment.py OBJ.x86_64-pc-linux-gnu/rowio.py OBJ.x86_64-pc-linux-gnu/temporal.py OBJ.x86_64-pc-linux-gnu/ogsf.py OBJ.x86_64-pc-linux-gnu/nviz.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/date.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gis.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gmath.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/proj.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/imagery.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rtree.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/display.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/stats.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/dbmi.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster3d.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/arraystats.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/cluster.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vedit.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/segment.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rowio.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/temporal.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ogsf.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/nviz.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/__init__.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ctypes_preamble.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.py /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/date.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gis.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/gmath.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/proj.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/imagery.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rtree.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/display.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/stats.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/dbmi.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/raster3d.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/arraystats.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/cluster.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vedit.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/segment.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/rowio.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/temporal.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ogsf.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/nviz.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/__init__.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ctypes_preamble.pyc /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/ctypes_loader.pyc
make[1]: Entering directory '/home/mneteler/software/grass77/lib/python/ctypes'
GISRC=/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/demolocation/.grassrc77 GISBASE=/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu PATH="/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts:$PATH" PYTHONPATH="/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/etc/python:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/gui/wxpython:$PYTHONPATH" LD_LIBRARY_PATH="/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/bin:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/lib:/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/lib:" LC_ALL=C LANG=C LANGUAGE=C ./ctypesgen.py --cpp "gcc -E       -I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include -I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG" -lgrass_datetime.7.7.svn   /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include/grass/datetime.h /home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include/grass/defs/datetime.h -o OBJ.x86_64-pc-linux-gnu/date.py
Status: Preprocessing /home/mneteler/tmp/tmpzwarkrsu.h
Status: gcc -E       -I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include -I/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline=" "-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/home/mneteler/tmp/tmpzwarkrsu.h"
Traceback (most recent call last):
  File "/usr/lib64/python3.7/sre_parse.py", line 1021, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 222, in parse
    token = self.lexer.token()
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 63, in token
    result = lex.Lexer.token(self)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/lex.py", line 355, in token
    newtok = func(tok)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 284, in t_ANY_string_literal
    t.value = StringLiteral(t.value)
  File "/home/mneteler/software/grass77/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 82, in __new__
    r'\x0\1', value[1:-1])
  File "/usr/lib64/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/usr/lib64/python3.7/re.py", line 309, in _subx
    template = _compile_repl(template, pattern)
  File "/usr/lib64/python3.7/re.py", line 300, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/usr/lib64/python3.7/sre_parse.py", line 1024, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \x at position 0
make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1
make[1]: Leaving directory '/home/mneteler/software/grass77/lib/python/ctypes'
make: *** [Makefile:81: default] Error 2

Any idea how to fix that?

[martinl]

Python 3.7 only related, see ​https://trac.osgeo.org/grass/ticket/3446#comment:12

[neteler]

Inspired by

• bug: ​https://sourceforge.net/p/sbml/libsbml/461/
  • --> fix: ​https://sourceforge.net/p/sbml/code/25372/
• and fix: ​https://github.com/wbsoft/frescobaldi/commit/c27a9c24249cb19cbf99de7d7bfc46c561421539

... I have created attached patch attempt to fix the compilation issue with Python-3.7. Seems to work but testing/checks before applying is needed.

[neteler]

Patch to fix ctypesgencore pplexer.py compilation error with Python3.7

[martinl]

In 73827:

ctypes compilation error with Python-3.7, closes #3704

[martinl]

I took liberty to apply the patch to trunk. It's working for me.

[hellik]

Replying to martinl:

I took liberty to apply the patch to trunk. It's working for me.

tested winGRASS compilation with Python 3.7.2, seems to work here too.