GRASS python library is not working with python 3

[lrntct]

Various pygrass modules fail to import in Python 3.6.

Python version:

Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
[GCC 7.2.0] on linux

For example:

import grass.pygrass.gis
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/grass72/etc/python/grass/pygrass/gis/__init__.py", line 14, in <module>
    libgis.G_gisinit('')
  File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit
    return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

This issue prevents pygrass and the temporal framework to work with python3:

>>> import grass.temporal as tgis
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/grass72/etc/python/grass/temporal/__init__.py", line 3, in <module>
    from .core import *
  File "/usr/lib/grass72/etc/python/grass/temporal/core.py", line 45, in <module>
    from .c_libraries_interface import *
  File "/usr/lib/grass72/etc/python/grass/temporal/c_libraries_interface.py", line 28, in <module>
    from grass.pygrass.rpc.base import RPCServerBase
  File "/usr/lib/grass72/etc/python/grass/pygrass/rpc/__init__.py", line 21, in <module>
    from grass.pygrass.vector import *
  File "/usr/lib/grass72/etc/python/grass/pygrass/vector/__init__.py", line 6, in <module>
    libgis.G_gisinit('')
  File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit
    return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

[zarch]

Just remove the keyword: temporal and add ctypes, since the issue is due to ctypes and affect all the library based on them: temporal, pygrass.

[neteler]

Just FYI - upstream there are a few new Python 3 fixes as PR available:

​https://github.com/davidjamesca/ctypesgen/pull/58

[neteler]

See also #2708

[lrntct]

Replying to neteler:

Just FYI - upstream there are a few new Python 3 fixes as PR available:

​https://github.com/davidjamesca/ctypesgen/pull/58

If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see ​https://github.com/davidjamesca/ctypesgen/issues/52).

[neteler]

Replying to lrntct:

Replying to neteler:

Just FYI - upstream there are a few new Python 3 fixes as PR available:

​https://github.com/davidjamesca/ctypesgen/pull/58

If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see ​https://github.com/davidjamesca/ctypesgen/issues/52).

Yeah, but the beauty of open source development is that someone else can take over:

​https://github.com/olsonse/ctypesgen

"This branch is 154 commits ahead of davidjamesca:master." (see also the many forks at ​https://github.com/olsonse/ctypesgen/network)

[neteler]

Ticket retargeted after milestone closed

[martinl]

BTW, currently ctypes compilation with Python3 enabled fails with

python3 -t -m py_compile /var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py
  File "/var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py", line 5779
    PORT_LONG_MAX = 2147483647L
                              ^
SyntaxError: invalid syntax

[epifanio]

Hi, do you have any directions on how to test and debug pygrass build against python3? Which version/branch of ctypegen? Is anyone from the dev team in contact (open issues?) with ctypegen devs? It seems there is no interest from the ctypegens dev in working on a py3 update ​https://github.com/davidjamesca/ctypesgen/issues/51 point to a wontfix and dead-links old prs. I can there is work done here by zarc here: ​https://github.com/zarch/ctypesgen/tree/pep8andpy3 Zarc what's the state of pygrass with py3?

[neteler]

This fork seems to contain Python3 changes: ​https://github.com/billhill00/ctypesgen

[martinl]

For record, currently compilation fails (using Python 3.7) here

Status: gcc -E       -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline=" "-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpi1g2g39s.h"
Traceback (most recent call last):
  File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1021, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 226, in parse
    token = self.lexer.token()
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 65, in token
    result = lex.Lexer.token(self)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/lex.py", line 359, in token
    newtok = func(tok)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 286, in t_ANY_string_literal
    t.value = StringLiteral(encode(t.value))
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 84, in __new__
    r'\x0\1', value[1:-1])
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 309, in _subx
    template = _compile_repl(template, pattern)
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 300, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1024, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \x at position 0
make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1
make[1]: Leaving directory '/opt/src/grass7_trunk/lib/python/ctypes'
make: *** [Makefile:81: default] Error 2

[martinl]

Good news: compilation does not fail with Python 3.6.

[epifanio]

@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!

[hellik]

Replying to epifanio:

@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!

see

​https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

[martinl]

Replying to hellik:

​https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

instead of building wx package I used system packages (python3-wx and python3-six) + virtualenv --system-site-packages.

[martinl]

Replying to martinl:

Replying to hellik:

​https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

instead of building wx package I used system packages (python3-wx and python3-six) + virtualenv --system-site-packages.

I have updated wiki to describe also quick system-side approach.

[neteler]

Replying to martinl:

For record, currently compilation fails (using Python 3.7) here

For a related patch attempt, see #3704: ctypes compilation error with Python-3.7

[neteler]

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
Traceback (most recent call last):
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module>
    sys.exit(main())
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension
    source=source, url=url)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
    outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code
    fix_newlines(directory)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

[lucadelu]

Replying to neteler:

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
Traceback (most recent call last):
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module>
    sys.exit(main())
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension
    source=source, url=url)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
    outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code
    fix_newlines(directory)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3

PS I think the Python3 support will required major release also to use the updated addons

[neteler]

Replying to lucadelu:

Replying to neteler:

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
...
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3

I had updated some and it also fails with those.

To me it looks like a bug in g.extension itself:

​https://trac.osgeo.org/grass/browser/grass/trunk/scripts/g.extension/g.extension.py#L1042

When I delete these lines

            #if '\0' in data:
            #    continue  # ignore binary files

then g.extension works fine.

[martinl]

In 73793:

fix g.extension for python3, see #3446

[martinl]

g.extension changed in r73793 in order to support Python3. Would be better to create for new issues separted tickets. This ticket seems to be to generic. I would suggest to close it since originally reported issue has been solved and remaining issue (Python 3.7) is tracked by a new issue #3704. I took liberty to close the ticket. Feel free to re-open if needed.