Opened 2 years ago

Closed 11 months ago

#3446 closed defect (fixed)

GRASS python library is not working with python 3

Reported by: lrntct Owned by: grass-dev@…
Priority: major Milestone: 7.8.0
Component: Python Version: 7.2.1
Keywords: python python3 ctypes, python 3.7 Cc:
CPU: x86-64 Platform: Linux

Description

Various pygrass modules fail to import in Python 3.6.

Python version:

Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
[GCC 7.2.0] on linux

For example:

import grass.pygrass.gis
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/grass72/etc/python/grass/pygrass/gis/__init__.py", line 14, in <module>
    libgis.G_gisinit('')
  File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit
    return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

This issue prevents pygrass and the temporal framework to work with python3:

>>> import grass.temporal as tgis
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/grass72/etc/python/grass/temporal/__init__.py", line 3, in <module>
    from .core import *
  File "/usr/lib/grass72/etc/python/grass/temporal/core.py", line 45, in <module>
    from .c_libraries_interface import *
  File "/usr/lib/grass72/etc/python/grass/temporal/c_libraries_interface.py", line 28, in <module>
    from grass.pygrass.rpc.base import RPCServerBase
  File "/usr/lib/grass72/etc/python/grass/pygrass/rpc/__init__.py", line 21, in <module>
    from grass.pygrass.vector import *
  File "/usr/lib/grass72/etc/python/grass/pygrass/vector/__init__.py", line 6, in <module>
    libgis.G_gisinit('')
  File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit
    return (G__gisinit (GIS_H_VERSION, pgm))
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

Change History (24)

comment:1 Changed 2 years ago by zarch

Keywords: ctypes added; temporal removed

Just remove the keyword: temporal and add ctypes, since the issue is due to ctypes and affect all the library based on them: temporal, pygrass.

comment:2 Changed 2 years ago by neteler

Just FYI - upstream there are a few new Python 3 fixes as PR available:

https://github.com/davidjamesca/ctypesgen/pull/58

comment:3 Changed 2 years ago by neteler

See also #2708

comment:4 in reply to:  2 ; Changed 2 years ago by lrntct

Replying to neteler:

Just FYI - upstream there are a few new Python 3 fixes as PR available:

https://github.com/davidjamesca/ctypesgen/pull/58

If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see https://github.com/davidjamesca/ctypesgen/issues/52).

comment:5 in reply to:  4 Changed 2 years ago by neteler

Replying to lrntct:

Replying to neteler:

Just FYI - upstream there are a few new Python 3 fixes as PR available:

https://github.com/davidjamesca/ctypesgen/pull/58

If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see https://github.com/davidjamesca/ctypesgen/issues/52).

Yeah, but the beauty of open source development is that someone else can take over:

https://github.com/olsonse/ctypesgen

"This branch is 154 commits ahead of davidjamesca:master." (see also the many forks at https://github.com/olsonse/ctypesgen/network)

comment:6 Changed 22 months ago by neteler

Milestone: 7.4.07.4.1

Ticket retargeted after milestone closed

comment:7 Changed 20 months ago by martinl

BTW, currently ctypes compilation with Python3 enabled fails with

python3 -t -m py_compile /var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py
  File "/var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py", line 5779
    PORT_LONG_MAX = 2147483647L
                              ^
SyntaxError: invalid syntax

comment:8 Changed 20 months ago by epifanio

Hi, do you have any directions on how to test and debug pygrass build against python3? Which version/branch of ctypegen? Is anyone from the dev team in contact (open issues?) with ctypegen devs? It seems there is no interest from the ctypegens dev in working on a py3 update https://github.com/davidjamesca/ctypesgen/issues/51 point to a wontfix and dead-links old prs. I can there is work done here by zarc here: https://github.com/zarch/ctypesgen/tree/pep8andpy3 Zarc what's the state of pygrass with py3?

Last edited 20 months ago by epifanio (previous) (diff)

comment:9 Changed 17 months ago by neteler

Milestone: 7.4.17.4.2

comment:10 Changed 14 months ago by neteler

This fork seems to contain Python3 changes: https://github.com/billhill00/ctypesgen

comment:11 Changed 14 months ago by martinl

Milestone: 7.4.27.8.0

comment:12 Changed 14 months ago by martinl

For record, currently compilation fails (using Python 3.7) here

Status: gcc -E       -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline=" "-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpi1g2g39s.h"
Traceback (most recent call last):
  File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1021, in parse_template
    this = chr(ESCAPES[this][1])
KeyError: '\\x'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./ctypesgen.py", line 162, in <module>
    descriptions = ctypesgencore.parser.parse(options.headers, options)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse
    parser.parse()
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse
    ctypesparser.CtypesParser.parse(self, fname, False)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse
    self.preprocessor_parser.parse(filename)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 226, in parse
    token = self.lexer.token()
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 65, in token
    result = lex.Lexer.token(self)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/lex.py", line 359, in token
    newtok = func(tok)
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 286, in t_ANY_string_literal
    t.value = StringLiteral(encode(t.value))
  File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 84, in __new__
    r'\x0\1', value[1:-1])
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 192, in sub
    return _compile(pattern, flags).sub(repl, string, count)
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 309, in _subx
    template = _compile_repl(template, pattern)
  File "/home/martin/python3-grass7/lib/python3.7/re.py", line 300, in _compile_repl
    return sre_parse.parse_template(repl, pattern)
  File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1024, in parse_template
    raise s.error('bad escape %s' % this, len(this))
re.error: bad escape \x at position 0
make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1
make[1]: Leaving directory '/opt/src/grass7_trunk/lib/python/ctypes'
make: *** [Makefile:81: default] Error 2

comment:13 Changed 14 months ago by martinl

Keywords: 3.7 added

Good news: compilation does not fail with Python 3.6.

comment:14 Changed 14 months ago by epifanio

@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!

comment:15 in reply to:  14 ; Changed 14 months ago by hellik

Replying to epifanio:

@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!

see

https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

comment:16 in reply to:  15 ; Changed 14 months ago by martinl

Replying to hellik:

https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

instead of building wx package I used system packages (python3-wx and python3-six) + virtualenv --system-site-packages.

comment:17 in reply to:  16 Changed 14 months ago by martinl

Replying to martinl:

Replying to hellik:

https://trac.osgeo.org/grass/wiki/Python3Support#Howtotest

instead of building wx package I used system packages (python3-wx and python3-six) + virtualenv --system-site-packages.

I have updated wiki to describe also quick system-side approach.

comment:18 in reply to:  12 Changed 11 months ago by neteler

Replying to martinl:

For record, currently compilation fails (using Python 3.7) here

For a related patch attempt, see #3704: ctypes compilation error with Python-3.7

comment:19 Changed 11 months ago by neteler

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
Traceback (most recent call last):
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module>
    sys.exit(main())
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension
    source=source, url=url)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
    outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code
    fix_newlines(directory)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

comment:20 in reply to:  19 ; Changed 11 months ago by lucadelu

Replying to neteler:

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)...
Traceback (most recent call last):
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module>
    sys.exit(main())
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main
    install_extension(source=source, url=url, xmlurl=xmlurl)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension
    source=source, url=url)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms
    outdev=outdev, directory=srcdir, tmpdir=TMPDIR)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code
    fix_newlines(directory)
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3

PS I think the Python3 support will required major release also to use the updated addons

comment:21 in reply to:  20 Changed 11 months ago by neteler

Replying to lucadelu:

Replying to neteler:

Now I have an issue with g.extension:

GRASS 7.7.svn (utm32N):~ > python --version
Python 3.7.1

GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf
...
  File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines
    if '\0' in data:
TypeError: a bytes-like object is required, not 'str'

Likewise with other addon modules to be installed...

I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3

I had updated some and it also fails with those.

To me it looks like a bug in g.extension itself:

https://trac.osgeo.org/grass/browser/grass/trunk/scripts/g.extension/g.extension.py#L1042

When I delete these lines

            #if '\0' in data:
            #    continue  # ignore binary files

then g.extension works fine.

comment:22 Changed 11 months ago by martinl

In 73793:

fix g.extension for python3, see #3446

comment:23 Changed 11 months ago by martinl

g.extension changed in r73793 in order to support Python3. Would be better to create for new issues separted tickets. This ticket seems to be to generic. I would suggest to close it since originally reported issue has been solved and remaining issue (Python 3.7) is tracked by a new issue #3704. I took liberty to close the ticket. Feel free to re-open if needed.

comment:24 Changed 11 months ago by martinl

Resolution: fixed
Status: newclosed
Note: See TracTickets for help on using tickets.