Opened 7 years ago
Closed 6 years ago
#3446 closed defect (fixed)
GRASS python library is not working with python 3
Reported by: | lrntct | Owned by: | |
---|---|---|---|
Priority: | major | Milestone: | 7.8.0 |
Component: | Python | Version: | 7.2.1 |
Keywords: | python python3 ctypes, python 3.7 | Cc: | |
CPU: | x86-64 | Platform: | Linux |
Description
Various pygrass modules fail to import in Python 3.6.
Python version:
Python 3.6.3 (default, Oct 3 2017, 21:45:48) [GCC 7.2.0] on linux
For example:
import grass.pygrass.gis Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/grass72/etc/python/grass/pygrass/gis/__init__.py", line 14, in <module> libgis.G_gisinit('') File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit return (G__gisinit (GIS_H_VERSION, pgm)) ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
This issue prevents pygrass and the temporal framework to work with python3:
>>> import grass.temporal as tgis Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/grass72/etc/python/grass/temporal/__init__.py", line 3, in <module> from .core import * File "/usr/lib/grass72/etc/python/grass/temporal/core.py", line 45, in <module> from .c_libraries_interface import * File "/usr/lib/grass72/etc/python/grass/temporal/c_libraries_interface.py", line 28, in <module> from grass.pygrass.rpc.base import RPCServerBase File "/usr/lib/grass72/etc/python/grass/pygrass/rpc/__init__.py", line 21, in <module> from grass.pygrass.vector import * File "/usr/lib/grass72/etc/python/grass/pygrass/vector/__init__.py", line 6, in <module> libgis.G_gisinit('') File "/usr/lib/grass72/etc/python/grass/lib/gis.py", line 3489, in G_gisinit return (G__gisinit (GIS_H_VERSION, pgm)) ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type
Change History (24)
comment:1 by , 7 years ago
Keywords: | ctypes added; temporal removed |
---|
follow-up: 4 comment:2 by , 7 years ago
Just FYI - upstream there are a few new Python 3 fixes as PR available:
follow-up: 5 comment:4 by , 7 years ago
Replying to neteler:
Just FYI - upstream there are a few new Python 3 fixes as PR available:
If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see https://github.com/davidjamesca/ctypesgen/issues/52).
comment:5 by , 7 years ago
Replying to lrntct:
Replying to neteler:
Just FYI - upstream there are a few new Python 3 fixes as PR available:
If GRASS relies on ctypesgen, we have a problem, because the project is likely dead. The latest commit was 3 years ago, and the maintainer does not seems interested anymore (see https://github.com/davidjamesca/ctypesgen/issues/52).
Yeah, but the beauty of open source development is that someone else can take over:
https://github.com/olsonse/ctypesgen
"This branch is 154 commits ahead of davidjamesca:master." (see also the many forks at https://github.com/olsonse/ctypesgen/network)
comment:7 by , 7 years ago
BTW, currently ctypes compilation with Python3 enabled fails with
python3 -t -m py_compile /var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py File "/var/local/src/grass7_trunk/dist.x86_64-pc-linux-gnu/etc/python/grass/lib/vector.py", line 5779 PORT_LONG_MAX = 2147483647L ^ SyntaxError: invalid syntax
comment:8 by , 7 years ago
Hi, do you have any directions on how to test and debug pygrass
build against python3
? Which version/branch of ctypegen
? Is anyone from the dev team in contact (open issues?) with ctypegen devs?
comment:9 by , 6 years ago
Milestone: | 7.4.1 → 7.4.2 |
---|
comment:10 by , 6 years ago
This fork seems to contain Python3 changes: https://github.com/billhill00/ctypesgen
comment:11 by , 6 years ago
Milestone: | 7.4.2 → 7.8.0 |
---|
follow-up: 18 comment:12 by , 6 years ago
For record, currently compilation fails (using Python 3.7) here
Status: gcc -E -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -I/opt/src/grass7_trunk/dist.x86_64-pc-linux-gnu/include -D__GLIBC_HAVE_LONG_LONG -U __GNUC__ -dD "-Dinline=" "-D__inline__=" "-D__extension__=" "-D_Bool=uint8_t" "-D__const=const" "-D__asm__(x)=" "-D__asm(x)=" "-DCTYPESGEN=1" "/tmp/tmpi1g2g39s.h" Traceback (most recent call last): File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1021, in parse_template this = chr(ESCAPES[this][1]) KeyError: '\\x' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "./ctypesgen.py", line 162, in <module> descriptions = ctypesgencore.parser.parse(options.headers, options) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/__init__.py", line 22, in parse parser.parse() File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/datacollectingparser.py", line 74, in parse ctypesparser.CtypesParser.parse(self, fname, False) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/cparser.py", line 122, in parse self.preprocessor_parser.parse(filename) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 226, in parse token = self.lexer.token() File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/preprocessor.py", line 65, in token result = lex.Lexer.token(self) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/lex.py", line 359, in token newtok = func(tok) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 286, in t_ANY_string_literal t.value = StringLiteral(encode(t.value)) File "/opt/src/grass7_trunk/lib/python/ctypes/ctypesgencore/parser/pplexer.py", line 84, in __new__ r'\x0\1', value[1:-1]) File "/home/martin/python3-grass7/lib/python3.7/re.py", line 192, in sub return _compile(pattern, flags).sub(repl, string, count) File "/home/martin/python3-grass7/lib/python3.7/re.py", line 309, in _subx template = _compile_repl(template, pattern) File "/home/martin/python3-grass7/lib/python3.7/re.py", line 300, in _compile_repl return sre_parse.parse_template(repl, pattern) File "/home/martin/python3-grass7/lib/python3.7/sre_parse.py", line 1024, in parse_template raise s.error('bad escape %s' % this, len(this)) re.error: bad escape \x at position 0 make[1]: *** [Makefile:102: OBJ.x86_64-pc-linux-gnu/date.py] Error 1 make[1]: Leaving directory '/opt/src/grass7_trunk/lib/python/ctypes' make: *** [Makefile:81: default] Error 2
comment:13 by , 6 years ago
Keywords: | 3.7 added |
---|
Good news: compilation does not fail with Python 3.6.
follow-up: 15 comment:14 by , 6 years ago
@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!
follow-up: 16 comment:15 by , 6 years ago
Replying to epifanio:
@martinl do you have any direction on how to test a grass build against python3? @all, will be worth it to have a dedicated wiki page for it? Thanks!!!
see
follow-up: 17 comment:16 by , 6 years ago
Replying to hellik:
instead of building wx package I used system packages (python3-wx and python3-six) + virtualenv --system-site-packages
.
comment:17 by , 6 years ago
comment:18 by , 6 years ago
follow-up: 20 comment:19 by , 6 years ago
Now I have an issue with g.extension:
GRASS 7.7.svn (utm32N):~ > python --version Python 3.7.1 GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)... Traceback (most recent call last): File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module> sys.exit(main()) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main install_extension(source=source, url=url, xmlurl=xmlurl) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension source=source, url=url) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms outdev=outdev, directory=srcdir, tmpdir=TMPDIR) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code fix_newlines(directory) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines if '\0' in data: TypeError: a bytes-like object is required, not 'str'
Likewise with other addon modules to be installed...
follow-up: 21 comment:20 by , 6 years ago
Replying to neteler:
Now I have an issue with g.extension:
GRASS 7.7.svn (utm32N):~ > python --version Python 3.7.1 GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf Fetching <i.fusion.hpf> from GRASS GIS Addons repository (be patient)... Traceback (most recent call last): File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1758, in <module> sys.exit(main()) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1737, in main install_extension(source=source, url=url, xmlurl=xmlurl) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 697, in install_extension source=source, url=url) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1160, in install_extension_std_platforms outdev=outdev, directory=srcdir, tmpdir=TMPDIR) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1113, in download_source_code fix_newlines(directory) File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines if '\0' in data: TypeError: a bytes-like object is required, not 'str'Likewise with other addon modules to be installed...
I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3
PS I think the Python3 support will required major release also to use the updated addons
comment:21 by , 6 years ago
Replying to lucadelu:
Replying to neteler:
Now I have an issue with g.extension:
GRASS 7.7.svn (utm32N):~ > python --version Python 3.7.1 GRASS 7.7.svn (utm32N):~ > g.extension i.fusion.hpf ... File "/home/mneteler/software/grass77/dist.x86_64-pc-linux-gnu/scripts/g.extension", line 1043, in fix_newlines if '\0' in data: TypeError: a bytes-like object is required, not 'str'Likewise with other addon modules to be installed...
I think most of the addons since they are mostly written for Python >= 2.6 We need to port the addons to python3
I had updated some and it also fails with those.
To me it looks like a bug in g.extension itself:
https://trac.osgeo.org/grass/browser/grass/trunk/scripts/g.extension/g.extension.py#L1042
When I delete these lines
#if '\0' in data: # continue # ignore binary files
then g.extension works fine.
comment:23 by , 6 years ago
g.extension
changed in r73793 in order to support Python3. Would be better to create for new issues separted tickets. This ticket seems to be to generic. I would suggest to close it since originally reported issue has been solved and remaining issue (Python 3.7) is tracked by a new issue #3704. I took liberty to close the ticket. Feel free to re-open if needed.
comment:24 by , 6 years ago
Resolution: | → fixed |
---|---|
Status: | new → closed |
Just remove the keyword: temporal and add ctypes, since the issue is due to ctypes and affect all the library based on them: temporal, pygrass.