Opened 2 years ago

Closed 21 months ago

#3739 closed defect (fixed)

lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.

Reported by: pmav99 Owned by: grass-dev@…
Priority: normal Milestone: 7.8.0
Component: Compiling Version: git-releasebranch78
Keywords: ctypes, python3 Cc:
CPU: x86-64 Platform: Linux

Description

On a clean installation in a new VM, after checking out the SVN repo and compiling with python 3.6, svn diff shows that the following file has changed. On the exact same setup using python 2.7 svn diff shows nothing.

If this file is being generated then shouldn't it be ".svnignored" (or whatever that is called on SVN).

This is probably related to #3704 too

Index: lib/python/ctypes/ctypesgencore/parser/lextab.py
===================================================================
--- lib/python/ctypes/ctypesgencore/parser/lextab.py	(revision 74039)
+++ lib/python/ctypes/ctypesgencore/parser/lextab.py	(working copy)
@@ -1,8 +1,8 @@
 # lextab.py. This file automatically created by PLY (version 2.2). Don't edit!
-_lextokens    = {'PP_END_DEFINE': None, 'DIV_ASSIGN': None, 'AND_OP': None, 'PERIOD': None, 'PTR_OP': None, 'PP_MACRO_PARAM': None, 'OTHER': None, 'IDENTIFIER': None, 'PP_STRINGIFY': None, 'XOR_ASSIGN': None, 'PP_DEFINE_NAME': None, 'ELLIPSIS': None, 'INC_OP': None, 'GE_OP': None, 'LEFT_ASSIGN': None, 'MUL_ASSIGN': None, 'RIGHT_ASSIGN': None, 'SUB_ASSIGN': None, 'LPAREN': None, 'PP_IDENTIFIER_PASTE': None, 'PP_DEFINE': None, 'OR_ASSIGN': None, 'ADD_ASSIGN': None, 'HEADER_NAME': None, 'LE_OP': None, 'LEFT_OP': None, 'PP_NUMBER': None, 'DEC_OP': None, 'RIGHT_OP': None, 'MOD_ASSIGN': None, 'STRING_LITERAL': None, 'EQ_OP': None, 'OR_OP': None, 'AND_ASSIGN': None, 'CHARACTER_CONSTANT': None, 'PP_DEFINE_MACRO_NAME': None, 'NE_OP': None, 'NEWLINE': None}
+_lextokens    = {'HEADER_NAME': None, 'IDENTIFIER': None, 'PP_NUMBER': None, 'CHARACTER_CONSTANT': None, 'STRING_LITERAL': None, 'OTHER': None, 'PTR_OP': None, 'INC_OP': None, 'DEC_OP': None, 'LEFT_OP': None, 'RIGHT_OP': None, 'LE_OP': None, 'GE_OP': None, 'EQ_OP': None, 'NE_OP': None, 'AND_OP': None, 'OR_OP': None, 'MUL_ASSIGN': None, 'DIV_ASSIGN': None, 'MOD_ASSIGN': None, 'ADD_ASSIGN': None, 'SUB_ASSIGN': None, 'LEFT_ASSIGN': None, 'RIGHT_ASSIGN': None, 'AND_ASSIGN': None, 'XOR_ASSIGN': None, 'OR_ASSIGN': None, 'PERIOD': None, 'ELLIPSIS': None, 'LPAREN': None, 'NEWLINE': None, 'PP_DEFINE': None, 'PP_DEFINE_NAME': None, 'PP_DEFINE_MACRO_NAME': None, 'PP_MACRO_PARAM': None, 'PP_STRINGIFY': None, 'PP_IDENTIFIER_PASTE': None, 'PP_END_DEFINE': None}
 _lexreflags   = 0
 _lexliterals  = ''
-_lexstateinfo = {'DEFINE': 'exclusive', 'INITIAL': 'inclusive'}
-_lexstatere   = {'DEFINE': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier', 'DEFINE_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'), ('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])], 'INITIAL': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier', 'INITIAL_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'), ('t_INITIAL_pp_define', 'INITIAL_pp_define')])]}
-_lexstateignore = {'DEFINE': ' \t\x0b\x0c\r', 'INITIAL': ' \t\x0b\x0c\r'}
-_lexstateerrorf = {'DEFINE': 't_DEFINE_error', 'INITIAL': 't_INITIAL_error'}
+_lexstateinfo = {'INITIAL': 'inclusive', 'DEFINE': 'exclusive'}
+_lexstatere   = {'INITIAL': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier', 'INITIAL_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'), ('t_INITIAL_pp_define', 'INITIAL_pp_define')])], 'DEFINE': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier', 'DEFINE_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'), ('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])]}
+_lexstateignore = {'INITIAL': ' \t\x0b\x0c\r', 'DEFINE': ' \t\x0b\x0c\r'}
+_lexstateerrorf = {'INITIAL': 't_INITIAL_error', 'DEFINE': 't_DEFINE_error'}

Change History (9)

comment:1 Changed 2 years ago by pmav99

As far as _lextokens go, the only difference is the key ordering. So, this might has to do with Python's dictionaries keeping the "insertion order" in Python 3.6+. _lexstatere has different values too, though.

comment:2 Changed 2 years ago by martinl

Milestone: 7.8.0

comment:3 Changed 2 years ago by neteler

Issue confirmed also on Fedora 29 with Python-3.7

@pmav99: do you suggest to commit the modified lextab.py since it mainly comes with a different order?

comment:4 Changed 2 years ago by pmav99

Not really. I don't have much experience with ctypes, but I am under the impression that this will file will be different under each python version (2.7 vs 3.6/3.7). Maybe even under different minor versions too but I haven't checked it.

So if it gets committed for Python 3, I guess that there is going to be a dirty svn diff under Python 2.

Furthermore, since this file is clearly changing when GRASS is compiled, I am not really sure why it is added to the repo, instead of getting generated upon compilation.

Finally, there have already been 2 instances where the file was accidentally added to the repo, only to be removed right away: https://trac.osgeo.org/grass/log/grass/trunk/lib/python/ctypes/ctypesgencore?rev=74047

So, before deciding on a course of action, I think it is better to further investigate how it is being used/generated. This might provide some insight: #3331

comment:5 Changed 2 years ago by pmav99

Just checked with 3.6 vs 3.7. There is no difference between them (i.e. they produce the same svn diff).

comment:6 Changed 2 years ago by pmav99

For the record, this seems to be a maintained ctypesgen fork: https://github.com/olsonse/ctypesgen

comment:7 in reply to:  6 Changed 2 years ago by neteler

Keywords: ctypes python3 added

Replying to pmav99:

For the record, this seems to be a maintained ctypesgen fork: https://github.com/olsonse/ctypesgen

It is worth trying? Did anyone do so?

comment:8 in reply to:  6 Changed 21 months ago by neteler

Replying to pmav99:

For the record, this seems to be a maintained ctypesgen fork: https://github.com/olsonse/ctypesgen

See related #3900.

comment:9 Changed 21 months ago by neteler

Resolution: fixed
Status: newclosed
Version: svn-trunkgit-releasebranch78

Meanwhile lib/python/ctypes/ctypesgencore/parser/lextab.py has been removed from the source code and thing now work.

Closing.

Note: See TracTickets for help on using tickets.