lib/python/ctypes/ctypesgencore/parser/lextab.py changes after compilation.

[pmav99]

On a clean installation in a new VM, after checking out the SVN repo and compiling with python 3.6, svn diff shows that the following file has changed. On the exact same setup using python 2.7 svn diff shows nothing.

If this file is being generated then shouldn't it be ".svnignored" (or whatever that is called on SVN).

This is probably related to #3704 too

Index: lib/python/ctypes/ctypesgencore/parser/lextab.py
===================================================================
--- lib/python/ctypes/ctypesgencore/parser/lextab.py	(revision 74039)
+++ lib/python/ctypes/ctypesgencore/parser/lextab.py	(working copy)
@@ -1,8 +1,8 @@
 # lextab.py. This file automatically created by PLY (version 2.2). Don't edit!
-_lextokens    = {'PP_END_DEFINE': None, 'DIV_ASSIGN': None, 'AND_OP': None, 'PERIOD': None, 'PTR_OP': None, 'PP_MACRO_PARAM': None, 'OTHER': None, 'IDENTIFIER': None, 'PP_STRINGIFY': None, 'XOR_ASSIGN': None, 'PP_DEFINE_NAME': None, 'ELLIPSIS': None, 'INC_OP': None, 'GE_OP': None, 'LEFT_ASSIGN': None, 'MUL_ASSIGN': None, 'RIGHT_ASSIGN': None, 'SUB_ASSIGN': None, 'LPAREN': None, 'PP_IDENTIFIER_PASTE': None, 'PP_DEFINE': None, 'OR_ASSIGN': None, 'ADD_ASSIGN': None, 'HEADER_NAME': None, 'LE_OP': None, 'LEFT_OP': None, 'PP_NUMBER': None, 'DEC_OP': None, 'RIGHT_OP': None, 'MOD_ASSIGN': None, 'STRING_LITERAL': None, 'EQ_OP': None, 'OR_OP': None, 'AND_ASSIGN': None, 'CHARACTER_CONSTANT': None, 'PP_DEFINE_MACRO_NAME': None, 'NE_OP': None, 'NEWLINE': None}
+_lextokens    = {'HEADER_NAME': None, 'IDENTIFIER': None, 'PP_NUMBER': None, 'CHARACTER_CONSTANT': None, 'STRING_LITERAL': None, 'OTHER': None, 'PTR_OP': None, 'INC_OP': None, 'DEC_OP': None, 'LEFT_OP': None, 'RIGHT_OP': None, 'LE_OP': None, 'GE_OP': None, 'EQ_OP': None, 'NE_OP': None, 'AND_OP': None, 'OR_OP': None, 'MUL_ASSIGN': None, 'DIV_ASSIGN': None, 'MOD_ASSIGN': None, 'ADD_ASSIGN': None, 'SUB_ASSIGN': None, 'LEFT_ASSIGN': None, 'RIGHT_ASSIGN': None, 'AND_ASSIGN': None, 'XOR_ASSIGN': None, 'OR_ASSIGN': None, 'PERIOD': None, 'ELLIPSIS': None, 'LPAREN': None, 'NEWLINE': None, 'PP_DEFINE': None, 'PP_DEFINE_NAME': None, 'PP_DEFINE_MACRO_NAME': None, 'PP_MACRO_PARAM': None, 'PP_STRINGIFY': None, 'PP_IDENTIFIER_PASTE': None, 'PP_END_DEFINE': None}
 _lexreflags   = 0
 _lexliterals  = ''
-_lexstateinfo = {'DEFINE': 'exclusive', 'INITIAL': 'inclusive'}
-_lexstatere   = {'DEFINE': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier', 'DEFINE_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'), ('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])], 'INITIAL': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\|\\||\\+\\+|\\*=|\\|=|\\+=|>>=|\\^=|<<=|\\)|-=|<%|<<|%>|\\+|>>|<=|/=|->|!=|&&|\\*|\\[|\\?|--|>=|\\.|:>|&=|==|<:|%=|\\||\\^|/|:|{|~|,|>|-|%|}|;|!|=|&|]|<))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier', 'INITIAL_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'), ('t_INITIAL_pp_define', 'INITIAL_pp_define')])]}
-_lexstateignore = {'DEFINE': ' \t\x0b\x0c\r', 'INITIAL': ' \t\x0b\x0c\r'}
-_lexstateerrorf = {'DEFINE': 't_DEFINE_error', 'INITIAL': 't_INITIAL_error'}
+_lexstateinfo = {'INITIAL': 'inclusive', 'DEFINE': 'exclusive'}
+_lexstatere   = {'INITIAL': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_INITIAL_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_INITIAL_identifier', 'INITIAL_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_INITIAL_newline>\\n)|(?P<t_INITIAL_pp_define>\\#define)', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_INITIAL_newline', 'INITIAL_newline'), ('t_INITIAL_pp_define', 'INITIAL_pp_define')])], 'DEFINE': [('(?P<t_ANY_directive>\\#\\s+(\\d+)\\s+"([^"]+)"[ \\d]*\\n)|(?P<t_ANY_punctuator>(\\.\\.\\.|\\+\\+|\\|\\||>>=|<<=|\\+=|\\*=|\\^=|\\|=|-=|/=|%=|&=|>>|<<|--|->|&&|<=|>=|==|!=|<:|:>|<%|%>|\\)|\\[|\\.|\\+|\\*|\\^|\\||\\?|;|{|}|,|:|=|]|&|!|~|-|/|%|<|>))', [None, ('t_ANY_directive', 'ANY_directive'), None, None, ('t_ANY_punctuator', 'ANY_punctuator')]), ('(?P<t_DEFINE_identifier>[a-zA-Z_]([a-zA-Z_]|[0-9])*)', [None, ('t_DEFINE_identifier', 'DEFINE_identifier')]), ('(?P<t_ANY_float>(?P<p1>[0-9]+)?(?P<dp>[.]?)(?P<p2>(?(p1)[0-9]*|[0-9]+))(?P<exp>(?:[Ee][+-]?[0-9]+)?)(?P<suf>([FfLl]|d[dfl]|D[DFL]|[fFdD][0-9]+x?)?)(?!\\w))', [None, ('t_ANY_float', 'ANY_float'), None, None, None, None, None]), ('(?P<t_ANY_int>(?P<p1>(?:0x[a-fA-F0-9]+)|(?:[0-9]+))(?P<suf>[uUlL]*))', [None, ('t_ANY_int', 'ANY_int'), None, None]), ('(?P<t_ANY_character_constant>L?\'(\\\\.|[^\\\\\'])+\')|(?P<t_ANY_string_literal>L?"(\\\\.|[^\\\\"])*")|(?P<t_ANY_lparen>\\()|(?P<t_DEFINE_newline>\\n)|(?P<t_DEFINE_pp_param_op>(\\#\\#)|(\\#))', [None, ('t_ANY_character_constant', 'ANY_character_constant'), None, ('t_ANY_string_literal', 'ANY_string_literal'), None, ('t_ANY_lparen', 'ANY_lparen'), ('t_DEFINE_newline', 'DEFINE_newline'), ('t_DEFINE_pp_param_op', 'DEFINE_pp_param_op')])]}
+_lexstateignore = {'INITIAL': ' \t\x0b\x0c\r', 'DEFINE': ' \t\x0b\x0c\r'}
+_lexstateerrorf = {'INITIAL': 't_INITIAL_error', 'DEFINE': 't_DEFINE_error'}

[pmav99]

As far as _lextokens go, the only difference is the key ordering. So, this might has to do with Python's dictionaries keeping the "insertion order" in Python 3.6+. _lexstatere has different values too, though.

[neteler]

Issue confirmed also on Fedora 29 with Python-3.7

@pmav99: do you suggest to commit the modified lextab.py since it mainly comes with a different order?

[pmav99]

Not really. I don't have much experience with ctypes, but I am under the impression that this will file will be different under each python version (2.7 vs 3.6/3.7). Maybe even under different minor versions too but I haven't checked it.

So if it gets committed for Python 3, I guess that there is going to be a dirty svn diff under Python 2.

Furthermore, since this file is clearly changing when GRASS is compiled, I am not really sure why it is added to the repo, instead of getting generated upon compilation.

Finally, there have already been 2 instances where the file was accidentally added to the repo, only to be removed right away: ​https://trac.osgeo.org/grass/log/grass/trunk/lib/python/ctypes/ctypesgencore?rev=74047

So, before deciding on a course of action, I think it is better to further investigate how it is being used/generated. This might provide some insight: #3331

[pmav99]

Just checked with 3.6 vs 3.7. There is no difference between them (i.e. they produce the same svn diff).

[pmav99]

For the record, this seems to be a maintained ctypesgen fork: ​https://github.com/olsonse/ctypesgen

[neteler]

Replying to pmav99:

For the record, this seems to be a maintained ctypesgen fork: ​https://github.com/olsonse/ctypesgen

It is worth trying? Did anyone do so?

[neteler]

Replying to pmav99:

For the record, this seems to be a maintained ctypesgen fork: ​https://github.com/olsonse/ctypesgen

See related #3900.

[neteler]

Meanwhile lib/python/ctypes/ctypesgencore/parser/lextab.py has been removed from the source code and thing now work.

Closing.