Context Navigation

← Previous Ticket
Next Ticket →

#3837 closed defect (fixed)

OSGeo4W winGRASS77svn - startup error: TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Reported by:	hellik	Owned by:	grass-dev@…
Priority:	blocker	Milestone:	7.8.0
Component:	Python	Version:	svn-trunk
Keywords:	python3, py3, wingrass	Cc:
CPU:	Unspecified	Platform:	MSWindows

Description

taken from dev ML 1

Hi,

just tried to start the latest OSGeo4W winGRASS77svn daily build:

----
C:\>grass77svn
Starting GRASS GIS...
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\gis_set.py", line 34, in
<module>
    from core import globalvar
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\globalvar.py", line
35, in <module>
    from core.debug import Debug
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 77,
in <module>
    Debug = DebugMsg()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 39,
in __init__
    self.SetLevel()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 45,
in SetLevel
    self.debuglevel = int(grass.gisenv().get('WX_DEBUG', 0))
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 1074, in gisenv
    s = read_command("g.gisenv", flags='n', env=env)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 494, in read_command
    process = pipe_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 463, in pipe_command
    return start_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 393, in start_command
    return Popen(args, **popts)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 54, in __init__
    cmd = shutil_which(args[0])
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
ERROR: Error in GUI startup. See messages above (if any) and if necessary,
please report this error to the GRASS developers.
On systems with package manager, make sure you have the right GUI package,
probably named grass-gui, installed.
To run GRASS GIS in text mode use the --text flag.
Use '--help' for further options
     grass77 --help
See also: https://grass.osgeo.org/grass77/manuals/helptext.html
Exiting...
Drücken Sie eine beliebige Taste . . .
----

anyone any idea?

additional info dev ML 2

C:\>g.version -g
version=7.7.svn
date=2019
revision=r74428M
build_date=2019-04-26
build_platform=x86_64-w64-mingw32
build_off_t_size=8

and dev ML 3

Same error on Vista with same version. Locale is set to Latvia.

Change History (18)

in reply to: description ; follow-up: 2 comment:1 by hellik, 6 years ago

Replying to hellik:

taken from dev ML 1

Hi,

just tried to start the latest OSGeo4W winGRASS77svn daily build:

----
C:\>grass77svn
Starting GRASS GIS...
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\gis_set.py", line 34, in
<module>
    from core import globalvar
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\globalvar.py", line
35, in <module>
    from core.debug import Debug
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 77,
in <module>
    Debug = DebugMsg()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 39,
in __init__
    self.SetLevel()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 45,
in SetLevel
    self.debuglevel = int(grass.gisenv().get('WX_DEBUG', 0))
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 1074, in gisenv
    s = read_command("g.gisenv", flags='n', env=env)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 494, in read_command
    process = pipe_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 463, in pipe_command
    return start_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 393, in start_command
    return Popen(args, **popts)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py",
line 54, in __init__
    cmd = shutil_which(args[0])
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
ERROR: Error in GUI startup. See messages above (if any) and if necessary,
please report this error to the GRASS developers.
On systems with package manager, make sure you have the right GUI package,
probably named grass-gui, installed.
To run GRASS GIS in text mode use the --text flag.
Use '--help' for further options
     grass77 --help
See also: https://grass.osgeo.org/grass77/manuals/helptext.html
Exiting...
Drücken Sie eine beliebige Taste . . .
----

anyone any idea?

additional info dev ML 2

C:\>g.version -g
version=7.7.svn
date=2019
revision=r74428M
build_date=2019-04-26
build_platform=x86_64-w64-mingw32
build_off_t_size=8

and dev ML 3

Same error on Vista with same version. Locale is set to Latvia.

in reply to: 1 ; follow-up: 3 comment:2 by hellik, 6 years ago

Replying to hellik:

see also

https://trac.osgeo.org/grass/ticket/3733

and also see

https://trac.osgeo.org/grass/ticket/3723

in reply to: 2 ; follow-up: 4 comment:3 by hellik, 6 years ago

Replying to hellik:

Replying to hellik:

see also

https://trac.osgeo.org/grass/ticket/3733

and also see

https://trac.osgeo.org/grass/ticket/3723

it seems there is still some encoding/decoding issue

in reply to: 3 comment:4 by hellik, 6 years ago

Replying to hellik:

Replying to hellik:

Replying to hellik:

see also

https://trac.osgeo.org/grass/ticket/3733

and also see

https://trac.osgeo.org/grass/ticket/3723

it seems there is still some encoding/decoding issue

it's around

195	    if sys.platform == "win32":
196	        # The current directory takes precedence on Windows.
197	        if not os.curdir in path:
198	            path.insert(0, os.curdir)
199	
200	        # PATHEXT is necessary to check on Windows (force lowercase)
201	        pathext = list(map(lambda x: encode(x.lower()),
202	                           os.environ.get("PATHEXT", "").split(os.pathsep)))
203	        if b'.py' not in pathext:
204	            # we assume that PATHEXT contains always '.py'
205	            pathext.insert(0, b'.py')
206	        # See if the given file matches any of the expected path extensions.
207	        # This will allow us to short circuit when given "python.exe".
208	        # If it does match, only test that one, otherwise we have to try
209	        # others.
210	        if any(cmd.lower().endswith(ext) for ext in pathext):
211	            files = [cmd]
212	        else:
213	            files = [cmd + ext for ext in pathext]

follow-ups: 6 8 comment:5 by pmav99, 6 years ago

The traceback in the original post indicates that the problematic line is:

    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):

which currently has been changed to (I guess with other changes, too):

    if any(cmd.lower().endswith(ext) for ext in pathext):

Could you please paste an updated traceback?

in reply to: 5 ; follow-up: 7 comment:6 by hellik, 6 years ago

Replying to pmav99:

The traceback in the original post indicates that the problematic line is:
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
which currently has been changed to (I guess with other changes, too):
    if any(cmd.lower().endswith(ext) for ext in pathext):
Could you please paste an updated traceback?

https://trac.osgeo.org/grass/browser/grass/trunk/lib/python/script/core.py#L210

195	    if sys.platform == "win32":
196	        # The current directory takes precedence on Windows.
197	        if not os.curdir in path:
198	            path.insert(0, os.curdir)
199	
200	        # PATHEXT is necessary to check on Windows (force lowercase)
201	        pathext = list(map(lambda x: encode(x.lower()),
202	                           os.environ.get("PATHEXT", "").split(os.pathsep)))
203	        if b'.py' not in pathext:
204	            # we assume that PATHEXT contains always '.py'
205	            pathext.insert(0, b'.py')
206	        # See if the given file matches any of the expected path extensions.
207	        # This will allow us to short circuit when given "python.exe".
208	        # If it does match, only test that one, otherwise we have to try
209	        # others.
210	        if any(cmd.lower().endswith(ext) for ext in pathext):
211	            files = [cmd]
212	        else:
213	            files = [cmd + ext for ext in pathext]

C:\>g.version -breg
version=7.7.svn
date=2019
revision=r74474M
build_date=2019-05-08
build_platform=x86_64-w64-mingw32
build_off_t_size=8

 ./configure  --host=x86_64-w64-mingw32 '--with-libs=/c/msys64/usr/src/grass_trunk/mswindows/osgeo4w/lib ' --with-includes=/c/OSGeo4W64/include --libexecdir=/c/OSGeo4W64/bin --prefix=/c/OSGeo4W64/apps/grass --bindir=/c/OSGeo4W64/bin --includedir=/c/OSGeo4W64/include --without-x --with-cxx --enable-shared --enable-largefile --with-fftw --with-freetype --with-proj-share=/c/OSGeo4W64/share/proj --with-gdal=/c/msys64/usr/src/grass_trunk/mswindows/osgeo4w/gdal-config --with-geos=/c/msys64/usr/src/grass_trunk/mswindows/osgeo4w/geos-config --with-sqlite --with-regex --with-nls --with-freetype-includes=/c/OSGeo4W64/include/freetype2 --with-zstd --with-odbc --with-cairo --with-postgres --with-opengl=windows --with-bzlib --with-liblas=/c/msys64/usr/src/grass_trunk/mswindows/osgeo4w/liblas-config
libgis_revision=74118
libgis_date="2019-02-21 10:38:28 +0100 (Thu, 21 Feb 2019) "
proj4=5.2.0
gdal=2.4.1
geos=3.7.0
sqlite=3.26.0

with r74474

C:\>g.gui wxpython
Launching <wxpython> GUI in the background, please wait...
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass77/gui/wxpython/wxgui.py", line 32, in <module>
    from core import globalvar
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\globalvar.py", line 35, in <module>
    from core.debug import Debug
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 77, in <module>
    Debug = DebugMsg()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 39, in __init__
    self.SetLevel()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 45, in SetLevel
    self.debuglevel = int(grass.gisenv().get('WX_DEBUG', 0))
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 1075, in gisenv
    s = read_command("g.gisenv", flags='n', env=env)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 495, in read_command
    process = pipe_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 464, in pipe_command
    return start_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 394, in start_command
    return Popen(args, **popts)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 54, in __init__
    cmd = shutil_which(args[0])
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

in reply to: 6 comment:7 by hellik, 6 years ago

Replying to hellik

with r74474

C:\>g.gui wxpython
Launching <wxpython> GUI in the background, please wait...
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass77/gui/wxpython/wxgui.py", line 32, in <module>
    from core import globalvar
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\globalvar.py", line 35, in <module>
    from core.debug import Debug
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 77, in <module>
    Debug = DebugMsg()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 39, in __init__
    self.SetLevel()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 45, in SetLevel
    self.debuglevel = int(grass.gisenv().get('WX_DEBUG', 0))
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 1075, in gisenv
    s = read_command("g.gisenv", flags='n', env=env)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 495, in read_command
    process = pipe_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 464, in pipe_command
    return start_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 394, in start_command
    return Popen(args, **popts)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 54, in __init__
    cmd = shutil_which(args[0])
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

starting the GUI

Starting GRASS GIS...
Traceback (most recent call last):
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\gis_set.py", line 34, in <module>
    from core import globalvar
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\globalvar.py", line 35, in <module>
    from core.debug import Debug
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 77, in <module>
    Debug = DebugMsg()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 39, in __init__
    self.SetLevel()
  File "C:\OSGEO4~1\apps\grass\grass77\gui\wxpython\core\debug.py", line 45, in SetLevel
    self.debuglevel = int(grass.gisenv().get('WX_DEBUG', 0))
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 1075, in gisenv
    s = read_command("g.gisenv", flags='n', env=env)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 495, in read_command
    process = pipe_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 464, in pipe_command
    return start_command(*args, **kwargs)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 394, in start_command
    return Popen(args, **popts)
  File "C:\OSGEO4~1\apps\grass\grass77\etc\python\grass\script\core.py", line 54, in __init__
    cmd = shutil_which(args[0])
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str
ERROR: Error in GUI startup. See messages above (if any) and if necessary, please report this error to the GRASS developers.
On systems with package manager, make sure you have the right GUI package, probably named grass-gui, installed.
To run GRASS GIS in text mode use the --text flag.
Use '--help' for further options
     grass77 --help
See also: https://grass.osgeo.org/grass77/manuals/helptext.html
Exiting...
Drücken Sie eine beliebige Taste . . .

in reply to: 5 comment:8 by hellik, 6 years ago

Replying to pmav99:

The traceback in the original post indicates that the problematic line is:
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
which currently has been changed to (I guess with other changes, too):
    if any(cmd.lower().endswith(ext) for ext in pathext):
Could you please paste an updated traceback?

looking into

C:\OSGeo4W64\apps\grass\grass77\etc\python\grass\script\core.py

        if any(cmd.lower().endswith(ext) for ext in pathext):
            files = [cmd]
        else:
            files = [cmd + ext for ext in pathext]

the question is why the traceback reports then:

  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in which
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
  File "C:\OSGEO4~1\apps\Python37\lib\shutil.py", line 1151, in <genexpr>
    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):

follow-up: 11 comment:9 by pmav99, 6 years ago

I am not sure I understand the last question. If you are asking why the same line is being repeated, then, quoting from here:

In Python 3, all comprehensions (and in Python 2 all except list comprehensions) are actually compiled as nested functions. Executing the generator comprehension calls that hidden nested function, using up an extra stack frame.

Generator functions are treated the same way. You can easily check this like this:

$ python -c 'list(1 / i for i in (1, 2, 3, 0, 4, 5))'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <genexpr>
ZeroDivisionError: division by zero

comment:10 by pmav99, 6 years ago

Now to the problem at hand.

    if any(cmd.lower().endswith(ext.lower()) for ext in pathext):
TypeError: endswith first arg must be bytes or a tuple of bytes, not str

Pathext is an iterable (in this case a list) and the problem seems to be that at least one of the pathext elements is a string instead of a bytes object. Now, I don't understand why this happens, since lines 200-201 should handle the conversion to bytes. Anyway, to debug this try to add the following line just before the problematic one:

import pprint
pprint.pprint([ext, type(ext) for ext in pathext])

in reply to: 9 comment:11 by hellik, 6 years ago

Replying to pmav99:

I am not sure I understand the last question. If you are asking why the same line is being repeated, then, quoting from here:

it's not the repeated Line, but in the script

if any(cmd.lower().endswith(ext) for ext in pathext):

and in the traceback

if any(cmd.lower().endswith(ext.lower()) for ext in pathext):

do you see the difference?

see your remark above.

follow-up: 13 comment:12 by pmav99, 6 years ago

OK, I see what is going on now.

You are using Python 3.7. As a a result the shutil_which name is an alias to shutil.which function defined in the standard library.

On Python 2 though, shutil_which is the function defined in scripts/core.py.

This is why there is this difference. The function in scripts.core is never executed in Python 3.

in reply to: 12 comment:13 by hellik, 6 years ago

Replying to pmav99:

OK, I see what is going on now.

You are using Python 3.7.

yes. OSGeo4W-winGRASS7.7.svn is compiled/built against Python 3.x as a test build for preparing the upcoming OSGeo4W-winGRASS7.8

As a a result the shutil_which name is an alias to shutil.which function defined in the standard library.

On Python 2 though, shutil_which is the function defined in scripts/core.py.

This is why there is this difference. The function in scripts.core is never executed in Python 3.

yes it seems so.

follow-ups: 15 16 comment:14 by pmav99, 6 years ago

OK, it seems that this will be probably fixed on Python 3.8: https://bugs.python.org/issue18283 https://github.com/python/cpython/commit/5680f6546dcda550ad70eefa0a5ebf1375303307

In the meantime, @hellik can you please try to copy paste the 3.8 version of shutil.which from github and test if it works? https://github.com/python/cpython/blob/d8320ecb86da8df7c13d8bf8582507f736aa2924/Lib/shutil.py#L1293-L1369

Version 0, edited 6 years ago by pmav99 (next)

in reply to: 14 comment:15 by hellik, 6 years ago

Replying to pmav99:

In the meantime, @hellik can you please try to copy paste the 3.8 version of shutil.which from github and test if it works?

https://github.com/python/cpython/blob/d8320ecb86da8df7c13d8bf8582507f736aa2924/Lib/shutil.py#L1293-L1369

will try a copy/paste; into core.py, right?

what is missing in the 3.8 github version of shutil.which, is:

203	        if b'.py' not in pathext:
204	            # we assume that PATHEXT contains always '.py'
205	            pathext.insert(0, b'.py')

this should be added to while copy/paste

in reply to: 14 comment:16 by hellik, 6 years ago

Replying to pmav99:

OK, it seems that this will be probably fixed on Python 3.8:

https://bugs.python.org/issue18283
https://github.com/python/cpython/commit/5680f6546dcda550ad70eefa0a5ebf1375303307
In the meantime, @hellik can you please try to copy paste the 3.8 version of shutil.which from github and test if it works?

https://github.com/python/cpython/blob/d8320ecb86da8df7c13d8bf8582507f736aa2924/Lib/shutil.py#L1293-L1369

now what I've done:

I copied def which and def _access_check (L1285-L1369) from python github into C:\OSGeo4W64\apps\Python37\lib\shutil.py

=> the wxGUI start up window to select a location/mapset pops up.

thus, the python fix for 3.8 (shutil.which() should support bytes) seems to work.

then there is another issue (in C:\OSGEO4~1\apps\Python37\lib\xml\etree\ElementTree.py), but then this may be another ticket.

comment:17 by hellik, 5 years ago

tested now with

GRASS Version: 7.7.dev                                                          
Code revision: 2929fffa9                                                        
Build date: 2019-07-19                                                          
Build platform: x86_64-w64-mingw32                                              
GDAL: 2.4.1                                                                     
PROJ.4: 5.2.0                                                                   
GEOS: 3.7.2                                                                     
SQLite: 3.26.0                                                                  
Python: 3.7.0                                                                   
wxPython: 4.0.3                                                                 
Platform: Windows-10-10.0.18362-SP0 (OSGeo4W)

wxGUI opens now; closing ticket.

comment:18 by hellik, 5 years ago

Resolution:	→ fixed
Status:	new → closed

Note: See TracTickets for help on using tickets.

Download in other formats: