Opened 5 years ago
Last modified 4 years ago
#3737 new defect
Testsuite gunittest/multirunner.py: TypeError: str expected, not bytes
| Reported by: | AnikaBettge | Owned by: | |
|---|---|---|---|
| Priority: | normal | Milestone: | 7.8.3 |
| Component: | Python | Version: | git-releasebranch78 |
| Keywords: | Python3, testsuite | Cc: | |
| CPU: | x86-64 | Platform: | Linux |
Description
Tzhe multirunner in trunk currently fails with Python3:
(grasspy3) ✔ ~/src/grass-7.7.svn/testsuite/examples
11:41 $ bash test_framework_GRASS_GIS_with_NC.sh test_framework_GRASS_GIS_with_NC_AB.conf
--2019-01-23 13:42:48-- https://grass.osgeo.org/sampledata/north_carolina/nc_spm_08_grass7.tar.gz
...
Starting GRASS GIS...
Cleaning up temporary files...
Executing <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh> ...
WARNUNG: <basin> already exists. File not copied.
WARNUNG: <boundary> already exists. File not copied.
...
Execution of <sh /home/abettge/grassdata/tests-grassdata/tmp_rename.sh> finished.
Cleaning up default sqlite database ...
Cleaning up temporary files...
Traceback (most recent call last):
File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py", line 125, in <module>
sys.exit(main())
File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multirunner.py", line 76, in main
os.environ['GISBASE'] = gisbase
File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py", line 730, in __setitem__
value = self.encodevalue(value)
File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.5/os.py", line 798, in encode
raise TypeError("str expected, not %s" % type(value).__name__)
TypeError: str expected, not bytes
Version:
grass77 --config svn_revision 74007
Change History (13)
comment:2 by , 5 years ago
That being said, even after applying the patch the testrunner is still not running as expected. More specifically all the tests are failing with this message on stderr.txt:
.../stderr.txt Unknown option: -3 usage: python3 [option] ... [-c cmd | -m mod | file | -] [arg] ... Try `python -h' for more information.
Python 2 does indeed have a -3 switch but Python 3 does not. I am looking but I can't find where this switch is added. If anyone has any idea, please post.
comment:3 by , 5 years ago
This is also needed
Index: lib/python/gunittest/invoker.py
===================================================================
--- lib/python/gunittest/invoker.py (revision 74025)
+++ lib/python/gunittest/invoker.py (working copy)
@@ -161,7 +161,7 @@
# ignoring shebang line to use current Python
# and also pass parameters to it
# add also '-Qwarn'?
- p = subprocess.Popen([sys.executable, '-tt', '-3',
+ p = subprocess.Popen([sys.executable, '-tt',
module.abs_file_path],
cwd=cwd, env=env,
stdout=stdout, stderr=stderr)
comment:4 by , 5 years ago
After applying the previous patches, the vast majority of the tests were again failing due to trying to mix strings and bytes in shutil_which(). The following patch fixes that and the tests can finally run. There are still failing ones, but that's a different issue. I opted to add an additional decode() call since it seems less likely to break anything.
Index: script/core.py
===================================================================
--- script/core.py (revision 74025)
+++ script/core.py (working copy)
@@ -234,7 +234,9 @@
if not normdir in seen:
seen.add(normdir)
for thefile in files:
- name = os.path.join(encode(dir), thefile)
+ name = os.path.join(encode(dir), encode(thefile))
if _access_check(name, mode):
return name
return None
comment:5 by , 5 years ago
Thanks for looking into it, I fixed the things in gunittest in r74032, but I have still some local changes related to shutil_which in core.py I need to test more first.
follow-up: 7 comment:6 by , 5 years ago
Hello Anna, is it necessary to redefine encode() and decode()? They are already defined in a bunch of places already...
$ ag -l 'def encode' lib/python lib/python/script/utils.py lib/python/ctypes/preamble.py lib/python/ctypes/ctypesgencore/printer/preamble.py
The one in script/utils.py seems to be OK.
comment:7 by , 5 years ago
Replying to pmav99:
Hello Anna, is it necessary to redefine
encode()anddecode()? They are already defined in a bunch of places already...$ ag -l 'def encode' lib/python lib/python/script/utils.py lib/python/ctypes/preamble.py lib/python/ctypes/ctypesgencore/printer/preamble.pyThe one in
script/utils.pyseems to be OK.
True, but the library is not loaded at that point when I need to decode.
comment:8 by , 5 years ago
Cleaning up default sqlite database ...
Cleaning up temporary files...
raster3d_lib_test from ./lib/raster3d failed (3 tests failed)
test_assertions from ./lib/python/gunittest failed (3 tests failed)
test_checkers from ./lib/python/gunittest failed (1 test failed)
test_gmodules from ./lib/python/gunittest failed (4 tests failed)
test_assertions_vect from ./lib/python/gunittest failed (7 tests failed)
test_assertions_rast3d from ./lib/python/gunittest failed (2 tests failed)
test_module_assertions from ./lib/python/gunittest failed (2 tests failed)
test_doctests from ./lib/python/gunittest failed (2 tests failed)
unittests_temporal_raster_conditionals from ./lib/python/temporal failed (50 tests failed)
unittests_temporal_raster_conditionals_complement_else from ./lib/python/temporal failed (4 tests failed)
unittests_temporal_vector_algebra from ./lib/python/temporal failed (14 tests failed)
unittests_temporal_conditionals from ./lib/python/temporal failed (34 tests failed)
unittests_temporal_raster_algebra_grs from ./lib/python/temporal failed (13 tests failed)
unittests_temporal_algebra_grs from ./lib/python/temporal failed (6 tests failed)
unittests_temporal_raster_algebra from ./lib/python/temporal failed (80 tests failed)
unittests_temporal_raster3d_algebra from ./lib/python/temporal failed (4 tests failed)
unittests_temporal_raster_algebra_equal_ts from ./lib/python/temporal failed (10 tests failed)
unittests_temporal_algebra from ./lib/python/temporal failed (39 tests failed)
unittests_temporal_raster_algebra_spatial_topology from ./lib/python/temporal failed (12 tests failed)
test_doctests from ./lib/python/temporal failed (15 tests failed)
unittests_temporal_algebra_mixed_stds from ./lib/python/temporal failed (9 tests failed)
test_doctests from ./lib/python/script failed (1 test failed)
Traceback (most recent call last):
File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
sys.exit(main())
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 174, in main
results_dir=results_dir)
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 237, in run_in_location
gisdbase=gisdbase, location=location)
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 193, in _run_test_module
self._file_anonymizer.anonymize([stdout_path, stderr_path])
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/reporters.py", line 103, in anonymize
replace_in_file(filename, path + path_end, '')
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/reporters.py", line 61, in replace_in_file
for line in old_file:
File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/codecs.py", line 321, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 439: invalid continuation byte
GRASS Location <other_location> does not exist in GRASS Database </home/abettge/grassdata/tests-grassdata>
Traceback (most recent call last):
File "/home/abettge/src/grass-7.7.svn/lib/python/gunittest/multireport.py", line 28, in <module>
from grass.gunittest.checkers import text_to_keyvalue
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/checkers.py", line 18, in <module>
from grass.script.core import KeyValue
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/__init__.py", line 5, in <module>
from .core import *
File "/home/abettge/src/grass-7.7.svn/dist.x86_64-pc-linux-gnu/etc/python/grass/script/core.py", line 38, in <module>
gettext.install('grasslibs', os.path.join(os.getenv("GISBASE"), 'locale'))
File "/home/abettge/src/grass-7.7.svn/grasspy3/lib/python3.6/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
grass77 --config svn_revision 74046M
comment:9 by , 5 years ago
| Version: | svn-trunk → git-releasebranch78 |
|---|
State as of today:
g.version -rge
version=7.8.dev
date=2019
revision=726b55ea2
build_date=2019-09-06
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=00000
libgis_date="?"
proj4=5.2.0
gdal=2.3.2
geos=3.7.1
sqlite=3.26.0
bash test_framework_GRASS_GIS_with_NC.sh test_framework_GRASS_GIS_with_NC.conf
--2019-09-06 22:11:34-- http://fatra.cnr.ncsu.edu/data/nc_spm_full_v2alpha.tar.gz
Resolving fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)... 152.1.72.43
Connecting to fatra.cnr.ncsu.edu (fatra.cnr.ncsu.edu)|152.1.72.43|:80... connected.
HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable
The file is already fully retrieved; nothing to do.
+ echo 'Testing of GRASS GIS started: 2019-09-06-20-11'
+ '[' no = yes ']'
+ cd testreports/reports_for_date-2019-09-06-20-11
+ python /home/mneteler/software/grass78_git/lib/python/gunittest/multirunner.py --grassbin /home/mneteler/software/grass78_git/bin.x86_64-pc-linux-gnu/grass78 --grasssrc /home/mneteler/software/grass78_git --grassdata /home/mneteler/grassdata --location nc_spm_full_v2alpha --location-type nc
test_v_out_lidar from ./vector/v.out.lidar failed (2 tests failed)
decimation_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_filter from ./vector/v.in.lidar failed (1 test failed)
test_v_in_lidar_basic from ./vector/v.in.lidar failed (1 test failed)
mask_test from ./vector/v.in.lidar failed (1 test failed)
test_v_in_pdal_basic from ./vector/v.in.pdal failed (1 test failed)
test_v_in_pdal_filter from ./vector/v.in.pdal failed (1 test failed)
Traceback (most recent call last):
File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 178, in <module>
sys.exit(main())
File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/main.py", line 174, in main
results_dir=results_dir)
File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 271, in run_in_location
gisdbase=gisdbase, location=location)
File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/gunittest/invoker.py", line 218, in _run_test_module
stdout_file.write(stdout)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 596-597: ordinal not in range(128)
+ export PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python:
+ PYTHONPATH=/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python:
+ python /home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py --output summary_report 'reports_for_date-*/*'
Traceback (most recent call last):
File "/home/mneteler/software/grass78_git/lib/python/gunittest/multireport.py", line 28, in <module>
from grass.gunittest.checkers import text_to_keyvalue
File "/home/mneteler/software/grass78_git/dist.x86_64-pc-linux-gnu/etc/python/grass/__init__.py", line 21, in <module>
_LOCALE_DIR = os.path.join(os.getenv("GISBASE"), 'locale')
File "/usr/lib64/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
comment:13 by , 4 years ago
| Milestone: | → 7.8.3 |
|---|
Note:
See TracTickets
for help on using tickets.
In order to make
multirunner.pyrun on Python 3 you need to apply something like this.Index: lib/python/gunittest/multirunner.py =================================================================== --- lib/python/gunittest/multirunner.py (revision 74025) +++ lib/python/gunittest/multirunner.py (working copy) @@ -64,13 +64,13 @@ # we assume that GRASS GIS' start script is available and in the PATH # the shell=True is here because of MS Windows? (code taken from wiki) startcmd = grass7bin + ' --config path' p = subprocess.Popen(startcmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, err = p.communicate() if p.returncode != 0: print("ERROR: Cannot find GRASS GIS 7 start script (%s):\n%s" % (startcmd, err), file=sys.stderr) return 1 - gisbase = out.strip('\n') + gisbase = out.strip().decode("utf-8") # set GISBASE environment variable os.environ['GISBASE'] = gisbase @@ -95,7 +95,7 @@ gsetup.init(gisbase, gisdb, location, mapset) reports = [] - for location, location_type in itertools.izip(locations, locations_types): + for location, location_type in zip(locations, locations_types): # here it is quite a good place to parallelize # including also type to make it unique and preserve it for sure report = 'report_for_' + location + '_' + location_typeSpecifying
\nis probably redundant.str.strip()already strips that. More specifically it strips all the characters contained instring.whitespace.Other than that, in python3 subprocess output is bytes. The
.decode("utf-8")call is converting the bytes to python3 strings which are the equivalent of unicode strings in python2.Note: the patch should work with python2 too, but I have not tested it.