SciTools IRIS 2.10 missing from Python3

[darkblueb]

the OSGeoLive version 12 based on Ubuntu Bionic (1804) currently installs Scitools IRIS via dot-deb packaging found here [0]

IRIS setup.py [1] "may be broken for Python3" in this environment per the packager Angelos Tzotsos. This ticket is a place to find a workaround or longer-term fix.

[0] ​https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/iris

[1] ​https://github.com/SciTools/iris/blob/v2.1.x/setup.py

[darkblueb]

Phil Elson writes:

Thanks for letting us know. I’m sorry to say I can’t find the detail regarding the python 3 issue. 
If I were to guess though, I suspect the problem is that pyke forked itself and maintains two branches 
for its python 2 -> python 3 journey (don’t know why one would choose to do that). 

As a result, pyke for python 3 needs to come from a different source. 
To see how to work around that, the conda-forge build for this is done in 

https://github.com/conda-forge/pyke-feedstock/blob/master/recipe/meta.yaml#L12.

Hopefully my guess is on target, but if not, 
could you point me at the detail and I’ll try to help?
 
--

Philip Elson | Scientific Python Technical Lead
Analysis, Visualisation and Data Team (AVD)
Met Office  FitzRoy Road  Exeter  Devon  EX1 3PB  United Kingdom
Email: philip.elson@metoffice.gov.uk  Website: www.metoffice.gov.uk
Our magazine Barometer is now available online at http://www.metoffice.gov.uk/barometer/

[darkblueb]

12dev beta2

defer until after RC 12.0

[kalxas]

Ticket retargeted after milestone closed

[darkblueb]

open in o13 alpha2

[kalxas]

Thank you for the information that there is a separate source for python3.

I am now trying to make a python3-pyke package.

[kalxas]

Iris 2.2.1 has landed in our ppa.

python3-pyke has also been created, but python3-iris fails to build.

[kalxas]

During the OSGeo Virtual Code Sprint we decided: we will try to package py3 version (pike etc), else we will include as py2 in VM. e-mail to project to ask about 20.04

[kalxas]

pyke3 now available for focal:

​https://launchpad.net/~gcpp-kalxas/+archive/ubuntu/osgeolive/+sourcepub/11822729/+listing-archive-extra

[kalxas]

Current Iris build error:

creating /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/000003000000.03.236.000128.1990.12.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/000003000000.03.236.004224.1990.12.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/000003000000.03.236.008320.1990.12.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/000003000000.16.202.000128.1860.09.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/001000000000.00.000.000000.1860.01.01.00.00.f.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/002000000000.44.101.131200.1920.09.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/008000000000.44.101.000128.1890.09.01.00.00.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/12187.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/HadCM2_ts_SAT_ann_18602100.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_level_lat_orig.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_lon_lat_press_orig.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_lon_lat_several.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_n10r13xy.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_time_press.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/aaxzc_tseries.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/abcza_pa19591997_daily_29.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/abcza_pa19591997_daily_29.b_1.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/abcza_pa19591997_daily_29.b_2.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/abxpa_press_lat.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/integer.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/model.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/ocean_xsect.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/st0fc699.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/st0fc942.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf/st30211.b_0.cdl -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_netcdf
creating /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_pp
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_pp/000003000000.03.236.000128.1990.12.01.00.00.b.pp.txt -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_pp
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_pp/000003000000.03.236.004224.1990.12.01.00.00.b.pp.txt -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_pp
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_pp/000003000000.03.236.008320.1990.12.01.00.00.b.pp.txt -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_pp
copying lib/iris/tests/results/usecases/pp_to_cf_conversion/to_pp/000003000000.16.202.000128.1860.09.01.00.00.b.pp.txt -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/usecases/pp_to_cf_conversion/to_pp
creating /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/util
copying lib/iris/tests/results/util/as_compatible_shape_collapsed.cml -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/results/util
copying lib/iris/fileformats/_pyke_rules/fc_rules_cf.krb -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/fileformats/_pyke_rules
creating /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/unit/experimental/representation
copying lib/iris/tests/unit/experimental/representation/test_CubeRepresentation.py -> /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/tests/unit/experimental/representation
/usr/bin/python3 tools/generate_std_names.py etc/cf-standard-name-table.xml /<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/std_names.py
/usr/bin/python3 -c import os; from pyke.target_pkg import target_pkg; target_pkg.load = lambda *args, **kwargs: None; os.chdir(os.path.join('/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build', 'iris', 'fileformats', '_pyke_rules')); from pyke import knowledge_engine; knowledge_engine.engine('')
/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/config.py:84: UserWarning: Ignoring config item 'Resources':'test_data_dir' (section:option) as '/home/h05/itwl/projects/git/iris-test-data/test_data' is not a valid directory path.
  warnings.warn(msg.format(section, option, c_path))
/<<PKGBUILDDIR>>/.pybuild/cpython3_3.8_scitools-iris/build/iris/fileformats/netcdf.py:1052: SyntaxWarning: "is" with a literal. Did you mean "=="?
  if data_dtype.itemsize is 1:
Unable to create '/usr/lib/python3/dist-packages/pyke/krb_compiler/krbparser_tables.py'
[Errno 13] Permission denied: '/usr/lib/python3/dist-packages/pyke/krb_compiler/krbparser_tables.py'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/pyke/knowledge_engine.py", line 108, in __init__
    target_package.compile(self)
  File "/usr/lib/python3/dist-packages/pyke/target_pkg.py", line 285, in compile
    self.do_by_ext('compile',
  File "/usr/lib/python3/dist-packages/pyke/target_pkg.py", line 265, in do_by_ext
    return getattr(self, "%s_%s" % (prefix, ext))(filename, *args)
  File "/usr/lib/python3/dist-packages/pyke/target_pkg.py", line 298, in compile_krb
    return krb_compiler.compile_krb(rb_name, self.package_name,
  File "/usr/lib/python3/dist-packages/pyke/krb_compiler/__init__.py", line 114, in compile_krb
    ast = krbparser.parse(krbparser, filename)
  File "/usr/lib/python3/dist-packages/pyke/krb_compiler/krbparser.py", line 595, in parse
    scanner.init(scanner, debug, check_tables)
  File "/usr/lib/python3/dist-packages/pyke/krb_compiler/scanner.py", line 688, in init
    lexer = lex.lex(module=this_module, optimize=1,
  File "/usr/lib/python3/dist-packages/pyke/krb_compiler/ply/lex.py", line 1004, in lex
    lexobj.writetab(lextab,outputdir)
  File "/usr/lib/python3/dist-packages/pyke/krb_compiler/ply/lex.py", line 176, in writetab
    tf = open(filename,"w")
PermissionError: [Errno 13] Permission denied: '/usr/lib/python3/dist-packages/pyke/krb_compiler/scanner_tables.py'
error: command '/usr/bin/python3' failed with exit status 1
E: pybuild pybuild:341: build: plugin distutils failed with: exit code=1: /usr/bin/python3 setup.py build 
dh_auto_build: error: pybuild --build --test-nose -i python{version} -p 3.8 returned exit code 13
make: *** [debian/rules:8: build] Error 25
dpkg-buildpackage: error: debian/rules build subprocess returned exit status 2
--------------------------------------------------------------------------------
Build finished at 2020-12-17T16:02:00Z

[kalxas]

We need to reach out to IRIS team, perhaps they will have some feedback on the above error.

[kalxas]

feedback from sebastic:

The build script is writing outside the build/install directory, it mustn't do that. The build is broken by design, because if an unpriviledged user builds the project it will fail as well. The iris people will need to help you to make it behave more appropriately.

[kalxas]

Based on the above feedback I propose we move Iris to VM only, where we can build it in a virtualenv

[kalxas]

Added in VM:

​https://github.com/OSGeo/OSGeoLive/commit/a0ac9cec167b80bfc440b58359b7b23e021aa68e#diff-ef511aa71b0cc6067701b89dfcf72b8ea8388cc17db285b7428808677a2400ecR96-R97