Opened 12 months ago

Last modified 10 months ago

#2428 new enhancement

xarray io for netCDF

Reported by: darkblueb Owned by: osgeolive@…
Priority: major Milestone: OSGeoLive16.0
Component: OSGeoLive Keywords: python, datacube
Cc:

Description

some sample code shows xarray.open_dataset() yet it is not installed

xarray optional dependencies are described here:

​https://docs.xarray.dev/en/stable/getting-started-guide/installing.html

Change history (6)

comment:1 by darkblueb, 12 months ago

debian / ubuntu apply these patches ? 

Fetched 2,631 kB in 16s (166 kB/s)                                                                        
dpkg-source: info: extracting python-xarray in python-xarray-0.21.1
dpkg-source: info: unpacking python-xarray_0.21.1.orig.tar.xz
dpkg-source: info: unpacking python-xarray_0.21.1-1.debian.tar.xz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: applying silence-warnings.patch
dpkg-source: info: applying Use-packaged-docs.patch
dpkg-source: info: applying Use-packaged-MathJax.patch
dpkg-source: info: applying ARM_xfail_tests.patch
dpkg-source: info: applying bottlneck-xfail.patch
dpkg-source: info: applying disable-iris.patch
dpkg-source: info: applying ipython-fix.patch
dpkg-source: info: applying no-installer.patch
dpkg-source: info: applying xfail-on-download.patch
dpkg-source: info: applying pip3.patch
dpkg-source: info: applying no-accessors.patch
dpkg-source: info: applying requirejs-path.patch
dpkg-source: info: applying no-sphinx-panels.patch
dpkg-source: info: applying no-sphinxext.patch
dpkg-source: info: applying xfail-pad-constant.patch

comment:2 by darkblueb, 12 months ago

python3 pip install of [io] option: 

$ python3 -m pip install --upgrade-strategy=only-if-needed --user  "xarray[io]"
Requirement already satisfied: xarray[io] in /usr/lib/python3/dist-packages (0.16.1)
Collecting cfgrib
  Using cached cfgrib-0.9.10.3.tar.gz (6.4 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: cftime in /usr/lib/python3/dist-packages (from xarray[io]) (1.5.2)
Requirement already satisfied: fsspec in /usr/lib/python3/dist-packages (from xarray[io]) (2022.1.0)
Collecting h5netcdf
  Using cached h5netcdf-1.1.0-py2.py3-none-any.whl (26 kB)
Requirement already satisfied: netCDF4 in /usr/lib/python3/dist-packages (from xarray[io]) (1.5.8)
Collecting pooch
  Using cached pooch-1.7.0-py3-none-any.whl (60 kB)
Collecting pydap
  Using cached pydap-3.4.0-py3-none-any.whl (2.3 MB)
Requirement already satisfied: rasterio in /usr/lib/python3/dist-packages (from xarray[io]) (1.3.4)
Requirement already satisfied: scipy in /usr/lib/python3/dist-packages (from xarray[io]) (1.8.0)
Collecting zarr
  Using cached zarr-2.14.2-py3-none-any.whl (203 kB)
Requirement already satisfied: click in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (8.0.3)
Collecting eccodes>=0.9.8
  Using cached eccodes-1.5.2.tar.gz (2.3 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: attrs>=19.2 in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (21.2.0)
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (1.21.5)
Collecting h5py
  Downloading h5py-3.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 255.5 kB/s eta 0:00:00
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from h5netcdf->xarray[io]) (21.3)
Requirement already satisfied: requests>=2.19.0 in /usr/lib/python3/dist-packages (from pooch->xarray[io]) (2.25.1)
Collecting platformdirs>=2.5.0
  Downloading platformdirs-3.5.0-py3-none-any.whl (15 kB)
Collecting docopt
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: beautifulsoup4 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (4.10.0)
Collecting Webob
  Downloading WebOb-1.8.7-py2.py3-none-any.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.0/115.0 KB 245.6 kB/s eta 0:00:00
Requirement already satisfied: Jinja2 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (3.0.3)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (1.16.0)
Requirement already satisfied: click-plugins in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (1.1.1)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (59.6.0)
Requirement already satisfied: snuggs>=1.4.1 in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (1.4.7)
Requirement already satisfied: affine in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (2.3.0)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (2020.6.20)
Requirement already satisfied: cligj>=0.5 in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (0.7.2)
Collecting asciitree
  Downloading asciitree-0.3.3.tar.gz (4.0 kB)
  Preparing metadata (setup.py) ... done
Collecting numcodecs>=0.10.0
  Downloading numcodecs-0.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.7/6.7 MB 253.8 kB/s eta 0:00:00
Collecting fasteners
  Downloading fasteners-0.18-py3-none-any.whl (18 kB)
Collecting cffi
  Downloading cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.8/441.8 KB 256.7 kB/s eta 0:00:00
Collecting findlibs
  Downloading findlibs-0.0.5.tar.gz (6.6 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: entrypoints in /usr/lib/python3/dist-packages (from numcodecs>=0.10.0->zarr->xarray[io]) (0.4)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 232.3 kB/s eta 0:00:00
Building wheels for collected packages: cfgrib, eccodes, asciitree, docopt, findlibs
  Building wheel for cfgrib (pyproject.toml) ... done
  Created wheel for cfgrib: filename=cfgrib-0.9.10.3-py3-none-any.whl size=46720 sha256=9c0f409d17cb75e892d404f7a201920631326e53e5fe5336dca291b11333b361
  Stored in directory: /home/dbb/.cache/pip/wheels/64/d8/f6/7caecd8a01d42f7db5a764d8c06dbb2dced1a2aa9d9c1920d5
  Building wheel for eccodes (setup.py) ... done
  Created wheel for eccodes: filename=eccodes-1.5.2-py3-none-any.whl size=40560 sha256=85cca204181cab6bb7eb722fd69072998183ddf0c29787c34f1275eff4f5814c
  Stored in directory: /home/dbb/.cache/pip/wheels/30/a8/9b/b80ae7683c2187d95eab008b4c57a5a57c50836aef413262a5
  Building wheel for asciitree (setup.py) ... done
  Created wheel for asciitree: filename=asciitree-0.3.3-py3-none-any.whl size=5050 sha256=098ebc03f4d682dbbf19c37bf7087ad0cf5788b5daa7d2cbc8cf8a18947f520e
  Stored in directory: /home/dbb/.cache/pip/wheels/7f/4e/be/1171b40f43b918087657ec57cf3b81fa1a2e027d8755baa184
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=0ea18e36ff95bf6459c270c6dd830bbd90fe81c2195da630611a12f53f49bac2
  Stored in directory: /home/dbb/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac
  Building wheel for findlibs (setup.py) ... done
  Created wheel for findlibs: filename=findlibs-0.0.5-py3-none-any.whl size=6954 sha256=76365fd7f480032d11d70dc1429e503555c2577aa08b3e5394cebf2ac9bd8c8e
  Stored in directory: /home/dbb/.cache/pip/wheels/2e/67/14/22fa5b9fd9c41be520b37e908597d6a262803c0fcf2ba7c2c3
Successfully built cfgrib eccodes asciitree docopt findlibs
Installing collected packages: findlibs, docopt, asciitree, Webob, pycparser, platformdirs, numcodecs, h5py, fasteners, zarr, pydap, pooch, h5netcdf, cffi, eccodes, cfgrib
  WARNING: The scripts dods and pydap are installed in '/home/dbb/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                                   
  WARNING: The script cfgrib is installed in '/home/dbb/.local/bin' which is not on PATH.                  
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                                   
Successfully installed Webob-1.8.7 asciitree-0.3.3 cffi-1.15.1 cfgrib-0.9.10.3 docopt-0.6.2 eccodes-1.5.2 fasteners-0.18 findlibs-0.0.5 h5netcdf-1.1.0 h5py-3.8.0 numcodecs-0.11.0 platformdirs-3.5.0 pooch-1.7.0 pycparser-2.21 pydap-3.4.0 zarr-2.14.2

comment:3 by darkblueb, 11 months ago

python3 import missing a dependency of cfgrib 

import logging

import rasterio
import rasterio.features
import shapely.affinity
import shapely.geometry
import shapely.ops
from datacube.index.hl import Doc2Dataset
from datacube.utils import changes
from rasterio import RasterioIOError

/usr/lib/python3/dist-packages/xarray/backends/cfgrib_.py:27: UserWarning: Failed to load cfgrib - most likely there is a problem accessing the ecCodes library. 
...
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
/tmp/ipykernel_86812/857012844.py in <module>
----> 1 import cfgrib

~/.local/lib/python3.10/site-packages/cfgrib/__init__.py in <module>
     18 # cfgrib core API depends on the ECMWF ecCodes C-library only
     19 from .abc import Field, Fieldset, Index, MappingFieldset
---> 20 from .cfmessage import COMPUTED_KEYS
     21 from .dataset import (
     22     Dataset,
 ...

~/.local/lib/python3.10/site-packages/gribapi/bindings.py in <module>
     33 library_path = findlibs.find("eccodes")
     34 if library_path is None:
---> 35     raise RuntimeError("Cannot find the ecCodes library")
     36 
     37 # default encoding for ecCodes strings

RuntimeError: Cannot find the ecCodes library

--

comment:4 by darkblueb, 11 months ago

.. after using pip3: 

$ pip3 install --user ecCodes
Req: ecCodes in /home/user/.local/lib/python3.10/site-packages (1.5.2)
Req: findlibs in /home/user/.local/lib/python3.10/site-packages (from ecCodes) (0.0.5)
Req: numpy in /usr/lib/python3/dist-packages (from ecCodes) (1.21.5)
Req: attrs in /usr/lib/python3/dist-packages (from ecCodes) (21.2.0)
Req: cffi in /home/user/.local/lib/python3.10/site-packages (from ecCodes) (1.15.1)
Req: pycparser in /home/user/.local/lib/python3.10/site-packages (from cffi->ecCodes) (2.21)
Last edited 11 months ago by darkblueb (previous) (diff)

comment:5 by darkblueb, 11 months ago

ol16 b73 

user@osgeolive:~/odc$ pip3 install --user --upgrade-strategy=only-if-needed xarray[io]                 
Requirement already satisfied: xarray[io] in /usr/lib/python3/dist-packages (0.16.1)
Collecting cfgrib
    ....
Installing collected packages: findlibs, docopt, asciitree, Webob, pycparser, platformdirs, numcodecs, h5py, fasteners, zarr, pydap, pooch, h5netcdf, cffi, eccodes, cfgrib
  WARNING: The scripts dods and pydap are installed in '/home/user/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                           
  WARNING: The script cfgrib is installed in '/home/user/.local/bin' which is not on PATH.             
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                           
Successfully installed Webob-1.8.7 asciitree-0.3.3 cffi-1.15.1 cfgrib-0.9.10.4 docopt-0.6.2 eccodes-1.5.2 fasteners-0.18 findlibs-0.0.5 h5netcdf-1.1.0 h5py-3.8.0 numcodecs-0.11.0 platformdirs-3.5.1 pooch-1.7.0 pycparser-2.21 pydap-3.4.1 zarr-2.14.2

comment:6 by darkblueb, 10 months ago

ol16 beta1 

user@osgeolive:~$ python3 -m pip install --upgrade-strategy=only-if-needed --user  "xarray[io]"
Requirement already satisfied: xarray[io] in /usr/lib/python3/dist-packages (0.16.1)
Collecting cfgrib
  Downloading cfgrib-0.9.10.4-py3-none-any.whl (47 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.1/47.1 KB 270.7 kB/s eta 0:00:00
Requirement already satisfied: cftime in /usr/lib/python3/dist-packages (from xarray[io]) (1.5.2)
Requirement already satisfied: fsspec in /usr/lib/python3/dist-packages (from xarray[io]) (2022.1.0)
Collecting h5netcdf
  Downloading h5netcdf-1.2.0-py3-none-any.whl (43 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 43.3/43.3 KB 319.2 kB/s eta 0:00:00
Requirement already satisfied: netCDF4 in /usr/lib/python3/dist-packages (from xarray[io]) (1.5.8)
Collecting pooch
  Downloading pooch-1.7.0-py3-none-any.whl (60 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.9/60.9 KB 285.2 kB/s eta 0:00:00
Collecting pydap
  Downloading pydap-3.4.1-py3-none-any.whl (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 269.0 kB/s eta 0:00:00
Requirement already satisfied: rasterio in /usr/lib/python3/dist-packages (from xarray[io]) (1.3.4)
Requirement already satisfied: scipy in /usr/lib/python3/dist-packages (from xarray[io]) (1.8.0)
Collecting zarr
  Downloading zarr-2.15.0-py3-none-any.whl (206 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 206.1/206.1 KB 265.8 kB/s eta 0:00:00
Requirement already satisfied: numpy in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (1.21.5)
Collecting eccodes>=0.9.8
  Downloading eccodes-1.5.2.tar.gz (2.3 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 269.4 kB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: click in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (8.0.3)
Requirement already satisfied: attrs>=19.2 in /usr/lib/python3/dist-packages (from cfgrib->xarray[io]) (21.2.0)
Requirement already satisfied: packaging in /usr/lib/python3/dist-packages (from h5netcdf->xarray[io]) (21.3)
Collecting h5py
  Downloading h5py-3.8.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.6/4.6 MB 268.9 kB/s eta 0:00:00
Requirement already satisfied: requests>=2.19.0 in /usr/lib/python3/dist-packages (from pooch->xarray[io]) (2.25.1)
Collecting platformdirs>=2.5.0
  Downloading platformdirs-3.5.3-py3-none-any.whl (15 kB)
Requirement already satisfied: six>=1.4.0 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (1.16.0)
Requirement already satisfied: beautifulsoup4 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (4.10.0)
Requirement already satisfied: Jinja2 in /usr/lib/python3/dist-packages (from pydap->xarray[io]) (3.0.3)
Collecting docopt
  Downloading docopt-0.6.2.tar.gz (25 kB)
  Preparing metadata (setup.py) ... done
Collecting Webob
  Downloading WebOb-1.8.7-py2.py3-none-any.whl (114 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.0/115.0 KB 244.7 kB/s eta 0:00:00
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (2020.6.20)
Requirement already satisfied: affine in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (2.3.0)
Requirement already satisfied: snuggs>=1.4.1 in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (1.4.7)
Requirement already satisfied: setuptools in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (59.6.0)
Requirement already satisfied: cligj>=0.5 in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (0.7.2)
Requirement already satisfied: click-plugins in /usr/lib/python3/dist-packages (from rasterio->xarray[io]) (1.1.1)
Collecting fasteners
  Downloading fasteners-0.18-py3-none-any.whl (18 kB)
Collecting numcodecs>=0.10.0
  Downloading numcodecs-0.11.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 6.7/6.7 MB 267.0 kB/s eta 0:00:00
Collecting asciitree
  Downloading asciitree-0.3.3.tar.gz (4.0 kB)
  Preparing metadata (setup.py) ... done
Collecting cffi
  Downloading cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 441.8/441.8 KB 272.7 kB/s eta 0:00:00
Collecting findlibs
  Downloading findlibs-0.0.5.tar.gz (6.6 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: entrypoints in /usr/lib/python3/dist-packages (from numcodecs>=0.10.0->zarr->xarray[io]) (0.4)
Collecting pycparser
  Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 KB 262.5 kB/s eta 0:00:00
Building wheels for collected packages: eccodes, asciitree, docopt, findlibs
  Building wheel for eccodes (setup.py) ... done
  Created wheel for eccodes: filename=eccodes-1.5.2-py3-none-any.whl size=40560 sha256=01468a9adef4291140484cdac7bcfc0335dc329fd9ce2fd42991faa5bc87fc05
  Stored in directory: /home/user/.cache/pip/wheels/30/a8/9b/b80ae7683c2187d95eab008b4c57a5a57c50836aef413262a5
  Building wheel for asciitree (setup.py) ... done
  Created wheel for asciitree: filename=asciitree-0.3.3-py3-none-any.whl size=5050 sha256=afd530a7cbc2223f02da2928fb9c3f57c83cd280ab87938eb7e7a5c8c0ea1a4d
  Stored in directory: /home/user/.cache/pip/wheels/7f/4e/be/1171b40f43b918087657ec57cf3b81fa1a2e027d8755baa184
  Building wheel for docopt (setup.py) ... done
  Created wheel for docopt: filename=docopt-0.6.2-py2.py3-none-any.whl size=13723 sha256=2b12e02f016c854d20ae9c67b247542066cc31169945cdaa4e9f86bc4d2c1359
  Stored in directory: /home/user/.cache/pip/wheels/fc/ab/d4/5da2067ac95b36618c629a5f93f809425700506f72c9732fac
  Building wheel for findlibs (setup.py) ... done
  Created wheel for findlibs: filename=findlibs-0.0.5-py3-none-any.whl size=6954 sha256=e6a0a12f8c18f6153a247b25b53d442c77c2b504c5704b8807ca93adb6a697c1
  Stored in directory: /home/user/.cache/pip/wheels/2e/67/14/22fa5b9fd9c41be520b37e908597d6a262803c0fcf2ba7c2c3
Successfully built eccodes asciitree docopt findlibs
Installing collected packages: findlibs, docopt, asciitree, Webob, pycparser, platformdirs, numcodecs, h5py, fasteners, zarr, pydap, pooch, h5netcdf, cffi, eccodes, cfgrib
  WARNING: The scripts dods and pydap are installed in '/home/user/.local/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                             
  WARNING: The script cfgrib is installed in '/home/user/.local/bin' which is not on PATH.              
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.                                                                                             
Successfully installed Webob-1.8.7 asciitree-0.3.3 cffi-1.15.1 cfgrib-0.9.10.4 docopt-0.6.2 eccodes-1.5.2 fasteners-0.18 findlibs-0.0.5 h5netcdf-1.2.0 h5py-3.8.0 numcodecs-0.11.0 platformdirs-3.5.3 pooch-1.7.0 pycparser-2.21 pydap-3.4.1 zarr-2.15.0
Note: See TracTickets for help on using tickets.