Changes between Version 1 and Version 2 of Conda_GDAL_lite


Ignore:
Timestamp:
08/03/24 21:58:30 (2 months ago)
Author:
darkblueb
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Conda_GDAL_lite

    v1 v2  
    1 Introducing Lightweight Versions of GDAL and PDAL
     1
     2== Introducing Lightweight Versions of GDAL and PDAL
    23Quansight
    34· Jul 25, 2024
    45
    5 See how Hobu teamed with Quansight to fund the transition to a deferred plugin system in both GDAL and PDAL. The new architecture was implemented in GDAL 3.9.1 and PDAL 2.7.2.
     6See how Hobu teamed with Quansight to fund the transition to a deferred plugin system in both GDAL and PDAL. The new architecture was implemented in GDAL 3.9.1 and PDAL 2.7.2. (this article was originally published on the Quansight Blog by Isuru Fernando).
    67
    7 This article was originally published on the Quansight Blog by Isuru Fernando.
     8The evolution of geospatial data processing has taken a significant step forward with the introduction of lightweight versions of the Geospatial Data Abstraction Library (**GDAL**) and the Point Data Abstraction Library (**PDAL**). This new architecture addresses the long-standing issue of dependency bloat, significantly improving solve times, download speeds, and overall package manageability for users. This post delves into the history, technical implementation, and benefits of this transition.
    89
    9 The evolution of geospatial data processing has taken a significant step forward with the introduction of lightweight versions of the Geospatial Data Abstraction Library (GDAL) and the Point Data Abstraction Library (PDAL). This new architecture addresses the long-standing issue of dependency bloat, significantly improving solve times, download speeds, and overall package manageability for users. This post delves into the history, technical implementation, and benefits of this transition.
     10**GDAL** (Geospatial Data Abstraction Library) is a translator library for raster and vector geospatial data formats. Being a translator library supporting multiple different geospatial data formats, it has a lot of libraries as dependencies. For example The `hdf5` for HDF5 package format support. PDAL (Point Data Abstraction Library) is a library built on top of GDAL and has similar support for package formats.
    1011
    11 GDAL (Geospatial Data Abstraction Library) is a translator library for raster and vector geospatial data formats. Being a translator library supporting multiple different geospatial data formats, it has a lot of libraries as dependencies. For example The `hdf5` for HDF5 package format support. PDAL (Point Data Abstraction Library) is a library built on top of GDAL and has similar support for package formats.
    1212
    13 A Little Bit of History
     13== A Little Bit of History
    1414conda-forge was started by a few people, including a couple of oceanographers, who wanted a way to distribute gdal easily. Hence `gdal-feedstock` is one of the first feedstocks to be made on conda-forge and was the 49th PR on staged-recipes. The initial commit to the `gdal-feedstock` which builds the conda package, only used a few packages, including `hdf4`, `hdf5`, , `postgresql`, `libnetcdf`, `kealib`.
    1515
     
    1818This is where the partnership with Hobu and Quansight comes in to fund the transition to a deferred plugin system in both GDAL and PDAL. The new architecture was implemented in GDAL 3.9.1 and PDAL 2.7.2.
    1919
    20 Deferred C++ plugin loading
     20
     21== Deferred C++ plugin loading
    2122GDAL RFC 96 enables the support of deferred plugins. Plugins in GDAL support the various raster and vector geospatial data formats. These plugins are usually built into the core library, `libgdal.(dylib/so/dll)` , but RFC 96 introduced deferred plugins that build these plugins separately such that only the necessary plugin dependencies are needed.
    2223
     
    2829This concept was first used for `libarrow/libparquet` dependency since it is a large dependency and especially because gdal supports four different major versions on conda-forge. By separating this dependency, only the plugin needs to be built for the four different arrow/parquet versions as opposed to the core libgdal library being built for the four different versions. The conda package for the plugin was called `libgdal-arrow-parquet` and depended on the core library conda package `libgdal` which included the rest of the plugins.
    2930
    30 libgdal-core and libgdal
     31
     32== libgdal-core and libgdal
    3133In order to generalize the above strategy to more plugins, we are now introducing a `libgdal-core` conda package and more plugins as conda packages with all plugins (except arrow/parquet) being installable with `libgdal` . We also made the python bindings depend on `libgdal-core` instead of `libgdal` so that users can select the plugins that they need.
    3234
     
    7072Similar to `libgdal` and `libgdal-core` , we have introduced `libgdal` and `libgdal-core` conda packages. Previously the `pdal` conda package provided only the C++ library, but now it also provides the python package to match the `gdal` conda package.
    7173
    72 pdal conda packages
     74
     75== pdal conda packages
    7376
    7477– `libpdal-core` — core C++ library
     
    7780– `pdal` — python library and all plugins
    7881
    79 pdal plugin conda packages
     82
     83== pdal plugin conda packages
    8084
    8185– `libpdal-trajectory` : `filters.trajectory` driver as a plugin