RFC 24: GDAL Progressive Data Support

Author: Norman Barker
Contact: nbarker@…
Status: Development

Summary

Provide an interface for data streaming support in GDAL. The RFC focuses on JPIP, but should be generic enough to apply to other streaming / progressive formats.

Definitions

JPIP: JPEG 2000 Interactive Protocol

Objective

To provide an interface to a streaming data source in a convenient manner for RasterIO operations within GDAL and to expose this interface via swig.

JPIPKAK - JPIP Streaming

JPEG 2000 Interactive Protocol (JPIP) flexibility with respect to random access, code stream reordering and incremental decoding is highly exploitable in a networked environment allowing access to remote large files using limited bandwidth connections or high contention networks.

JPIPKAK - JPIP Overview

A brief overview of the JPIP event sequence is presented in this section, more information can be found at ​JPEG 2000 Interactive Protocol (Part 9 – JPIP) and the specification can (and should) be purchased from ​ISO.

An earlier version of JPEG 2000 Part 9 is available here ​http://www.jpeg.org/public/fcd15444-9v2.pdf, noting the ISO copyright, diagrams are not replicated in this documentation.

The JPIP protocol has been abstracted in this format driver, requests are made at the 1:1 resolution level.

1. Initial JPIP request for a target image, a target id, a session over http, data to be returned as a jpp-stream are requested and a maximum length is put on the response. In this case no initial window is requested, though it can be. Server responds with a target identifier that can be used to identify the image on the server and a JPIP-cnew response header which includes the path to the JPIP server which will handle all future requests and a cid session identifier. A session is required so that that the server can model the state of the client connection, only sending the data that is required.
2. Client requests particular view windows on the target image with a maximum response length and includes the session identifier established in the previous communication. 'fsiz' is used to identify the resolution associated with the requested view-window. The values 'fx' and 'fy' specify the dimensions of the desired image resolution. 'roff' is used to identify the upper left hand corner off the spatial region associated with the requested view-windw. 'rsiz' is used to identify the horizontal and vertical extents of the spatial region associated with the requested view-window.

JPIPKAK - approach

The JPIPKAK driver uses an approach that was first demonstrated here, ​J2KViewer, by Juan Pablo Garcia Ortiz of separating the communication layer (socket / http) from the Kakadu kdu_cache object. Separating the communication layer from the data object is desirable since it allows the use of optimized http client libraries such as libcurl, Apache HttpClient (note that jportiz used a plain Java socket) and allows SSL communication between the client and server.

Kakadu's implementation of client communication with a JPIP server uses a socket, and this socket connection holds the state for this client session. A client session with Kakadu can be recreated using the JPIP cache operations between client and server, but no use of traditional HTTP cookies is supported since JPIP is neutral to the transport layer.

The JPIPKAK driver is written using a HTTP client library with the Kakadu cache object and supports optimized communication with a JPIP server (which may or may not support HTTP sessions) and the high performance of the kakadu kdu_region_decompressor.

JPIPKAK - implementation