RFC 24: GDAL Progressive Data Support

Author: Norman Barker
Contact: nbarker@…
Status: Development

Summary

To provide an interface for data streaming support to GDAL by overloading the RasterIO function to include a callback function when there are buffer updates. The RFC focuses on JPIP but should be generic to apply to other streaming / progressive formats.

Definitions

JPIP: JPEG 2000 Interactive Protocol

Objective

To provide a callback function to allow users of a progressive format driver to receive notifications of updates to the underlying dataset for a particular requested region of the data. The notification mechanism should be accessible to all the swig wrappers.

Implementation

The implementation is a definition of an interface, in particular an overloaded function definition for RasterIO. Concrete implementations of this interface will follow. Currently the most convenient JPIP streaming developer library is ​Kakadu however since GDAL is also developer library, only stubs can be distributed to conform to Kakadu licensing (JP2KAK). Commercial vendors are also interested in using GDAL for streaming support and a standard interface will allow these commercial plugins to be incorporated. e.g. ​ECW, ​MrSID.

Discussion

​Tamas Szekeres

Question: It seems we are tending to spawn new threads in every RasterIO operations at driver level, which is quite inconvenient at the moment, therefore it should be considered with care. Will you allow RasterIO to be re-entered from the GDALProgFunc event handler or by another thread? Will you provide a copy of pBuff in GDALProgFunc or the same pointer will be passed back to the caller? How this buffer will be protected from the simultaneous access of the multiple threads?

Response:

There will one thread at the RasterIO driver level communicating to the server and this will be separate to application thread, the driver will accept new window requests from the application and change the requests being made to the server within the thread. There is a degree of synchronization required here, but this is hidden within the format driver.

The callback function and the raster display will be in the main application thread, it will not be possible to update the format driver window from the callback function, but will be possible using RasterIO read as normal.

pBuff will be small, it only contains the data for the currently request window at a particular resolution level and these are normally proportional, so small window implies higher resolutions, large window implies low resolution. As such an initial version will provide a copy of the buffer rather than a pointer this should protect the buffer from corruption by multiple threads.

Proposed Progressive Streaming Sequence of Events

Sequence shows GDALDataset, same sequence for GDALRasterBand