RFC 24: GDAL Progressive Data Support

Author: Norman Barker
Contact: nbarker@…
Status: Development

Summary

Provide an interface for data streaming support in GDAL. The RFC focuses on JPIP, but should be generic enough to apply to other streaming / progressive formats.

Definitions

JPIP: JPEG 2000 Interactive Protocol

Objective

To provide an interface to a streaming data source in a convenient manner for RasterIO operations within GDAL and to expose this interface via swig.

Note this RFC originally explored using an Observer pattern as per implementations in Java ​imageio update. The RFC then also discussed the ​Asynchronous Message Queue pattern, but after detailed discussions with ​Tamas Szekeres, Even Rouault, Adam Nowacki and Frank Warmerdam an even simpler interface is proposed which wraps the more unpleasant developer tasks of operating with the streaming protocol but allows the user greater flexibility in interacting with the stream in their preferred language. This is implemented by using a context object to act as an intermediary between the user layer and the format driver. The approach recommended in the RFC is a half-way abstraction of the protocol as it requires some knowledge that this is a streaming format and how to optimise the driver from the user interface. This is an acceptable, and actually desirable trade-off, since it allows the swig wrappers to implement high speed data access / display in a pattern suitable for the language. The protocol may support a finer level of messaging than is available through this layer.

Note that we may wish to revisit the name the classes within this RFC since with this proposal the the RasterIO, and block methods are blocking, hence there is not any asynchronous communication in the format driver and this is delegated to the user code layer.

Implementation

The implementation is a definition of an interface for streaming support data support within GDAL. Concrete implementations of this interface will follow. Currently the most convenient JPIP streaming developer library is ​Kakadu however since GDAL is also a developer library, only stubs can be distributed to conform to Kakadu licensing (JP2KAK). Vendors are also interested in using GDAL for streaming support and a standard interface will allow these plugins to be incorporated. e.g. ​ECW, ​MrSID.

Pseudo-Code

interaction with the driver, the Definition below should be used for function names, not the pseudo-code

   GDALDataset *ds = GDALOpen(name, Access.GA_ReadOnly);
   
   // create an IO context for interaction with the remote server
   // the driver will initiate communication to the server and the context status to IO_PENDING
   // similar interaction with RasterBand

   GDALRasterIOContext *ctx = ds->CreateRasterIOContext(GF_Read, nDSXOff, nDSYOff, nDSXSize, nDSYSize, nBXSize, nBYSize, GDT_Byte, nBandCount, panBandMap, nPixelSpace, nLineSpace, nBandSpace);

   // optional : invoke a thread to execute communication to the server, and update the window when the thread terminates

   while (ctx-> status != IO_COMPLETE)
   {
      // Update the buffer with the next stage of data.  The driver is allowed to modify 
      // the buffer within ReadNextBlock only
      // ReadNextBlock will wait for the next data response from the server
      ctx->ReadNextBlock(buffer, options);

      // check has the view updated, is the request cancelled
      if (bCancel || viewChanged)
      {
         ctx -> CancelIO();
         break;
      }
   }

   if (!viewChanged)
     updateDisplay()
      

Definition

Header and dummy class file below

GDALAsyncDataset.h

/******************************************************************************
* 
* Project: Progressive Renderer Client Driver
* Purpose: Implementation of Dataset and RasterBand classes for JPIP and other
*          progressive rendering protocols / formats
*******************************************************************************
*/

#include "gdal_pam.h"


class GDALAsyncDataset;
class GDALAsyncRasterBand;
class GDALRasterIOContext;

/**
* status of the asynchronous stream
*/
typedef enum 
{	
	IO_PENDING = 0,
	IO_UPDATE = 1,
	IO_ERROR = 2,
	IO_COMPLETE = 3
} AsyncStatus;

/**********************************************************************/
/*                           GDALAsyncDataset                         */
/**********************************************************************/
class CPL_DLL GDALAsyncDataset : public GDALPamDataset
{
	friend class GDALAsyncRasterBand;
	friend class GDALRasterIOContext;
public:	
	GDALAsyncDataset(void);
	virtual ~GDALAsyncDataset(){};
	virtual GDALRasterIOContext *CreateRasterIOContext(GDALRWFlag eRWFlag, 
						int nDSXOff, int nDSYOff, 
						int nDSXSize, int nDSYSize,
						int nBXSize, int nBYSize,
						GDALDataType eBDataType, 
						int nBandCount, int* panBandMap,
						int nPixelSpace, int nLineSpace, int nBandSpace);


	/*
	*  GDALAsyncDatasetOpen called with Access.GA_ReadOnly
	*  Implementations (with the exception of transactional jpip)
	*  are read only
	*/
	static GDALDataset *GDALAsyncDatasetOpen(GDALOpenInfo *poOpenInfo);

};


/*********************************************************************/
/*                        GDALAsyncRasterBand                        */
/*********************************************************************/
class CPL_DLL GDALAsyncRasterBand : public GDALPamRasterBand {
	friend class GDALAsyncDataset;
public:
	GDALAsyncRasterBand(GDALAsyncDataset *parent_dataset, int band, double scale);
	virtual GDALRasterIOContext *CreateRasterIOContext(GDALRWFlag eRWFlag, 
						int nDSXOff, int nDSYOff, 
						int nDSXSize, int nDSYSize,
						int nBXSize, int nBYSize,
						GDALDataType eBDataType, 
						int nPixelSpace, int nLineSpace);
	virtual ~GDALAsyncRasterBand(){};
};

class CPL_DLL GDALRasterIOContext
{
private:
	AsyncStatus status;

public:
	/**
	* Returns the current status of the asynchronous process
	* This value cannot be modified by the calling function, and 
	* the value is true at the time of the function being called.
	*/
	AsyncStatus GetStatus(){return status;}

	virtual CPLErr SetView(int xOff, int yOff, int xSize, int ySize);
	virtual CPLErr ReadNextBlock(void *pData, char **papszOptions);
	virtual CPLErr RasterIO(void *pData, int nTimeoutMilliseconds = -1, char **papszOptions);
	virtual void CancelIO();
};

GDALAsyncDataset.cpp

#include "GDALAsyncDataset.h"

/************************************************************************/
/* ==================================================================== */
/*                             GDALAsyncDataset                         */
/* ==================================================================== */
/************************************************************************/

/**
 * \class GDALAsyncDataset "GDALAsyncDataset.h"
 *
 */

/************************************************************************/
/*	                      CreateRasterIOContext()                       */
/************************************************************************/

/**
*  Create an IOContext for asynchronous operation.  The driver will initiate the transfer to the server
* and set the current status for the user to read, @see AsyncStatus
*
* @param eRWFlag Read/Write flag, most format driver implementations will be Access.GA_ReadOnly. 
*				JPIP transactional services will support GA_Update
* @param nDSXOff request x offset into the server image at resolution 1:1, 
		top left corner is the image origin
* @param nDSYOff request y offset into the server image at resolution 1:1, 
		top left corner is the image origin
* @param nDSXSize request width of the server image at resolution 1:1
* @param nDSYSize request height of the server image at resolution 1:1
* @param nBXSize Requested frame width, JPIP will select the resolution level that best fits nBXSize, nBYSize
* @param nBYSize Requested frame height, JPIP will select the resolution level that best fits nBXSize, nBYSize
* @param eBDataType 
* @param nBandCount
* @param panBandMap
* @param nPixelSpace
* @param nLineSpace
* @param nBandSpace
*/
GDALRasterIOContext *GDALAsyncDataset::CreateRasterIOContext(GDALRWFlag eRWFlag, 
							int nDSXOff, int nDSYOff, 
							int nDSXSize, int nDSYSize,
							int nBXSize, int nBYSize,
							GDALDataType eBDataType, 
							int nBandCount, int* panBandMap,
							int nPixelSpace, int nLineSpace, int nBandSpace)
{
	return 0;
}

/************************************************************************/
/* ==================================================================== */
/*                             GDALAsyncRasterBand                         */
/* ==================================================================== */
/************************************************************************/

/**
 * \class GDALAsyncRasterBand "GDALAsyncDataset.h"
 *
 */

/************************************************************************/
/*	                      CreateRasterIOContext()                       */
/************************************************************************/

/**
*  Create an IOContext for asynchronous operation.  The driver will initiate the transfer to the server
* and set the current status for the user to read, @see AsyncStatus
*
* @param eRWFlag Read/Write flag, most format driver implementations will be Access.GA_ReadOnly. 
*				JPIP transactional services will support GA_Update
* @param nDSXOff request x offset into the server image at resolution 1:1, 
		top left corner is the image origin
* @param nDSYOff request y offset into the server image at resolution 1:1, 
		top left corner is the image origin
* @param nDSXSize request width of the server image at resolution 1:1
* @param nDSYSize request height of the server image at resolution 1:1
* @param nBXSize Requested frame width, JPIP will select the resolution level that best fits nBXSize, nBYSize
* @param nBYSize Requested frame height, JPIP will select the resolution level that best fits nBXSize, nBYSize
* @param eBDataType 
* @param nPixelSpace
* @param nLineSpace
*/
GDALRasterIOContext *GDALAsyncRasterBand::CreateRasterIOContext(GDALRWFlag eRWFlag, 
								int nDSXOff, int nDSYOff, 
								int nDSXSize, int nDSYSize,
								int nBXSize, int nBYSize,
								GDALDataType eBDataType, 
								int nPixelSpace, int nLineSpace)
{
	return 0;
}

/************************************************************************/
/* ==================================================================== */
/*                             GDALRasterIOContext                      */
/* ==================================================================== */
/************************************************************************/

/**
 * \class GDALRasterIOContext "GDALAsyncDataset.h"
 *
 */

/**
*  Set the currently request window into the dataset on the server at 1:1 resolution
*
* @param xOff request x offset into the server image at resolution 1:1, 
		top left corner is the image origin
* @param yOff request y offset into the server image at resolution 1:1,
		top left corner is the image origin
* @param xSize request width of the server image at resolution 1:1
* @param ySize request height of the server image at resolution 1:1
*/
CPLErr GDALRasterIOContext::SetView(int xOff, int yOff, int xSize, int ySize)
{
	// return a warning, this method should be overridden
	return CE_Warning;
}

/*************************************************************************/
/* ===================================================================== */
/*                       CancelIO()                                      */
/* ===================================================================== */
/*************************************************************************/

/**
*  Cancel the communication channel to the remote server
* 
*/
void GDALRasterIOContext::CancelIO()
{
}

/*************************************************************************/
/* ===================================================================== */
/*                       ReadNextBlock()                                 */
/* ===================================================================== */
/*************************************************************************/

/**
* Read the next data block from the driver cache to the remote server
* data type is set in @see CreateRasterIOContext()
* @param pData, data buffer to fill, if the data in the driver cache does not contain a value the buffer will be zero filled
* @param papszOptions options for the driver implementation e.g. Quality="Xxx" to specify maximum jpip quality layers
*/
CPLErr GDALRasterIOContext::ReadNextBlock(void *pData, char **papszOptions)
{
	// return a warning, this method should be overridden
	return CE_Warning;
}

/*******************************************************************************/
/* =========================================================================== */
/*                     RasterIO()                                              */
/* =========================================================================== */
/*******************************************************************************/

/**
* Read a view from the remote image on the server
* @param pData buffer to fill
* @param nTimeoutMilliseconds connection timeout
* @papszOptions options for the driver implementation e.g. Quality="Xxx" to specify maximum jpip quality
*/
CPLErr RasterIO(void *pData, int nTimeoutMilliseconds = -1, char **papszOptions)
{
	// return a warning, this method should be overridden
	return CE_Warning;
}