Changes between Version 46 and Version 47 of WKTRaster/GDALDriverSpecificationWorking


Ignore:
Timestamp:
Jul 13, 2012, 7:01:13 AM (12 years ago)
Author:
dzwarg
Comment:

Updates to workflow & driver status after irregular raster arrangement mini-sprint by DZ

Legend:

Unmodified
Added
Removed
Modified

  • WKTRaster/GDALDriverSpecificationWorking

    v46 v47  
    1414}}}
    1515
    16 == '''Current status of the driver (April 2012)''' ==
     16== '''Current status of the driver (July 2012)''' ==
    1717
    1818The driver is:
    … …  
    2323   - ONE_RASTER_PER_TABLE ('mode = 2' in connection string): Each table is considered as a raster coverage, and each row is a raster tile.
    2424
    25  * Too slow (Reads the entire table metadata for constructing the [http://www.gdal.org/classGDALDataset.html GDALDataset object], and needs one server round per [http://www.gdal.org/classGDALRasterBand.html GDALRasterBand::IReadBlock] call)
    26 
     25 * Slow (Needs one server round per [http://www.gdal.org/classGDALRasterBand.html GDALRasterBand::IReadBlock] call)
     26 * Able to manage B, C, D, and F (partially) [http://trac.osgeo.org/postgis/raw-attachment/wiki/WKTRaster/Documentation01/WKTRasterArrangements.gif PostGIS Raster arrangements].
    2727The driver is not:
    2828 * Able to read out-db rasters
    2929 * Able to create new rasters
    30  * Able to manage all the [http://trac.osgeo.org/postgis/raw-attachment/wiki/WKTRaster/Documentation01/WKTRasterArrangements.gif PostGIS Raster arrangements]
     30 * Able to manage A, E, and F (fully) [http://trac.osgeo.org/postgis/raw-attachment/wiki/WKTRaster/Documentation01/WKTRasterArrangements.gif PostGIS Raster arrangements]
    3131 * Able to provide a color interpretation for bands
    3232
    … …  
    4747However, currently the driver only deals with continuous tiled raster layers, when all the raster tiles are the same size, snap to the same grid and do not overlap (the ideal case).
    4848
    49 '''Open question''': Are 2 working modes enough to manage all the raster arrangements? '''[SOLVED]: YES'''
     49'''UPDATE'''
     50The driver supports any raster arrangement on a regular grid. This means that tiles on the grid may be missing or sparse. The driver supports many rasters in one table if they are on different grids, and the "where" clause to the driver limits the selection to rasters that are on the same grid at one time.
     51
     52'''Question''': Are 2 working modes enough to manage all the raster arrangements? '''[SOLVED]: YES'''
    5053
    5154 Pierre: I think yes. We have to distinguish "want we want to produce" from "what we have to deal with". The two modes answer "want we want to produce" and the different table arrangement are "what we have to deal with".
    … …  
    7679  * Open the dataset (create db connection)
    7780  * Determine, in a 1st, very fast query to the db, by looking in the raster_overview view, what lower resolution table are available for the requested raster table
    78   * Determine, in a 2nd, fast enough query to the db, the extent and the maximum number of bands of the requested raster be aggregating the extents of all the rasters. This takes about 1 second on 360000 tiles even if there is no index.
     81  * Determine, in a 2nd, fast enough query to the db, the extent and the maximum number of bands of the requested raster be aggregating the extents of all the rasters. This takes about 1 second on 360000 tiles even if there is no index. '''[DONE]'''
     82  * Determine, in a 3rd, very fast query to the db, the pixel size & rotation, the band types and the nodata value for each band of ONLY ONE raster (LIMIT 1). The driver should assume those values will be the same for every other rasters in the table. If when fetching the other tiles, it realizes one does not, we must say that we do not support this arrangement. '''[DONE]'''
    7983
    80 Example:
    81 
    82 {{{
    83 SELECT ST_XMin(geom) xmin,
    84           ST_YMin(geom) ymin,
    85           ST_XMax(geom) xmax,
    86           ST_YMax(geom) ymax,
    87           nbband
    88    FROM (SELECT ST_Extent(rast::geometry) geom,
    89                 max(ST_NumBands(rast)) nbband
    90          FROM your_schema.raster_table
    91         ) foo
    92 }}}
    93 
    94 
    95   * Determine, in a 3rd, very fast query to the db, the pixel size & rotation, the band types and the nodata value for each band of ONLY ONE raster (LIMIT 1). The driver should assume those values will be the same for every other rasters in the table. If when fetching the other tiles, it realizes one does not, we must say that we do not support this arrangement. (I'm still a bit perplex about the nodata value though.)
    96 
    97 Example:
    98 
    99 {{{
    100 
    101    SELECT ST_ScaleX(rast) scalex,
    102           ST_ScaleY(rast) scaley,
    103           ST_SkewX(rast) skewx,
    104           ST_SkewY(rast) skewy, band,
    105           ST_BandPixelType(rast, band),
    106           ST_BandNodataValue(rast, band)
    107    FROM (SELECT rast, generate_series(1, ST_NumBands(rast)) band
    108          FROM your_schema.raster_table LIMIT 1
    109         ) foo
    110 }}}
    111 
    112 
    113 
    114 '''Open Question''': If in the first query we find a lower resolution table, does the rest of the work must be performed with this lower resolution table? At least these 3 queries, until we want to read the actual raster data to burn it into the buffer. The queries should be faster in an overview table, but the pixel size will not be the same using an overview table instead the normal resolution table. And you don't read from overviews unless you want to implement decimation because your buffer size is different from your raster size. Am I right?
     84'''Question''': If in the first query we find a lower resolution table, does the rest of the work must be performed with this lower resolution table? At least these 3 queries, until we want to read the actual raster data to burn it into the buffer. The queries should be faster in an overview table, but the pixel size will not be the same using an overview table instead the normal resolution table. And you don't read from overviews unless you want to implement decimation because your buffer size is different from your raster size. Am I right?
    11585
    11686  Pierre: Yes to the first question. This is why we query the overview table first. Yes you're right: I guest the driver must be demanding for a lower resolution only when a specific pixel size is requested from an application. right? Otherwise it request only the highest resolution one or all of them (in the case of gdal_translate for example) right? Hos does the driver knows if he has to produce only a specific resolution, only the best one or all of them?
    … …  
    12494
    12595{{{
    126 GDALRasterBand::IRasterIO(required tile metadata) {
     96#!c
     97/* Perform all read & write operations */
     98CPLErr PostGISRasterDataset::IRasterIO(GDALRWFlag eRWFlag, int nXOff, int nYOff, int nXSize, int nYSize, void * pData, int nBufXSize, int nBufYSize, GDALDataType eBufType, int nBandCount, int *panBandMap, int nPixelSpace, int nLineSpace, int nBandSpace)
     99{
     100  // Deduce a world coordinate rectangle which four corners are the center of the upper left, lower left, lower right, upper right pixels of the area requested.
     101 
     102  // Fetch, in a unique SQL query, every rasters intersecting this area along with the upper left X & Y, width, height, scalex, scaleY, skewX and skewY
     103 
     104  // If the blocks are not cached, cache them with this strategy:
     105  if (!bBlocksCached) {
     106    // If the raster arrangement is a regular grid
     107    if (ST_IsRegularlyTiled()) {
    127108
    128   Deduce a world coordinate rectangle which four corners are the center of the upper left, lower left, lower right, upper right pixels of the area requested.
    129  
    130   Fetch, in a unique SQL query, every rasters intersecting this area along with the upper left X & Y, width, height, scalex, scaleY, skewX and skewY
    131  
    132   If there is only one row fetched and the metadata of this raster fits the required tile metadata (meaning we are querying based on the natural block size)
     109      // Delegate to PostGISRasterBand->RasterIO(), which passes down to PostGISRasterBand::ReadBlock (see below)
     110    }
     111   
     112    // If the raster arrangement is not on a regular grid
     113    else {
    133114
    134      just copy the values as a block (memcopy) (is this possible? Should we have/isn't there a ST_Bytea() SQL function?)
    135        
    136   If not iterate on the required tile
     115      // memcpy rows one by one from each pixel line that lies within the intersecting area (that means if there is overlaps, only the pixel lines of the last raster fetched is burned. We can enhance this later by providing a BLENDING_MODE with the driver)
     116    }
    137117
    138      copy pixel values one by one from each raster fetched (that means if there is overlaps, only the pixels of the last raster fetched is burned. We can enhace this later by providing a BLENDING_MODE with the driver)
     118    bBlocksCached = TRUE;
     119  }
    139120
     121  // Once everything is cached, delegate reading of cached blocks with GDALDataSet->RasterIO()
    140122}
    141123}}}
    … …  
    146128
    147129{{{
    148 GDALRasterBand::!ReadBlock(block x & y){
     130#!c
     131/* This will only be called if the PostGISDataset detects that the raster is a regularly tiled table. */
     132CPLErr PostGISRasterRasterBand::IReadBlock(int nBlockXOff, int nBlockYOff, void* pImage)
     133{
     134  // Deduce IRasterIO required tile metadata
    149135
    150   Deduce IRasterIO required tile metadata
    151 
    152   call IRasterIO(required tile metadata)
     136  // memcpy raster binary data into the pImage
    153137}
    154138}}}