= GDAL WKT Raster format driver =

[[TOC]]

This was one of the selected projects for Google Summer of Code 2009. The goal is the implementation of read-only GDAL driver for WKT Raster extension to PostGIS. A prototype is working for the end of GSoC (August 17h 2009), but the code is under development at the time of the last update: '''August 17th 2009'''

== PostGIS WKT Raster type ==

The PostGIS WKT Raster type is an extension of PostGIS aiming at developing support for raster. Each raster is read from disk and stored at a PostgreSQL database with PostGIS and WKT extensions using the [https://svn.osgeo.org/postgis/spike/wktraster/scripts/gdal2wktraster.py gdal2wktraster] script. In the future, creation of new WKT Rasters directly from GDAL code will be allowed too.

Each WKT Raster is represented by a PostgreSQL table with a column of the new type "raster". All the tables with a "raster" column are registered in the RASTER_COLUMNS table. This new table:
  "(...) is a mean for applications to get a quick overview of which table have a raster column and to get the main characteristics (metadata) of the rasters stored in these columns"

You can check the structure of the RASTER_COLUMNS table in the [http://trac.osgeo.org/postgis/wiki/WKTRaster/SpecificationWorking01#RASTER_COLUMNSMetadataTable WKTRaster final specification page].

This "raster" format is expressed in different form depending of the level at which it is refered. The format in what the raster is written to the database is the [https://svn.osgeo.org/postgis/spike/wktraster/doc/RFC1-SerializedFormat serialized format], but the raster is read by GDAL WKT Raster driver in the [https://svn.osgeo.org/postgis/spike/wktraster/doc/RFC2-WellKnownBinaryFormat Hexadecimal WKB format]. This is the format what you get when outputting the value of a raster field by a query like "SELECT rast FROM table".

NOTE: The "int8" data type in Hex WKB format is interpreted like a datatype "Byte" by the GDAL WKT Raster driver.

Each row of a table with a column of the type "raster" will represent an '''image tile''', a '''raster object coverage''' (the result from the rasterization of a vector coverage) or a '''whole image''', unrelated with the rest of the table's rows. Only the first table arrangement (one row = one image tile) is fully understood by the GDAL WKT Raster driver at the current state (see last update). The rest of arrangements will be discussed with WKT Raster development team.

Focusing on this first arrangement, we have 2 different cases:
  1. Regularly tiled images
  1. Irregularly tiled images

In the first case, we say that the table with this arrangement is a "regularly blocked table". This implies that:
  1. All loaded tiles have the same width and height
  1. All tiles do not overlap and their upper left corner follows a regular block grid
  1. The global extent of the layer is rectangular and not rotated.

If not all the points are carried out for a given table, it will be an "irregularly blocked table".

Each WKT Raster could have zero or more overviews. Each overview is another raster table, like the original one. Currently, all the information of the overviews of a WKT Raster table is stored in another metadata table, called [http://trac.osgeo.org/postgis/wiki/WKTRaster/SpecificationWorking01#RASTER_OVERVIEWSMetadataTable RASTER_OVERVIEWS]. But the overviews support is still a working specification, not a final accepted one.

You can dump all the overviews of a WKT Raster table by using the '''dumpoverviews''' utility. I've slightly modified the ''dumpoverviews.cpp'' code to accept connection strings (see next section, ''Using the GDAL WKT Raster driver'') as valid datasets. 

And about the raster data storage, the WKT Raster format offers 2 ways of storing the data of a raster band:
  * in-db storage: The band data is stored in the database, in serialized format with the rest of the band's data.
  * out-db storage: The band data is registered in an external file residing in the file system.

At present, only the first storage system is allowed in WKT Raster code, but the second one is planned to be ready on this year (2009). Anyway, I've modified the loader script to add outdb support. The patch for the original code is in [http://trac.osgeo.org/postgis/ticket/227 PostGIS ticket 227]


You can find further information on the WKT Raster type at [http://trac.osgeo.org/postgis/wiki/WKTRaster WKT Raster home page]


== Using the GDAL WKT Raster driver ==

If you want to use the GDAL WKT Raster driver, you must provide a '''connection string''' as Dataset's name. The syntax of this connection string is (respect the quotes):


{{{
 PG":host='<host>' dbname='<dbname>' user='<user>' password='<password>' table='<raster_table>' [where='<sql_where>' mode='<working_mode>']"
}}}

Note that the string, until the part that starts with "table='" is a libpq-style connection string. That means that you can change the order of these fields (dbname, user, password, host), or leave out unnecessary ones (like password, in some cases). But the rest of the connection string must have the syntax and order shown above.

The "table" option requires the name of a WKT Raster table. This table only can be created by using the script gdal2tiles, from [http://trac.osgeo.org/postgis/wiki/WKTRaster WKTRaster code]

The "where" option is used to filter the results of the raster table. Any SQL-WHERE expression is valid. The "mode" option is used to know the expected arrangement of the raster table. As the driver is currently working with only one table arrangement (regularly blocked tables), you can omit this option, or use it with value "REGULARLY_TILED_MODE". Otherwise, the driver won't work.

You must use this dataset's format in all the gdal tools, like gdalinfo, gdal_translate, gdalwarp, etc.

== TODO Tasks ==

The relevant TODO tasks, in planned order of implementation, are:

  1. Block reading improvement. Avoid one server round for each block reading call. (IRasterIO overriding).
  1. Out-db data reading support. (IRasterIO overriding).
  1. RASTER_COLUMNS table update with the values read on data blocks, if needed.
  1. Support for reading non-regularly blocked rasters.
  1. Bulk update of data blocks when any raster-related value changes on RASTER_COLUMNS table.
  1. Support for creating new WKT Rasters.

This list is always under revision.

== Development status at last update ==

The GDAL WKT Raster driver is able to read regularly blocked rasters stored in-db.

== Project plan ==

||'''Objectives and tasks'''||'''Approx. Schedule'''||'''Status'''||
||'''Objective 1 - Prototype of GDAL WKT Raster read-only driver'''||'''17th August'''||'''Done'''||
||'''Objective 2 - Block reading improvement'''||'''undecided'''||'''On going'''||
||'''Objective 3 - Out-db data reading support'''||'''undecided'''||'''On going'''||
||'''Objective 4 - Modify some GDAL tools (ex: gdal2tiles) to accept GDAL WKT Raster connection strings'''||'''undecided'''||'''Todo'''||
||'''Objective 5 - RASTER_COLUMNS table update'''||'''undecided'''||'''Todo'''||
||'''Objective 6 - Support for reading non-regularly blocked rasters'''||'''undecided'''||'''Todo'''||
||'''Objective 7 - Bulk update of data blocks'''||'''undecided'''||'''Todo'''||
||'''Objective 8 - Support for creating new WKT Rasters'''||'''undecided'''||'''Todo'''||


== GSoC 09 Weekly reports ==
[http://www.gis4free.org/blog/2009/05/30/gsoc-09-weekly-report-1-2305-2905/ Weekly report #1 (23/05 - 29/05)][[BR]]
[http://www.gis4free.org/blog/2009/06/09/gsoc-09-weekly-report-2-2905-0506/ Weekly report #2 (29/05 - 05/06)][[BR]]
[http://www.gis4free.org/blog/2009/06/16/gsoc-09-weekly-report-3-0506-1206/ Weekly report #3 (05/06 - 12/06)][[BR]]
[http://www.gis4free.org/blog/2009/06/20/gsoc-09-weekly-report-4-1206-1906/ Weekly report #4 (12/06 - 19/06)][[BR]]
[http://www.gis4free.org/blog/2009/06/27/gsoc-09-weekly-report-5-1906-2606/ Weekly report #5 (19/06 - 26/06)][[BR]]
[http://www.gis4free.org/blog/2009/07/04/gsoc-09-weekly-report-6-2606-0307/ Weekly report #6 (26/06 - 03/07)][[BR]]
[http://www.gis4free.org/blog/2009/07/13/gsoc-09-weekly-report-7-0307-1007/ Weekly report #7 (03/07 - 10/07)][[BR]]
[http://www.gis4free.org/blog/2009/07/20/gsoc-09-weekly-report-8-1007-1707/ Weekly report #8 (10/07 - 17/07)][[BR]]
[http://www.gis4free.org/blog/2009/07/26/gsoc-09-weekly-report-9-1707-2407/ Weekly report #9 (17/07 - 24/07)][[BR]]
[http://www.gis4free.org/blog/2009/08/02/gsoc-09-weekly-report-10-2407-3107/ Weekly report #10 (24/07 - 31/07)][[BR]]
[http://www.gis4free.org/blog/2009/08/11/gsoc-09-weekly-report-11-3107-0708/ Weekly report #11 (31/07 - 07/08)][[BR]]
[http://www.gis4free.org/blog/2009/08/17/gsoc-09-final-report-0708-1708/ Final GSoC 09 Weekly report (07/08 - 17/08)]


== GSoC 09 Participants info ==
  * Student: Jorge Arévalo (jorgearevalo at gis4free.org)
  * Mentors: Tamas Szekeres, Frank Warmerdam