GDAL WKT Raster format driver

This was one of the selected projects for Google Summer of Code 2009. The goal is the implementation of read-only GDAL driver for WKT Raster extension to PostGIS. A prototype is working for the end of GSoC (August 17h 2009), but the code is under development at the time of the last update: August 17th 2009

PostGIS WKT Raster type

The PostGIS WKT Raster type is an extension of PostGIS aiming at developing support for raster. Each raster is read from disk and stored at a PostgreSQL database with PostGIS and WKT extensions using the ​gdal2wktraster script. In the future, creation of new WKT Rasters directly from GDAL code will be allowed too.

Each WKT Raster is represented by a PostgreSQL table with a column of the new type "raster". All the tables with a "raster" column are registered in the RASTER_COLUMNS table. This new table:

"(...) is a mean for applications to get a quick overview of which table have a raster column and to get the main characteristics (metadata) of the rasters stored in these columns"

You can check the structure of the RASTER_COLUMNS table in the ​WKTRaster final specification page.

This "raster" format is expressed in different form depending of the level at which it is refered. The format in what the raster is written to the database is the ​serialized format, but the raster is read by GDAL WKT Raster driver in the ​Hexadecimal WKB format. This is the format what you get when outputting the value of a raster field by a query like "SELECT rast FROM table".

NOTE: The "int8" data type in Hex WKB format is interpreted like a datatype "Byte" by the GDAL WKT Raster driver.

Each row of a table with a column of the type "raster" will represent an image tile, a raster object coverage (the result from the rasterization of a vector coverage) or a whole image, unrelated with the rest of the table's rows. Only the first table arrangement (one row = one image tile) is fully understood by the GDAL WKT Raster driver at the current state (see last update). The rest of arrangements will be discussed with WKT Raster development team.

Focusing on this first arrangement, we have 2 different cases:

1. Regularly tiled images
2. Irregularly tiled images

In the first case, we say that the table with this arrangement is a "regularly blocked table". This implies that:

1. All loaded tiles have the same width and height
2. All tiles do not overlap and their upper left corner follows a regular block grid
3. The global extent of the layer is rectangular and not rotated.

If not all the points are carried out for a given table, it will be an "irregularly blocked table".

Each WKT Raster could have zero or more overviews. Each overview is another raster table, like the original one. Currently, all the information of the overviews of a WKT Raster table is stored in another metadata table, called ​RASTER_OVERVIEWS. But the overviews support is still a working specification, not a final accepted one. And about the raster data storage, the WKT Raster format offers 2 ways of storing the data of a raster band:

• in-db storage: The band data is stored in the database, in serialized format with the rest of the band's data.
• out-db storage: The band data is registered in an external file residing in the file system.

At present, only the first storage system is allowed in WKT Raster code, but the second one is planned to be ready on this year (2009). Anyway, I've modified the loader script to add outdb support. The patch for the original code is in ​PostGIS ticket 227

You can find further information on the WKT Raster type at ​WKT Raster home page

Using the GDAL WKT Raster driver

If you want to use the GDAL WKT Raster driver, you must provide a connection string as Dataset's name. The syntax of this connection string is (respect the quotes):

 PG":host='<host>' dbname='<dbname>' user='<user>' password='<password>' table='<raster_table>' [where='<sql_where>' mode='<working_mode>']"

Note that the string, until the part that starts with "table='" is a libpq-style connection string. That means that you can change the order of these fields (dbname, user, password, host), or leave out unnecessary ones (like password, in some cases). But the rest of the connection string must have the syntax and order shown above.

The "where" option is used to filter the results of the raster table. Any SQL-WHERE expression is valid. The "mode" option is used to know the expected arrangement of the raster table. As the driver is currently working with only one table arrangement (regularly blocked tables), you can omit this option, or use it with value "REGULARLY_TILED_MODE". Otherwise, the driver won't work.

You must use this dataset's format in all the gdal tools, like gdalinfo, gdal_translate, gdalwarp, etc.

TODO Tasks

The relevant TODO tasks, in planned order of implementation, are:

1. Block reading improvement. Avoid one server round for each block reading call. (IRasterIO overriding).
2. Out-db data reading support. (IRasterIO overriding).
3. RASTER_COLUMNS table update with the values read on data blocks, if needed.
4. Support for reading non-regularly blocked rasters.
5. Bulk update of data blocks when any raster-related value changes on RASTER_COLUMNS table.
6. Support for creating new WKT Rasters.

This list is always under revision.

Development status at last update

The GDAL WKT Raster driver is able to read regularly blocked rasters stored in-db.

Project plan

Objectives and tasks	Approx. Schedule	Status
Objective 1 - Prototype of GDAL WKT Raster read-only driver	17th August	Done
Objective 2 - Block reading improvement	undecided	On going
Objective 3 - Out-db data reading support	undecided	On going
Objective 4 - Modify some GDAL tools (ex: gdal2tiles) to accept GDAL WKT Raster connection strings	undecided	Todo
Objective 5 - RASTER_COLUMNS table update	undecided	Todo
Objective 6 - Support for reading non-regularly blocked rasters	undecided	Todo
Objective 7 - Bulk update of data blocks	undecided	Todo
Objective 8 - Support for creating new WKT Rasters	undecided	Todo

GSoC 09 Weekly reports

​Weekly report #1 (23/05 - 29/05)
​Weekly report #2 (29/05 - 05/06)
​Weekly report #3 (05/06 - 12/06)
​Weekly report #4 (12/06 - 19/06)
​Weekly report #5 (19/06 - 26/06)
​Weekly report #6 (26/06 - 03/07)
​Weekly report #7 (03/07 - 10/07)
​Weekly report #8 (10/07 - 17/07)
​Weekly report #9 (17/07 - 24/07)
​Weekly report #10 (24/07 - 31/07)
​Weekly report #11 (31/07 - 07/08)
​Final GSoC 09 Weekly report (07/08 - 17/08)

GSoC 09 Participants info

• Student: Jorge Arévalo (jorgearevalo at gis4free.org)
• Mentors: Tamas Szekeres, Frank Warmerdam