Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#200 closed task (fixed)

[wktraster] Resolution reduction built-in to gdal2wktraster script

Reported by: mloskot Owned by: mloskot
Priority: high Milestone: WKTRaster 0.1.6
Component: raster Version: master
Keywords: overview, pyramid, resolution, reduction Cc:

Description

Current version (r4163) of gdal2wktraster.py only generates tables with overviews if these are available in input dataset. In other words, users need to generate overviews first, i.e. using gdaladdo, and then gdal2wktraster can dump these overviews to tables.

This task is to enhance the loader to rely on cool features of RasterIO. So, if overview generation is requested and overview level (a reduction factor, see notes below) is specified:

  • and the input raster does consist of overviews, then RasterIO will automa(t|g)ically fetch pixel data from overview matching given level at best.
  • but no overviews are available in the input raster, then RasterIO will apply raster decimation on source window, so it's simply and properly downsampled.

If specified overview factor is higher than 1 (i.e. 2, 4, ...) then size of output raster requested from RasterIO operation is adequately smaller than size of source raster. This difference will trigger raster downsampling.

Currently, gdal2wktraster dumps all available overview during single execution of the script. It will change. The enhanced version will generate only one type of raster output at once: base raster (level = 1) or overview (level > 1).

I believe it's an advantage because it is a long process to dump large raster. The new version will open it to allow execution of parallel processes.

The main goal of this ticket is to mark a snapshot of gdal2wktraster between major modifications, so it's easier to find and use old version if desirable.

Work in progress.

Change History (10)

comment:1 Changed 11 years ago by mloskot

Owner: changed from pracine to mloskot
Status: newassigned

comment:2 Changed 11 years ago by mloskot

First set of changes (r4219). Note, georeferencing per tile is likely broken as well as it's not very bulletproof yet. Second, if level > 1, the script must generate table with base raster but not only downsampled version (overview). Work in progress. Yes, progress, this is the right term ;-)

comment:3 Changed 11 years ago by mloskot

Cut along x-axis (r4220)

comment:4 Changed 11 years ago by mloskot

Getting the script more robust fixing some of the issues mentioned above (r4221). Georeferencing should work.

comment:5 Changed 11 years ago by mloskot

Re-organized command line options (r4226).

Status summary:

  • script is more oriented on regular blocking processing now
  • to request regular blocking, use -k,--block-size option
    ./gdal2wktraster.py -r test.tif -t test -k 256x256
    
  • to request overview (level), use -l,--overview-level option:
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 2
    
  • if level = 1 base raster is generated, not an overview
    ./gdal2wktraster.py -r test.tif -t test -k 256x256
    or
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 1
    
  • if regular blocking requested, then single execution == single input raster == single level (overview) generation. In order to generate base raster (level=1) and overview levels 2,4,8 user has to run 4 processes separately:
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 1
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 2
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 4
    ./gdal2wktraster.py -r test.tif -t test -k 256x256 -l 8
    
  • The approach of multi-process execution may change in future to generate all oberviews in single step with all levels specified as -l 2 4 8. However, current version may easily benefit of parallel execution.
  • After such a big change of how gdal2wktraster performs the raster crunching job, it's obvious bugs are in there and (re)testing is required.

I'm not closing this ticket yet and I will after I'm sure and happy with what I've done.

BTW, somebody could refactor the script to a nice modularised one, with i.e. WKTRasterCoverage class that encapsulates the whole job of cutting usable as a standalone module. Then the rest of processing becomes simpler. Volunteers?

comment:6 Changed 11 years ago by mloskot

Tidy up (r4228)

comment:7 Changed 11 years ago by mloskot

Fixed overview table name bug (r4230)

comment:8 Changed 11 years ago by mloskot

Resolution: fixed
Status: assignedclosed

I'm happy with the result and closing. Feel free to reopen or submit new ticket if something is not working here.

comment:9 Changed 11 years ago by mloskot

Summary: Resolution reduction built-in to gdal2wktraster script[wktraster] Resolution reduction built-in to gdal2wktraster script

comment:10 Changed 11 years ago by pracine

Milestone: WKTRaster 0.1.6
Note: See TracTickets for help on using tickets.