[raster] raster2pgsql loader aborts

[andreasft]

raster2pgsql loader aborts processing for some rasters returning error: raster2pgsql: rt_api.c:4281: rt_raster_set_srid: Assertion `((void *)0) != raster' failed.

No pattern in which type of raster formats that crashes the loader.

Tried two different ArcInfo? Binary Grid Format. One works fine, the other crashes the import.

Crashes  http://bioval.jrc.ec.europa.eu/products/gam/download/access_50k.zip (200mb)

Works fine  http://sedac.ciesin.columbia.edu/gpw/global.jsp?file=gpwv3&data=pcount&type=wrk&resolut=half&year=90&version=gpw-v3 (461 kb)

Loader syntax crashing raster: raster2pgsql -s 4326 -I w001001.adf access > home/andreas/access.sql Processing 1/1: /mnt/pc258/prio_grid/source/access/access_50k/acc_50k/w001001.adf Warning 6: A dataset opened by GDALOpenShared should have the same filename (/mnt/pc258/prio_grid/source/access/access_50k/acc_50k/w001001.adf) and description (/mnt/pc258/prio_grid/source/access/access_50k/acc_50k) raster2pgsql: rt_api.c:4281: rt_raster_set_srid: Assertion `((void *)0) != raster' failed. Aborted

Loader syntax working raster: raster2pgsql -s 4326 -I /w001001.adf poptest > /home/andreas/poptest1.sql Processing 1/1: /mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30/w001001.adf Warning 6: A dataset opened by GDALOpenShared should have the same filename (/mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30/w001001.adf) and description (/mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30)

[andreasft]

Information about each file from gdal_info:

Driver: AIG/Arc/Info Binary Grid Files: c:\prio_grid\source\gpw\lrc30p90\glp90ag30

c:\prio_grid\source\gpw\lrc30p90\glp90ag30.aux c:\prio_grid\source\gpw\lrc30p90\glp90ag30\dblbnd.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.dbf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.prj c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.shp c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.shx c:\prio_grid\source\gpw\lrc30p90\glp90ag30\hdr.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\log c:\prio_grid\source\gpw\lrc30p90\glp90ag30\prj.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\sta.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\w001001.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\w001001x.adf

Size is 720, 286 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0,

AUTHORITY["EPSG","8901"]],

UNIT["degree",0.0174532925199433,

AUTHORITY["EPSG","9108"]],

AUTHORITY["EPSG","4326"]]

Origin = (-180.000000000000000,85.000000000114397) Pixel Size = (0.500000000000400,-0.500000000000400) Corner Coordinates: Upper Left (-180.0000000, 85.0000000) Lower Left (-180.0000000, -58.0000000) Upper Right ( 180.0000000, 85.0000000) Lower Right ( 180.0000000, -58.0000000) Center ( 0.0000000, 13.5000000) Band 1 Block=360x4 Type=Float32, ColorInterp?=Undefined

Min=0.000 Max=16172424.000 NoData? Value=-3.4028234663852886e+038 Metadata:

LAYER_TYPE=athematic

Driver: AIG/Arc/Info Binary Grid Files: c:\prio_grid\source\Access\access_50k\acc_50k

c:\prio_grid\source\Access\access_50k\acc_50k.aux c:\prio_grid\source\Access\access_50k\acc_50k.rrd c:\prio_grid\source\Access\access_50k\acc_50k.aux.xml c:\prio_grid\source\Access\access_50k\acc_50k\dblbnd.adf c:\prio_grid\source\Access\access_50k\acc_50k\hdr.adf c:\prio_grid\source\Access\access_50k\acc_50k\log c:\prio_grid\source\Access\access_50k\acc_50k\metadata.xml c:\prio_grid\source\Access\access_50k\acc_50k\prj.adf c:\prio_grid\source\Access\access_50k\acc_50k\sta.adf c:\prio_grid\source\Access\access_50k\acc_50k\vat.adf c:\prio_grid\source\Access\access_50k\acc_50k\w001001.adf c:\prio_grid\source\Access\access_50k\acc_50k\w001001x.adf

Size is 43200, 21600 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0,

AUTHORITY["EPSG","8901"]],

UNIT["degree",0.0174532925199433,

AUTHORITY["EPSG","9108"]],

AUTHORITY["EPSG","4326"]]

Origin = (-180.000000000000000,89.999999999280021) Pixel Size = (0.008333333333300,-0.008333333333300) Corner Coordinates: Upper Left (-180.0000000, 90.0000000) Lower Left (-180.0000000, -90.0000000) Upper Right ( 180.0000000, 90.0000000) Lower Right ( 180.0000000, -90.0000000) Center ( -0.0000000, -0.0000000) Band 1 Block=256x4 Type=Int32, ColorInterp?=Undefined

Min=0.000 Max=108536.000 Minimum=0.000, Maximum=108536.000, Mean=1543.449, StdDev?=2736.378 NoData? Value=-2147483647 Overviews: 10800x5400, 5400x2700, 2700x1350, 1350x675, 675x338, 338x169, 169x8

5, 85x43

Metadata:

LAYER_TYPE=athematic STATISTICS_MINIMUM=0 STATISTICS_MAXIMUM=108536 STATISTICS_MEAN=1543.4487145611 STATISTICS_STDDEV=2736.3775076303

[dustymugs]

I just tested the access_50k.zip and when run using the following command causes an abort:

raster2pgsql -s 4326 -I acc_50k access > access.sql

The following command does work due to the addition of the tiling flag:

raster2pgsql -t 50x50 -s 4326 -I acc_50k access > access.sql

From the looks of the raster, it has 43200x21600 pixels of datatype Int32. The loader attempts to allocate ~3732480000 bytes for the band and fails.

The loader uses PostGIS raster's C API for handling the conversion from GDAL to raster hex. I could replace the use of the API with a stream processor but I don't know if it is worth the effort. Everything in PostGIS raster is performed in memory with no disk access (for temp files) but it may need to be considered if users are trying to load large rasters without tiling.

It is strongly encouraged to tile your rasters as it does increase the performance of any queries run against the rasters.

[strk]

Would still be nice to get an "Out of memory" message rather than aborting. Would reduce user frustration.

[dustymugs]

Hooked the loader into rterror, rtwarn and rtinfo functions so that messages from the C API are emitted properly in r9012