Opened 12 years ago

Closed 12 years ago

Last modified 9 years ago

#1531 closed defect (wontfix)

[raster] raster2pgsql loader aborts

Reported by: andreasft Owned by: Bborie Park
Priority: medium Milestone: PostGIS 2.0.0
Component: raster Version: master
Keywords: raster2pgsql loader Cc:

Description

raster2pgsql loader aborts processing for some rasters returning error: raster2pgsql: rt_api.c:4281: rt_raster_set_srid: Assertion `((void *)0) != raster' failed.

No pattern in which type of raster formats that crashes the loader.

Tried two different ArcInfo Binary Grid Format. One works fine, the other crashes the import.

Crashes http://bioval.jrc.ec.europa.eu/products/gam/download/access_50k.zip (200mb)

Works fine http://sedac.ciesin.columbia.edu/gpw/global.jsp?file=gpwv3&data=pcount&type=wrk&resolut=half&year=90&version=gpw-v3 (461 kb)

Loader syntax crashing raster: raster2pgsql -s 4326 -I w001001.adf access > home/andreas/access.sql Processing 1/1: /mnt/pc258/prio_grid/source/access/access_50k/acc_50k/w001001.adf Warning 6: A dataset opened by GDALOpenShared should have the same filename (/mnt/pc258/prio_grid/source/access/access_50k/acc_50k/w001001.adf) and description (/mnt/pc258/prio_grid/source/access/access_50k/acc_50k) raster2pgsql: rt_api.c:4281: rt_raster_set_srid: Assertion `((void *)0) != raster' failed. Aborted

Loader syntax working raster: raster2pgsql -s 4326 -I /w001001.adf poptest > /home/andreas/poptest1.sql Processing 1/1: /mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30/w001001.adf Warning 6: A dataset opened by GDALOpenShared should have the same filename (/mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30/w001001.adf) and description (/mnt/pc258/prio_grid/source/gpw/lrc30p90/glp90ag30)

Change History (6)

comment:1 by andreasft, 12 years ago

Information about each file from gdal_info:

Driver: AIG/Arc/Info Binary Grid Files: c:\prio_grid\source\gpw\lrc30p90\glp90ag30

c:\prio_grid\source\gpw\lrc30p90\glp90ag30.aux c:\prio_grid\source\gpw\lrc30p90\glp90ag30\dblbnd.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.dbf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.prj c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.shp c:\prio_grid\source\gpw\lrc30p90\glp90ag30\gpw90.shx c:\prio_grid\source\gpw\lrc30p90\glp90ag30\hdr.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\log c:\prio_grid\source\gpw\lrc30p90\glp90ag30\prj.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\sta.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\w001001.adf c:\prio_grid\source\gpw\lrc30p90\glp90ag30\w001001x.adf

Size is 720, 286 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0,

AUTHORITY["EPSG","8901"]],

UNIT["degree",0.0174532925199433,

AUTHORITY["EPSG","9108"]],

AUTHORITY["EPSG","4326"]]

Origin = (-180.000000000000000,85.000000000114397) Pixel Size = (0.500000000000400,-0.500000000000400) Corner Coordinates: Upper Left (-180.0000000, 85.0000000) Lower Left (-180.0000000, -58.0000000) Upper Right ( 180.0000000, 85.0000000) Lower Right ( 180.0000000, -58.0000000) Center ( 0.0000000, 13.5000000) Band 1 Block=360x4 Type=Float32, ColorInterp=Undefined

Min=0.000 Max=16172424.000 NoData Value=-3.4028234663852886e+038 Metadata:

LAYER_TYPE=athematic

Driver: AIG/Arc/Info Binary Grid Files: c:\prio_grid\source\Access\access_50k\acc_50k

c:\prio_grid\source\Access\access_50k\acc_50k.aux c:\prio_grid\source\Access\access_50k\acc_50k.rrd c:\prio_grid\source\Access\access_50k\acc_50k.aux.xml c:\prio_grid\source\Access\access_50k\acc_50k\dblbnd.adf c:\prio_grid\source\Access\access_50k\acc_50k\hdr.adf c:\prio_grid\source\Access\access_50k\acc_50k\log c:\prio_grid\source\Access\access_50k\acc_50k\metadata.xml c:\prio_grid\source\Access\access_50k\acc_50k\prj.adf c:\prio_grid\source\Access\access_50k\acc_50k\sta.adf c:\prio_grid\source\Access\access_50k\acc_50k\vat.adf c:\prio_grid\source\Access\access_50k\acc_50k\w001001.adf c:\prio_grid\source\Access\access_50k\acc_50k\w001001x.adf

Size is 43200, 21600 Coordinate System is: GEOGCS["WGS 84",

DATUM["WGS_1984",

SPHEROID["WGS 84",6378137,298.257223563,

AUTHORITY["EPSG","7030"]],

TOWGS84[0,0,0,0,0,0,0], AUTHORITY["EPSG","6326"]],

PRIMEM["Greenwich",0,

AUTHORITY["EPSG","8901"]],

UNIT["degree",0.0174532925199433,

AUTHORITY["EPSG","9108"]],

AUTHORITY["EPSG","4326"]]

Origin = (-180.000000000000000,89.999999999280021) Pixel Size = (0.008333333333300,-0.008333333333300) Corner Coordinates: Upper Left (-180.0000000, 90.0000000) Lower Left (-180.0000000, -90.0000000) Upper Right ( 180.0000000, 90.0000000) Lower Right ( 180.0000000, -90.0000000) Center ( -0.0000000, -0.0000000) Band 1 Block=256x4 Type=Int32, ColorInterp=Undefined

Min=0.000 Max=108536.000 Minimum=0.000, Maximum=108536.000, Mean=1543.449, StdDev=2736.378 NoData Value=-2147483647 Overviews: 10800x5400, 5400x2700, 2700x1350, 1350x675, 675x338, 338x169, 169x8

5, 85x43

Metadata:

LAYER_TYPE=athematic STATISTICS_MINIMUM=0 STATISTICS_MAXIMUM=108536 STATISTICS_MEAN=1543.4487145611 STATISTICS_STDDEV=2736.3775076303

comment:2 by Bborie Park, 12 years ago

Owner: changed from pracine to Bborie Park
Status: newassigned
Summary: raster2pgsql loader aborts[raster] raster2pgsql loader aborts

comment:3 by Bborie Park, 12 years ago

Resolution: wontfix
Status: assignedclosed

I just tested the access_50k.zip and when run using the following command causes an abort:

raster2pgsql -s 4326 -I acc_50k access > access.sql

The following command does work due to the addition of the tiling flag:

raster2pgsql -t 50x50 -s 4326 -I acc_50k access > access.sql

From the looks of the raster, it has 43200x21600 pixels of datatype Int32. The loader attempts to allocate ~3732480000 bytes for the band and fails.

The loader uses PostGIS raster's C API for handling the conversion from GDAL to raster hex. I could replace the use of the API with a stream processor but I don't know if it is worth the effort. Everything in PostGIS raster is performed in memory with no disk access (for temp files) but it may need to be considered if users are trying to load large rasters without tiling.

It is strongly encouraged to tile your rasters as it does increase the performance of any queries run against the rasters.

comment:4 by strk, 12 years ago

Would still be nice to get an "Out of memory" message rather than aborting. Would reduce user frustration.

comment:5 by Bborie Park, 12 years ago

Hooked the loader into rterror, rtwarn and rtinfo functions so that messages from the C API are emitted properly in r9012

comment:6 by fil, 9 years ago

It seems that the original dataset ( ​http://bioval.jrc.ec.europa.eu/products/gam/download/access_50k.zip ) has all but disappeared from the net.

Note: See TracTickets for help on using tickets.