[raster]: Can't read out_db jpg in some cases

[robe]

This might be similar issue to what we've had before, but I get a set of project photos, but unfortunately all the ones exhibiting the problem are too big to attach.

When I do this or ST_AsPNG

SELECT ST_Resize(rast,0.1,0.1) FROM armory_outdb limit 1

I get this error

NOTICE:  The in-db representation of the out-db raster is not aligned. Band data may be incorrect

CONTEXT:  PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN
ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN

This issue doesn't happen if I load in db.

Command I used is:

raster2pgsql -F -Y -R C:/pics/back.jpg armory_outdb | psql -U postgres -d testpostgis21 -h localhost -p 5442

POSTGIS="2.1.0SVN r11085" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" (core procs from "2.1.0SVN r11079" need upgrade) RASTER (raster procs from "2.1.0SVN r11079" need upgrade)

I'll send you link in a bit.

[dustymugs]

Yep. I know that warning. Can you post the output of gdalinfo of the source raster file and ST_Metadata() of the out-db raster?

[robe]

gdalinfo

Driver: JPEG/JPEG JFIF
Files: back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
  EXIF_ColorSpace=1
  EXIF_DateTime=2012:11:28 15:18:54
  EXIF_Orientation=1
  EXIF_PixelXDimension=6000
  EXIF_PixelYDimension=4500
  EXIF_ResolutionUnit=2
  EXIF_Software=Adobe Photoshop CS3 Windows
  EXIF_XResolution=(150)
  EXIF_YResolution=(150)
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 4500.0)
Upper Right ( 6000.0,    0.0)
Lower Right ( 6000.0, 4500.0)
Center      ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
  Overviews: 3000x2250, 1500x1125, 750x563
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
  Overviews: 3000x2250, 1500x1125, 750x563
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
  Overviews: 3000x2250, 1500x1125, 750x563
  Image Structure Metadata:
    COMPRESSION=JPEG

I think on my server I was getting a can't create VRT or something (I presume because it has overviews?) (as I recall it wouldn't even load some of the pics using in db, though perhaps this particular one was not one I tested on prod server)

SELECT (ST_MetaData(rast)).*
from armory_outdb;

outputs

 upperleftx | upperlefty | width | height | scalex | scaley | skewx | skewy | srid| numbands
------------+------------+-------+--------+--------+--------+-------+-------+---
---+----------
          0 |          0 |  6000 |   4500 |      1 |     -1 |     0 |     0 |
 0 |        3

[dustymugs]

Fixed in r11094. Decided to change how handling of rasters with no spatial information is done.

[robe]

Still doesn't seem to be working for me for this example though I'm not getting the NOTICE anymore.

Just this:

ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN

Running latest winnie build —

select version() || ' ' || postgis_full_version();

PostgreSQL 9.2.2, compiled by Visual C++ build 1600, 64-bit POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER

[dustymugs]

The NOTICE is a flag for metadata differences between what GDAL sees in the raster file and what is inside the database. This message should never happen normally.

The error though is due to an error with GDALRasterIO() unable to read the data. I'll need to test with 1.9.2.

[dustymugs]

Are you able to run something even simpler than ST_Resize()? Something like…

SELECT ST_SummaryStats(rast) FROM armory_outdb limit 1

All out-db rasters are handled in the exact same manner so the above should result in the same error.

[dustymugs]

I can't replicate the problem using the JPEG you provided…

PostgreSQL 9.2.1 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.5.2, 64-b
it POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3755" PROJ="Rel. 4.8.0, 
6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNK
NOWN" RASTER

[robe]

Hmm that's odd. I thought maybe it was a permssion problem but I think I have it in same folder with where I put other images. I'll retest again on the original server I was having the issue and close this out if its a non-issue there.

does your change effect raster2pgsql. Its possible I didn't reload?

[robe]

Okay I just reloaded — here is what I found:

SELECT ST_SummaryStats(rast) FROM armory_outdb limit 1;

(27000000,2665449805,98.7203631481482,81.4870806656832,0,255)

Run the above a second time KABOOM server instance crashes and comes toppling down

Resize still doesn't work and to rule out the folder, I added another image to it and registered that file as an out db and that one works fine.

If it helps when I run resize I get this extra info in the database logs:

ERROR 1: libjpeg: Failed to create temporary file 
ERROR 1: C:/pics/back.jpg, band 2: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0

I'm going to try next on my mingw instance to see if its an EDB windows instance issue. haven't tried on my 32-bit instance yet either. I have 5GB of disk space left so I assume that is not the issue. I think I'm running with default postgres.conf for my development instance.

[robe]

Before I do the mingw test, I just tried on the production server having the original issue which is a Windows 2008 R2 64-bit running:

PostgreSQL 9.2.3, compiled by Visual C++ build 1600, 64-bit PostGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3755" PROJ="Rel. 4.8.0, 
6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNK
NOWN" RASTER

I can't even get the in db to work on this one with this image. This is the same notice I was getting before on this one before the upgrade too: Sorry can't cut and paste fromt his server so bare with me

ERROR 1: libjpeg: Failed to create temporary file 
ERROR 1: E:\gisdata\pics\armory\back.jpg, band 2: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
ERROR: could not convert VRT dataset to PostGIS raster
ERROR: process_rasters: Could not process raster: E:\gisdata\pics\armory\back.jpg

A bunch of the other pics I got in this set work fine but I suspect they don't have overviews. I thought it might be the / E:\ vs. E:/ but flipping the slashes doesn't help. So not sure why these two behave slightly differently but possibly a clue.

[robe]

In case it wasn't clear, the erro above is from raster2pgsql (not from SQL). I also tried copying the file to C:/pics to rule out the path difference and get the same error.

[dustymugs]

Wow. The problem gets weirder. My guess is that something is wrong with GDAL due to that libjpeg error, specifically the inability to create a temp file.

From what I can tell, that picture has no overviews (so says gdalinfo on back.jpg).

[dustymugs]

Just tested on OSX 10.6 and no problems…

PostgreSQL 9.1.3 on x86_64-apple-darwin10.8.0, compiled by i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3), 64-bit POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3765" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.7.3" LIBJSON="UNKNOWN" RASTER

Will do a final check on my 32-bit linux box…

[robe]

Really? What does your gdalinfo show? As shown above I assumed:

Overviews: 3000x2250, 1500x1125, 750x563

Means its got overviews

[dustymugs]

Nope. I have no overviews listed when I uses gdalinfo from 1.9.2

Driver: JPEG/JPEG JFIF
Files: back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
  EXIF_ColorSpace=1
  EXIF_DateTime=2012:11:28 15:18:54
  EXIF_Orientation=1
  EXIF_PixelXDimension=6000
  EXIF_PixelYDimension=4500
  EXIF_ResolutionUnit=2
  EXIF_Software=Adobe Photoshop CS3 Windows
  EXIF_XResolution=(150)
  EXIF_YResolution=(150)
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 4500.0)
Upper Right ( 6000.0,    0.0)
Lower Right ( 6000.0, 4500.0)
Center      ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
  Image Structure Metadata:
    COMPRESSION=JPEG

[robe]

Okay I've been cheating a bit. I was using the gdalinfo from above that came from Temas build: ​http://www.gisinternals.com/sdk/

gdalinfo --version
GDAL 1.10dev, released 2011/12/29

Using the gdalinfo that gets build with my mingw gdal

GDAL 1.9.2, released 2012/10/08

gdalinfo c:/pic/back.jpg
Driver: JPEG/JPEG JFIF
Files: c:/pics/back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
  EXIF_ColorSpace=1
  EXIF_DateTime=2012:11:28 15:18:54
  EXIF_Orientation=1
  EXIF_PixelXDimension=6000
  EXIF_PixelYDimension=4500
  EXIF_ResolutionUnit=2
  EXIF_Software=Adobe Photoshop CS3 Windows
  EXIF_XResolution=(150)
  EXIF_YResolution=(150)
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (    0.0,    0.0)
Lower Left  (    0.0, 4500.0)
Upper Right ( 6000.0,    0.0)
Lower Right ( 6000.0, 4500.0)
Center      ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
  Image Structure Metadata:
    COMPRESSION=JPEG

so my plain vanilla older version compile is not showing any overviews. But I think there are. Perhaps I have a gdla installed somewhere on my production server and it's picking up the libjpeg from somewhere and that is a separate issue from the can't read band (which both computers exhibit)

[dustymugs]

Huh. Well, you're right about the overviews being found with gdal 1.10dev. After upgrading my main dev box to gdal 1.10dev and recompiling postgis, everything still works…

[dustymugs]

No problems found with that file using GDAL 1.10dev and latest PostGIS in 32 and 64 bit Linux and OSX. I think something may be wrong with your windows env. Multiple versions of the same library? Multiple GDAL installed? The error with creating a temp file usually is a sign that either there isn't enough storage space or there is some privilege issue going on (lack of privileges or trying to create the temp file in the wrong place).

[robe]

Is there an easy way to tell where it's creating the temp file? Does it only need to create temp file when it does a VRT thing? That could explain the difference between the windows 2008 and my local windows 7 with being able to load/not load as indb. The windows 2008 is very locked down on permissions though it loads the other jpeg files just fine.

[robe]

If you don't know off hand, I'll just run procmon to see where its failing.

[dustymugs]

What's the size difference between the jpeg in question versus the others? Is this jpeg bigger?

You'll probably need to run procmon to see exactly what is going on…

[robe]

procmon doesn't seem to be telling me much. Maybe cause I haven't used it in a while.

Most of the files I have are smaller — around 4MB or less and were all created on the same day from another source. The ones I've having issue with are all created earlier and are between 10 and 30 MB in size and looks like they come from different source. I suspect all these older ones have overviews and the newer don't, but I'm double-checking that.

[dustymugs]

Very strange. I just set up a Windows 7 64-bit box with the latest PostGIS experimental binary for PG 9.2 and I have the same error. No crash, just that error.

Could you run a debug build of PostGIS, capture the output of the ST_Resize query and send me that dump?

[robe]

Well this is interesting. My ming64 postgresql build fails with ST_Resize similar to the VC one, but also fails with the ST_SummaryStats (where as the VC one first runs and second crashes the server instance). I'm about to run a debug version. Any particular debug level I should use?

[dustymugs]

Debug level 4 please! It's going to be a massive dump but zips really tiny.

[robe]

I guess I should run against 9.1 as well to rule out PG 9.2 as culprit. You don't have a 9.2 instance on your Mac or Linux do you?

[dustymugs]

OSX was on 9.1. Linux (both versions) were on 9.2.

[robe]

Notices from ST_SummaryStats:

NOTICE:  [rt_api.c:rt_raster_deserialize:8122] rt_raster_deserialize: Entering...
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8130] rt_raster_deserialize: Allocating memory for deserialized raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8138] rt_raster_deserialize: Deserialize raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8149] rt_raster_deserialize: Allocating memory for bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8157] rt_raster_deserialize: 3 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 0 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 1 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1

NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 2 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3126] starting
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3161] nodata = 0.000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3162] hasnodata = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3163] exclude_nodata_value = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3207] do_sample = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3229] sampling 27000000 of 27000000 available pixels w/ 4500 per set
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000048E7E18
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5871] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5875] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12569] rast1(ipX, ipxY) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12570] rast2(xr, yr) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12571] rast2(xw, yw) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_destroy:5217] Destroying rt_raster @ 00000000048E7E18
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1659] offsets: (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9118] Raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9121] Creating new raster
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000048E7E18
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9127] Created raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9141] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9174] GDAL Band 1 stats: 0.000000, 255.000000, 74.122972, 60.375771
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9180] Processing band 1 of 1
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9202] GDAL band dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9212] (hasnodata, nodataval) = (0, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 00000000048E7EB0 with pixtype 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5507] Adding band 00000000048E7EB0 to raster 00000000048E7E18
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5523] Oldbands at 0000000000000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5529] Checking bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5538] realloc returned 00000000048E7F48
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5555] Raster now has 1 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9226] Created band of dimension (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9232] (nXBlockSize, nYBlockSize) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9233] (nXBlocks, nYBlocks) = (47, 36)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9270] values len = 16384
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9276] (iXBlock, iYBlock) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9277] (x, y) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9293] (nXValid, nYValid) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1


ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  SQL function "st_summarystats" statement 1

********** Error **********

ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
SQL state: XX000
Context: SQL function "st_summarystats" statement 1

[dustymugs]

Ah. I think I may be onto something…

NOTICE:  [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
NOTICE:  [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)

That -1 vs 1 Y-scale will cause issues. I'm going to have to chew on this since it will involve poking at raster2pgsql (at it specifies the generic geotransform).

Just for kicks, can you up date the Y-scale of the raster and then rerun ST_SummaryStats()?

UPDATE armory_outdb SET rast = ST_SetScale(rast, 1, 1)

[robe]

After running:

UPDATE armory_outdb SET rast = ST_SetScale(rast, 1, 1);
SELECT ST_SummaryStats(rast) from armory_outdb;

I get:

NOTICE:  [rt_api.c:rt_raster_deserialize:8122] rt_raster_deserialize: Entering...
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8130] rt_raster_deserialize: Allocating memory for deserialized raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8138] rt_raster_deserialize: Deserialize raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8149] rt_raster_deserialize: Allocating memory for bands
CONTEXT:  SQL function "st_summarystats" statement 1

NOTICE:  [rt_api.c:rt_raster_deserialize:8157] rt_raster_deserialize: 3 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 0 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 1 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 2 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3126] starting
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3161] nodata = 0.000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3162] hasnodata = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3163] exclude_nodata_value = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3207] do_sample = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3229] sampling 27000000 of 27000000 available pixels w/ 4500 per set
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000049759D8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5871] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5875] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12569] rast1(ipX, ipxY) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12570] rast2(xr, yr) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12571] rast2(xw, yw) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_destroy:5217] Destroying rt_raster @ 00000000049759D8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1659] offsets: (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9118] Raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9121] Creating new raster
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000049759D8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9127] Created raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9141] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9174] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9180] Processing band 1 of 1
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9202] GDAL band dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9212] (hasnodata, nodataval) = (0, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 0000000004975A70 with pixtype 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5507] Adding band 0000000004975A70 to raster 00000000049759D8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5523] Oldbands at 0000000000000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5529] Checking bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5538] realloc returned 0000000004975B08
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5555] Raster now has 1 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9226] Created band of dimension (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9232] (nXBlockSize, nYBlockSize) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9233] (nXBlocks, nYBlocks) = (47, 36)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9270] values len = 16384
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9276] (iXBlock, iYBlock) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9277] (x, y) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9293] (nXValid, nYValid) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1

[robe]

oh and it still ends in error

ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  SQL function "st_summarystats" statement 1

[dustymugs]

Can you try r11098?

[robe]

I'm having trouble compiling with —enable-debug=4. Gives error:

rt_api.c: In function 'rt_band_load_offline_data':
rt_api.c:1635:3: error: expected expression before ')' token
make[2]: *** [rt_api.o] Error 1

Though seems to be compiling if I take debug configure out. I'll check the output after its' done. Can you see if you get errors compiling with debug flags on.

[robe]

Nope still doesn't work, though my mingw now seems to give values for first summary stats call and then erros if I call summary stats again.

I'm going to try on my 32-bit to see if it has the same issue.

[dustymugs]

Oops. Fixed the debug error in r11102. I'm going to have additional checks.

[robe]

For the record have same error on my 9.2.1 edb 32-bit instance with r11102. Will try with debugging next.

[robe]

an in db works fine on 32-bit (just like my local 64-bit):

SELECT version() || ' ' || postgis_full_version();
--PostgreSQL 9.2.1, compiled by Visual C++ build 1600, 32-bit POSTGIS="2.1.0SVN r11102" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER

SELECT (st_summarystats(rast)).* from armory_local;

  count   |    sum     |       mean       |      stddev      | min | max
----------+------------+------------------+------------------+-----+-----
 27000000 | 2665449805 | 98.7203631481482 | 81.4870806656832 |   0 | 255


SELECT ST_Resize(rast,0.1,0.1) from armory_local;

-- I can see a picture :) --

[robe]

Outputs from:

PostgreSQL 9.2.2 on x86_64-w64-mingw32, compiled by x86_64-w64-mingw32-gcc.exe (GCC) 4.5.4 20111030 (prerelease) [svn/rev.180676 - mingw-w64/oz], 64-bit POSTGIS="2.1.0SVN r11102" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER

SELECT ST_SummaryStats(rast) from armory_outdb;

NOTICE:  [rt_api.c:rt_raster_deserialize:8131] rt_raster_deserialize: Entering...
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8139] rt_raster_deserialize: Allocating memory for deserialized raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8147] rt_raster_deserialize: Deserialize raster header
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8158] rt_raster_deserialize: Allocating memory for bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8166] rt_raster_deserialize: 3 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 0 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 1 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 2 with pixel type 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3135] starting
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3170] nodata = 0.000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3171] hasnodata = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3172] exclude_nodata_value = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3216] do_sample = 0
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_get_summary_stats:3238] sampling 27000000 of 27000000 available pixels w/ 4500 per set
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1635] Using default geotransform matrix (0, 1, 0, 0, 0, -1)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1644] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5197] Created rt_raster @ 000000000490FDD8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5934] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5949] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5880] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_cell_to_geopoint:5884] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12578] rast1(ipX, ipxY) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12579] rast2(xr, yr) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_same_alignment:12580] rast2(xw, yw) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_destroy:5226] Destroying rt_raster @ 000000000490FDD8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5934] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_geopoint_to_cell:5949] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_load_offline_data:1668] offsets: (-0.000000, -0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9127] Raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9130] Creating new raster
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_new:5197] Created rt_raster @ 000000000490FDD8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9136] Created raster dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9150] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9183] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9189] Processing band 1 of 1
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9211] GDAL band dimensions (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9221] (hasnodata, nodataval) = (0, 0.000000)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 000000000490FE70 with pixtype 8BUI
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5516] Adding band 000000000490FE70 to raster 000000000490FDD8
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5532] Oldbands at 0000000000000000
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5538] Checking bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5547] realloc returned 000000000490FF08
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_add_band:5564] Raster now has 1 bands
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9235] Created band of dimension (width x height): 6000 x 4500
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9241] (nXBlockSize, nYBlockSize) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9242] (nXBlocks, nYBlocks) = (47, 36)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9279] values len = 16384
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9285] (iXBlock, iYBlock) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9286] (x, y) = (0, 0)
CONTEXT:  SQL function "st_summarystats" statement 1
NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9302] (nXValid, nYValid) = (128, 128)
CONTEXT:  SQL function "st_summarystats" statement 1
ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  SQL function "st_summarystats" statement 1

********** Error **********

ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
SQL state: XX000
Context: SQL function "st_summarystats" statement 1

Let me know if you need the local. the local with debug enabled takes forever to load.

[dustymugs]

There is definitely something weird going on in Windows.

Can you rerun ST_Summarystats() after making sure that the file back.jpg.aux.xml (should live with back.jpg) is removed and/or not present?

What I am looking for in the debug output is something like

NOTICE:  [rt_api.c:rt_raster_from_gdal_dataset:9183] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081

That line is the output from GDALComputeRasterStatistics() upon the band data. It will return the cached stats from *.aux.xml if it is avaiable, hence why I ask that you delete that file if it exists. By not having cached stats, GDAL is forced to go use the raw data to compute the stats. I'm eliminating as many possible failure points as I can…

[dustymugs]

In diffing the debug output you've provided to what I'm getting, everything (except memory addresses) matches up… gah!

[dustymugs]

Can you also set the following environment variable, restart PostgreSQL and then run ST_SummaryStats()?

CPL_DEBUG=ON

GDAL debug output will probably end up in PostgreSQL's log.

[robe]

I see this in the logs when I have that ON

GDAL: GDALOpen(C:/pics/back.jpg, this=000000000499FAD0) succeeds as JPEG.
VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MINIMUM,0,)

VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MAXIMUM,255,)

VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MEAN,98.720363148148,)

VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_STDDEV,81.487080665688,)

ERROR 1: libjpeg: Failed to create temporary file 
ERROR 1: C:/pics/back.jpg, band 1: IReadBlock failed at X offset 0, Y offset 0
ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
ERROR:  rt_raster_from_gdal_dataset: Unable to get data from GDAL raster
CONTEXT:  SQL function "st_summarystats" statement 1
STATEMENT:  SELECT ST_SummaryStats(rast) from armory_outdb

[dustymugs]

Again that "libjpeg: Failed to create temporary file". I wonder if the following might apply…

​http://lists.osgeo.org/pipermail/gdal-dev/2007-September/013996.html

​http://trac.osgeo.org/gdal/ticket/1795

What is the provenance of the GDAL library in the Windows builds? Is the windows library using the internal libjpeg or an external one?

[robe]

dustymugs,

Windows uses the internal libjpeg.

I'm pretty sure you are right about permissions now. That I think solves the mystery about why the raster2pgsql loader (when loading in db) works fine on this image with my windows 7 but fails on my windows 2008. I remembered that my windows 7 doesn't have all that UAC crap enabled on it so when I run raster2pgsql on my windows 7, I'm running under my full administrative powers. On the windows 2008, even though I was logged in as administrator, raster2pgsql failed to work to load the image in the db — HOWEVER, if I right-click on the batch script I have and choose — "Run as administrator" it works fine.

Now why the postgres outdb doesn't work I assume is probably a similar issue, but I would have expected it to work on my windows 7. When I run in windows 7 for development, I'm not running with an installed postgresql instance, but one I launch with a batch script that runs under my local admin account. I don't think I even have a postgres account on my window 7 or postgresql installed at all. So in theory it should have full rights to do damage to my whole system. So maybe there are other measures embedded in postgres that prevent from writting to the root drive.

[dustymugs]

This rabbit hole goes deep. I'm testing with Windows 7 64-bit with PostgreSQL 9.2.3 from EDB and the latest PostGIS build (r11103). ST_SummaryStats() works fine no matter how many times I run it.

With CPL_DEBUG=ON, I get the libjpeg error message for ST_Resize(). Since ALL out-db rasters are processed the exact same way (one function to rule them all), the error is happening on the GDAL side in the actual operation.

Turning off UAC doesn't eliminate the error.

[dustymugs]

Woohoo! What does fix the problem is the account being used for the PostgreSQL service. By default, the Network Service account is used. I changed that to use the Local System account. And boom, everything works fine…

This is most definitely a Windows account privilege issue…

[robe]

I'm still puzzled why my windows 7 admin account that launches a development postgres process isn't immune to this (even when I use my very own mingw compiled postgres). Anyrate I think we should just

1) Switch this to a documentation issue and just put it in the FAQ as a none windows permission issue

2) close this out and reopen a clean ticket (assuming you think redoing without VRT is better or even possible)

[dustymugs]

Before we change this ticket, I'd like to make sure that things work for you.

I may remove the out-db loading with VRT but that won't help resolve the problem defined in this ticket.

[robe]

BTW: I updated my ming64 zip. Hopefully its not as much of a pain to use as when Steve tried it.

​http://www.bostongis.com/postgisstuff/ming64.zip

[dustymugs]

No… still a pain I'll bug you over email…

[dustymugs]

I took a look using ProcMon and it looks like it is trying to create a temporary file at C:\.

[dustymugs]

Confirmed. libjpeg tries to create a temporary file at C:\. I gave my account full control of C:\ only and everything works.

[robe]

I switched to GDAL issue. Is there anyway to test this issue on Linux or it just doesn't happen because GDAL writes to a place accessible by all. I would think like SELinux and Ubuntu would be pretty locked down on rights. Though I guess this issue would never happen unless you happen to have a jpeg that has overviews? Which may be rare or does it have to do with the size of the ffile that it then switches to using disk?

[dustymugs]

From what I know, it just doesn't happen as GDAL writes to /tmp which is accessible by everyone and there really aren't other default temporary paths.

I actually don't know if it has anything to do with overviews but might have something to do with file size. Actually, I have no idea why libjpeg is involved as GDAL Warp API is passed a MEM dataset. So much voodoo.

[robe]

I don't think this is an issue anymore.