Opened 11 years ago
Closed 4 years ago
#2203 closed defect (wontfix)
[raster]: Can't read out_db jpg in some cases
| Reported by: | robe | Owned by: | Bborie Park |
|---|---|---|---|
| Priority: | medium | Milestone: | PostGIS GDAL |
| Component: | raster | Version: | master |
| Keywords: | history | Cc: |
Description
This might be similar issue to what we've had before, but I get a set of project photos, but unfortunately all the ones exhibiting the problem are too big to attach.
When I do this or ST_AsPNG
SELECT ST_Resize(rast,0.1,0.1) FROM armory_outdb limit 1
I get this error
NOTICE: The in-db representation of the out-db raster is not aligned. Band data may be incorrect CONTEXT: PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN
This issue doesn't happen if I load in db.
Command I used is:
raster2pgsql -F -Y -R C:/pics/back.jpg armory_outdb | psql -U postgres -d testpostgis21 -h localhost -p 5442
POSTGIS="2.1.0SVN r11085" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" (core procs from "2.1.0SVN r11079" need upgrade) RASTER (raster procs from "2.1.0SVN r11079" need upgrade)
I'll send you link in a bit.
Change History (57)
comment:1 by , 11 years ago
comment:2 by , 11 years ago
gdalinfo
Driver: JPEG/JPEG JFIF
Files: back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
EXIF_ColorSpace=1
EXIF_DateTime=2012:11:28 15:18:54
EXIF_Orientation=1
EXIF_PixelXDimension=6000
EXIF_PixelYDimension=4500
EXIF_ResolutionUnit=2
EXIF_Software=Adobe Photoshop CS3 Windows
EXIF_XResolution=(150)
EXIF_YResolution=(150)
Image Structure Metadata:
COMPRESSION=JPEG
INTERLEAVE=PIXEL
SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4500.0)
Upper Right ( 6000.0, 0.0)
Lower Right ( 6000.0, 4500.0)
Center ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
Overviews: 3000x2250, 1500x1125, 750x563
Image Structure Metadata:
COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
Overviews: 3000x2250, 1500x1125, 750x563
Image Structure Metadata:
COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
Overviews: 3000x2250, 1500x1125, 750x563
Image Structure Metadata:
COMPRESSION=JPEG
I think on my server I was getting a can't create VRT or something (I presume because it has overviews?) (as I recall it wouldn't even load some of the pics using in db, though perhaps this particular one was not one I tested on prod server)
SELECT (ST_MetaData(rast)).* from armory_outdb;
outputs
upperleftx | upperlefty | width | height | scalex | scaley | skewx | skewy | srid| numbands
------------+------------+-------+--------+--------+--------+-------+-------+---
---+----------
0 | 0 | 6000 | 4500 | 1 | -1 | 0 | 0 |
0 | 3
comment:3 by , 11 years ago
| Keywords: | history added |
|---|---|
| Resolution: | → fixed |
| Status: | new → closed |
Fixed in r11094. Decided to change how handling of rasters with no spatial information is done.
comment:4 by , 11 years ago
| Resolution: | fixed |
|---|---|
| Status: | closed → reopened |
Still doesn't seem to be working for me for this example though I'm not getting the NOTICE anymore.
Just this:
ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: PL/pgSQL function st_resize(raster,double precision,double precision,text,double precision) line 23 at RETURN
Running latest winnie build —
select version() || ' ' || postgis_full_version();
PostgreSQL 9.2.2, compiled by Visual C++ build 1600, 64-bit POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER
comment:5 by , 11 years ago
The NOTICE is a flag for metadata differences between what GDAL sees in the raster file and what is inside the database. This message should never happen normally.
The error though is due to an error with GDALRasterIO() unable to read the data. I'll need to test with 1.9.2.
comment:6 by , 11 years ago
Are you able to run something even simpler than ST_Resize()? Something like…
SELECT ST_SummaryStats(rast) FROM armory_outdb limit 1
All out-db rasters are handled in the exact same manner so the above should result in the same error.
comment:7 by , 11 years ago
I can't replicate the problem using the JPEG you provided…
PostgreSQL 9.2.1 on x86_64-unknown-linux-gnu, compiled by gcc (GCC) 4.5.2, 64-b it POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3755" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNK NOWN" RASTER
comment:8 by , 11 years ago
Hmm that's odd. I thought maybe it was a permssion problem but I think I have it in same folder with where I put other images. I'll retest again on the original server I was having the issue and close this out if its a non-issue there.
does your change effect raster2pgsql. Its possible I didn't reload?
comment:9 by , 11 years ago
Okay I just reloaded — here is what I found:
SELECT ST_SummaryStats(rast) FROM armory_outdb limit 1; (27000000,2665449805,98.7203631481482,81.4870806656832,0,255)
Run the above a second time KABOOM server instance crashes and comes toppling down 
Resize still doesn't work and to rule out the folder, I added another image to it and registered that file as an out db and that one works fine.
If it helps when I run resize I get this extra info in the database logs:
ERROR 1: libjpeg: Failed to create temporary file ERROR 1: C:/pics/back.jpg, band 2: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0
I'm going to try next on my mingw instance to see if its an EDB windows instance issue. haven't tried on my 32-bit instance yet either. I have 5GB of disk space left so I assume that is not the issue. I think I'm running with default postgres.conf for my development instance.
comment:10 by , 11 years ago
Before I do the mingw test, I just tried on the production server having the original issue which is a Windows 2008 R2 64-bit running:
PostgreSQL 9.2.3, compiled by Visual C++ build 1600, 64-bit PostGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3755" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNK NOWN" RASTER
I can't even get the in db to work on this one with this image. This is the same notice I was getting before on this one before the upgrade too: Sorry can't cut and paste fromt his server so bare with me
ERROR 1: libjpeg: Failed to create temporary file ERROR 1: E:\gisdata\pics\armory\back.jpg, band 2: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ERROR: could not convert VRT dataset to PostGIS raster ERROR: process_rasters: Could not process raster: E:\gisdata\pics\armory\back.jpg
A bunch of the other pics I got in this set work fine but I suspect they don't have overviews. I thought it might be the / E:\ vs. E:/ but flipping the slashes doesn't help. So not sure why these two behave slightly differently but possibly a clue.
comment:11 by , 11 years ago
In case it wasn't clear, the erro above is from raster2pgsql (not from SQL). I also tried copying the file to C:/pics to rule out the path difference and get the same error.
comment:12 by , 11 years ago
Wow. The problem gets weirder. My guess is that something is wrong with GDAL due to that libjpeg error, specifically the inability to create a temp file.
From what I can tell, that picture has no overviews (so says gdalinfo on back.jpg).
comment:13 by , 11 years ago
Just tested on OSX 10.6 and no problems…
PostgreSQL 9.1.3 on x86_64-apple-darwin10.8.0, compiled by i686-apple-darwin10-gcc-4.2.1 (GCC) 4.2.1 (Apple Inc. build 5666) (dot 3), 64-bit POSTGIS="2.1.0SVN r11095" GEOS="3.4.0dev-CAPI-1.8.0 r3765" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.10dev, released 2011/12/29" LIBXML="2.7.3" LIBJSON="UNKNOWN" RASTER
Will do a final check on my 32-bit linux box…
comment:14 by , 11 years ago
Really? What does your gdalinfo show? As shown above I assumed:
Overviews: 3000x2250, 1500x1125, 750x563
Means its got overviews
comment:15 by , 11 years ago
Nope. I have no overviews listed when I uses gdalinfo from 1.9.2
Driver: JPEG/JPEG JFIF
Files: back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
EXIF_ColorSpace=1
EXIF_DateTime=2012:11:28 15:18:54
EXIF_Orientation=1
EXIF_PixelXDimension=6000
EXIF_PixelYDimension=4500
EXIF_ResolutionUnit=2
EXIF_Software=Adobe Photoshop CS3 Windows
EXIF_XResolution=(150)
EXIF_YResolution=(150)
Image Structure Metadata:
COMPRESSION=JPEG
INTERLEAVE=PIXEL
SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4500.0)
Upper Right ( 6000.0, 0.0)
Lower Right ( 6000.0, 4500.0)
Center ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
Image Structure Metadata:
COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
Image Structure Metadata:
COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
Image Structure Metadata:
COMPRESSION=JPEG
comment:16 by , 11 years ago
Okay I've been cheating a bit. I was using the gdalinfo from above that came from Temas build: http://www.gisinternals.com/sdk/
gdalinfo --version GDAL 1.10dev, released 2011/12/29
Using the gdalinfo that gets build with my mingw gdal
GDAL 1.9.2, released 2012/10/08
gdalinfo c:/pic/back.jpg
Driver: JPEG/JPEG JFIF
Files: c:/pics/back.jpg
Size is 6000, 4500
Coordinate System is `'
Metadata:
EXIF_ColorSpace=1
EXIF_DateTime=2012:11:28 15:18:54
EXIF_Orientation=1
EXIF_PixelXDimension=6000
EXIF_PixelYDimension=4500
EXIF_ResolutionUnit=2
EXIF_Software=Adobe Photoshop CS3 Windows
EXIF_XResolution=(150)
EXIF_YResolution=(150)
Image Structure Metadata:
COMPRESSION=JPEG
INTERLEAVE=PIXEL
SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4500.0)
Upper Right ( 6000.0, 0.0)
Lower Right ( 6000.0, 4500.0)
Center ( 3000.0, 2250.0)
Band 1 Block=6000x1 Type=Byte, ColorInterp=Red
Image Structure Metadata:
COMPRESSION=JPEG
Band 2 Block=6000x1 Type=Byte, ColorInterp=Green
Image Structure Metadata:
COMPRESSION=JPEG
Band 3 Block=6000x1 Type=Byte, ColorInterp=Blue
Image Structure Metadata:
COMPRESSION=JPEG
so my plain vanilla older version compile is not showing any overviews. But I think there are. Perhaps I have a gdla installed somewhere on my production server and it's picking up the libjpeg from somewhere and that is a separate issue from the can't read band (which both computers exhibit)
comment:17 by , 11 years ago
Huh. Well, you're right about the overviews being found with gdal 1.10dev. After upgrading my main dev box to gdal 1.10dev and recompiling postgis, everything still works…
comment:18 by , 11 years ago
No problems found with that file using GDAL 1.10dev and latest PostGIS in 32 and 64 bit Linux and OSX. I think something may be wrong with your windows env. Multiple versions of the same library? Multiple GDAL installed? The error with creating a temp file usually is a sign that either there isn't enough storage space or there is some privilege issue going on (lack of privileges or trying to create the temp file in the wrong place).
comment:19 by , 11 years ago
Is there an easy way to tell where it's creating the temp file? Does it only need to create temp file when it does a VRT thing? That could explain the difference between the windows 2008 and my local windows 7 with being able to load/not load as indb. The windows 2008 is very locked down on permissions though it loads the other jpeg files just fine.
comment:20 by , 11 years ago
If you don't know off hand, I'll just run procmon to see where its failing.
comment:21 by , 11 years ago
What's the size difference between the jpeg in question versus the others? Is this jpeg bigger?
You'll probably need to run procmon to see exactly what is going on…
comment:22 by , 11 years ago
procmon doesn't seem to be telling me much. Maybe cause I haven't used it in a while.
Most of the files I have are smaller — around 4MB or less and were all created on the same day from another source. The ones I've having issue with are all created earlier and are between 10 and 30 MB in size and looks like they come from different source. I suspect all these older ones have overviews and the newer don't, but I'm double-checking that.
comment:23 by , 11 years ago
| Status: | reopened → new |
|---|
Very strange. I just set up a Windows 7 64-bit box with the latest PostGIS experimental binary for PG 9.2 and I have the same error. No crash, just that error.
Could you run a debug build of PostGIS, capture the output of the ST_Resize query and send me that dump?
comment:24 by , 11 years ago
| Status: | new → assigned |
|---|
comment:25 by , 11 years ago
Well this is interesting. My ming64 postgresql build fails with ST_Resize similar to the VC one, but also fails with the ST_SummaryStats (where as the VC one first runs and second crashes the server instance). I'm about to run a debug version. Any particular debug level I should use?
comment:26 by , 11 years ago
Debug level 4 please! It's going to be a massive dump but zips really tiny.
comment:27 by , 11 years ago
I guess I should run against 9.1 as well to rule out PG 9.2 as culprit. You don't have a 9.2 instance on your Mac or Linux do you?
comment:29 by , 11 years ago
Notices from ST_SummaryStats:
NOTICE: [rt_api.c:rt_raster_deserialize:8122] rt_raster_deserialize: Entering... CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8130] rt_raster_deserialize: Allocating memory for deserialized raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8138] rt_raster_deserialize: Deserialize raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8149] rt_raster_deserialize: Allocating memory for bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8157] rt_raster_deserialize: 3 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 0 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 1 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 2 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3126] starting CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3161] nodata = 0.000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3162] hasnodata = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3163] exclude_nodata_value = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3207] do_sample = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3229] sampling 27000000 of 27000000 available pixels w/ 4500 per set CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000048E7E18 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5871] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5875] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12569] rast1(ipX, ipxY) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12570] rast2(xr, yr) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12571] rast2(xw, yw) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_destroy:5217] Destroying rt_raster @ 00000000048E7E18 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1659] offsets: (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9118] Raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9121] Creating new raster CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000048E7E18 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9127] Created raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9141] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9174] GDAL Band 1 stats: 0.000000, 255.000000, 74.122972, 60.375771 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9180] Processing band 1 of 1 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9202] GDAL band dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9212] (hasnodata, nodataval) = (0, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 00000000048E7EB0 with pixtype 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5507] Adding band 00000000048E7EB0 to raster 00000000048E7E18 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5523] Oldbands at 0000000000000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5529] Checking bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5538] realloc returned 00000000048E7F48 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5555] Raster now has 1 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9226] Created band of dimension (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9232] (nXBlockSize, nYBlockSize) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9233] (nXBlocks, nYBlocks) = (47, 36) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9270] values len = 16384 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9276] (iXBlock, iYBlock) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9277] (x, y) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9293] (nXValid, nYValid) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1 ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: SQL function "st_summarystats" statement 1 ********** Error ********** ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster SQL state: XX000 Context: SQL function "st_summarystats" statement 1
comment:30 by , 11 years ago
Ah. I think I may be onto something…
NOTICE: [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) NOTICE: [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000)
That -1 vs 1 Y-scale will cause issues. I'm going to have to chew on this since it will involve poking at raster2pgsql (at it specifies the generic geotransform).
Just for kicks, can you up date the Y-scale of the raster and then rerun ST_SummaryStats()?
UPDATE armory_outdb SET rast = ST_SetScale(rast, 1, 1)
comment:31 by , 11 years ago
After running:
UPDATE armory_outdb SET rast = ST_SetScale(rast, 1, 1); SELECT ST_SummaryStats(rast) from armory_outdb;
I get:
NOTICE: [rt_api.c:rt_raster_deserialize:8122] rt_raster_deserialize: Entering... CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8130] rt_raster_deserialize: Allocating memory for deserialized raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8138] rt_raster_deserialize: Deserialize raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8149] rt_raster_deserialize: Allocating memory for bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8157] rt_raster_deserialize: 3 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 0 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 1 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8183] rt_raster_deserialize: band 2 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8251] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8252] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8295] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3126] starting CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3161] nodata = 0.000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3162] hasnodata = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3163] exclude_nodata_value = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3207] do_sample = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3229] sampling 27000000 of 27000000 available pixels w/ 4500 per set CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1636] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000049759D8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5871] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5875] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12569] rast1(ipX, ipxY) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12570] rast2(xr, yr) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12571] rast2(xw, yw) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_destroy:5217] Destroying rt_raster @ 00000000049759D8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5925] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5940] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1659] offsets: (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9118] Raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9121] Creating new raster CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5188] Created rt_raster @ 00000000049759D8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9127] Created raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9141] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, 1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9174] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9180] Processing band 1 of 1 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9202] GDAL band dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9212] (hasnodata, nodataval) = (0, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 0000000004975A70 with pixtype 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5507] Adding band 0000000004975A70 to raster 00000000049759D8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5523] Oldbands at 0000000000000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5529] Checking bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5538] realloc returned 0000000004975B08 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5555] Raster now has 1 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9226] Created band of dimension (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9232] (nXBlockSize, nYBlockSize) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9233] (nXBlocks, nYBlocks) = (47, 36) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9270] values len = 16384 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9276] (iXBlock, iYBlock) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9277] (x, y) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9293] (nXValid, nYValid) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1
comment:32 by , 11 years ago
oh and it still ends in error
ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: SQL function "st_summarystats" statement 1
comment:34 by , 11 years ago
I'm having trouble compiling with —enable-debug=4. Gives error:
rt_api.c: In function 'rt_band_load_offline_data': rt_api.c:1635:3: error: expected expression before ')' token make[2]: *** [rt_api.o] Error 1
Though seems to be compiling if I take debug configure out. I'll check the output after its' done. Can you see if you get errors compiling with debug flags on.
comment:35 by , 11 years ago
Nope still doesn't work, though my mingw now seems to give values for first summary stats call and then erros if I call summary stats again.
I'm going to try on my 32-bit to see if it has the same issue.
comment:36 by , 11 years ago
Oops. Fixed the debug error in r11102. I'm going to have additional checks.
comment:37 by , 11 years ago
For the record have same error on my 9.2.1 edb 32-bit instance with r11102. Will try with debugging next.
comment:38 by , 11 years ago
an in db works fine on 32-bit (just like my local 64-bit):
SELECT version() || ' ' || postgis_full_version(); --PostgreSQL 9.2.1, compiled by Visual C++ build 1600, 32-bit POSTGIS="2.1.0SVN r11102" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER SELECT (st_summarystats(rast)).* from armory_local; count | sum | mean | stddev | min | max ----------+------------+------------------+------------------+-----+----- 27000000 | 2665449805 | 98.7203631481482 | 81.4870806656832 | 0 | 255 SELECT ST_Resize(rast,0.1,0.1) from armory_local; -- I can see a picture :) --
comment:39 by , 11 years ago
Outputs from:
PostgreSQL 9.2.2 on x86_64-w64-mingw32, compiled by x86_64-w64-mingw32-gcc.exe (GCC) 4.5.4 20111030 (prerelease) [svn/rev.180676 - mingw-w64/oz], 64-bit POSTGIS="2.1.0SVN r11102" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" RASTER
SELECT ST_SummaryStats(rast) from armory_outdb; NOTICE: [rt_api.c:rt_raster_deserialize:8131] rt_raster_deserialize: Entering... CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8139] rt_raster_deserialize: Allocating memory for deserialized raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8147] rt_raster_deserialize: Deserialize raster header CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8158] rt_raster_deserialize: Allocating memory for bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8166] rt_raster_deserialize: 3 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 0 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 1 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8192] rt_raster_deserialize: band 2 with pixel type 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8260] rt_raster_deserialize: has nodata flag 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8261] rt_raster_deserialize: nodata value 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_deserialize:8304] rt_raster_deserialize: skip 4 bytes of 8-bytes boundary padding CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3135] starting CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3170] nodata = 0.000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3171] hasnodata = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3172] exclude_nodata_value = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3216] do_sample = 0 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_get_summary_stats:3238] sampling 27000000 of 27000000 available pixels w/ 4500 per set CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1631] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1635] Using default geotransform matrix (0, 1, 0, 0, 0, -1) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1644] Offline geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5197] Created rt_raster @ 000000000490FDD8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5934] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5949] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5880] gt = (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_cell_to_geopoint:5884] GDALApplyGeoTransform (c -> g) for (-0.000000, -0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12578] rast1(ipX, ipxY) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12579] rast2(xr, yr) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_same_alignment:12580] rast2(xw, yw) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_destroy:5226] Destroying rt_raster @ 000000000490FDD8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5934] GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (0.000000, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_geopoint_to_cell:5949] Corrected GDALApplyGeoTransform (g -> c) for (0.000000, 0.000000) = (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_load_offline_data:1668] offsets: (-0.000000, -0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9127] Raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9130] Creating new raster CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_new:5197] Created rt_raster @ 000000000490FDD8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9136] Created raster dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9150] Raster geotransform (0.000000, 1.000000, 0.000000, 0.000000, 0.000000, -1.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9183] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9189] Processing band 1 of 1 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9211] GDAL band dimensions (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9221] (hasnodata, nodataval) = (0, 0.000000) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_pixtype_size:973] Pixel type = 8BUI and size = 1 bytes CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1328] Created rt_band @ 000000000490FE70 with pixtype 8BUI CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_band_new_inline:1341] Created rt_band with dimensions 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5516] Adding band 000000000490FE70 to raster 000000000490FDD8 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5532] Oldbands at 0000000000000000 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5538] Checking bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5547] realloc returned 000000000490FF08 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_add_band:5564] Raster now has 1 bands CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9235] Created band of dimension (width x height): 6000 x 4500 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9241] (nXBlockSize, nYBlockSize) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9242] (nXBlocks, nYBlocks) = (47, 36) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9279] values len = 16384 CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9285] (iXBlock, iYBlock) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9286] (x, y) = (0, 0) CONTEXT: SQL function "st_summarystats" statement 1 NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9302] (nXValid, nYValid) = (128, 128) CONTEXT: SQL function "st_summarystats" statement 1 ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: SQL function "st_summarystats" statement 1 ********** Error ********** ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster SQL state: XX000 Context: SQL function "st_summarystats" statement 1
Let me know if you need the local. the local with debug enabled takes forever to load.
comment:40 by , 11 years ago
There is definitely something weird going on in Windows.
Can you rerun ST_Summarystats() after making sure that the file back.jpg.aux.xml (should live with back.jpg) is removed and/or not present?
What I am looking for in the debug output is something like
NOTICE: [rt_api.c:rt_raster_from_gdal_dataset:9183] GDAL Band 1 stats: 0.000000, 255.000000, 98.720363, 81.487081
That line is the output from GDALComputeRasterStatistics() upon the band data. It will return the cached stats from *.aux.xml if it is avaiable, hence why I ask that you delete that file if it exists. By not having cached stats, GDAL is forced to go use the raw data to compute the stats. I'm eliminating as many possible failure points as I can…
comment:41 by , 11 years ago
In diffing the debug output you've provided to what I'm getting, everything (except memory addresses) matches up… gah!
comment:42 by , 11 years ago
Can you also set the following environment variable, restart PostgreSQL and then run ST_SummaryStats()?
CPL_DEBUG=ON
GDAL debug output will probably end up in PostgreSQL's log.
comment:43 by , 11 years ago
I see this in the logs when I have that ON
GDAL: GDALOpen(C:/pics/back.jpg, this=000000000499FAD0) succeeds as JPEG. VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MINIMUM,0,) VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MAXIMUM,255,) VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_MEAN,98.720363148148,) VRT: VRTSourcedRasterBand::SetMetadataItem(STATISTICS_STDDEV,81.487080665688,) ERROR 1: libjpeg: Failed to create temporary file ERROR 1: C:/pics/back.jpg, band 1: IReadBlock failed at X offset 0, Y offset 0 ERROR 1: GetBlockRef failed at X block offset 0, Y block offset 0 ERROR: rt_raster_from_gdal_dataset: Unable to get data from GDAL raster CONTEXT: SQL function "st_summarystats" statement 1 STATEMENT: SELECT ST_SummaryStats(rast) from armory_outdb
comment:44 by , 11 years ago
Again that "libjpeg: Failed to create temporary file". I wonder if the following might apply…
http://lists.osgeo.org/pipermail/gdal-dev/2007-September/013996.html
http://trac.osgeo.org/gdal/ticket/1795
What is the provenance of the GDAL library in the Windows builds? Is the windows library using the internal libjpeg or an external one?
comment:45 by , 11 years ago
dustymugs,
Windows uses the internal libjpeg.
I'm pretty sure you are right about permissions now. That I think solves the mystery about why the raster2pgsql loader (when loading in db) works fine on this image with my windows 7 but fails on my windows 2008. I remembered that my windows 7 doesn't have all that UAC crap enabled on it so when I run raster2pgsql on my windows 7, I'm running under my full administrative powers. On the windows 2008, even though I was logged in as administrator, raster2pgsql failed to work to load the image in the db — HOWEVER, if I right-click on the batch script I have and choose — "Run as administrator" it works fine.
Now why the postgres outdb doesn't work I assume is probably a similar issue, but I would have expected it to work on my windows 7. When I run in windows 7 for development, I'm not running with an installed postgresql instance, but one I launch with a batch script that runs under my local admin account. I don't think I even have a postgres account on my window 7 or postgresql installed at all. So in theory it should have full rights to do damage to my whole system. So maybe there are other measures embedded in postgres that prevent from writting to the root drive.
comment:46 by , 11 years ago
This rabbit hole goes deep. I'm testing with Windows 7 64-bit with PostgreSQL 9.2.3 from EDB and the latest PostGIS build (r11103). ST_SummaryStats() works fine no matter how many times I run it.
With CPL_DEBUG=ON, I get the libjpeg error message for ST_Resize(). Since ALL out-db rasters are processed the exact same way (one function to rule them all), the error is happening on the GDAL side in the actual operation.
Turning off UAC doesn't eliminate the error.
comment:47 by , 11 years ago
Woohoo! What does fix the problem is the account being used for the PostgreSQL service. By default, the Network Service account is used. I changed that to use the Local System account. And boom, everything works fine…
This is most definitely a Windows account privilege issue…
comment:48 by , 11 years ago
I'm still puzzled why my windows 7 admin account that launches a development postgres process isn't immune to this (even when I use my very own mingw compiled postgres). Anyrate I think we should just
1) Switch this to a documentation issue and just put it in the FAQ as a none windows permission issue
2) close this out and reopen a clean ticket (assuming you think redoing without VRT is better or even possible)
comment:49 by , 11 years ago
Before we change this ticket, I'd like to make sure that things work for you.
I may remove the out-db loading with VRT but that won't help resolve the problem defined in this ticket.
comment:50 by , 11 years ago
BTW: I updated my ming64 zip. Hopefully its not as much of a pain to use as when Steve tried it.
I'll bug you over email…
comment:52 by , 11 years ago
I took a look using ProcMon and it looks like it is trying to create a temporary file at C:\.
comment:53 by , 11 years ago
Confirmed. libjpeg tries to create a temporary file at C:\. I gave my account full control of C:\ only and everything works.
comment:54 by , 11 years ago
| Milestone: | PostGIS 2.1.0 → PostGIS GDAL |
|---|
I switched to GDAL issue. Is there anyway to test this issue on Linux or it just doesn't happen because GDAL writes to a place accessible by all. I would think like SELinux and Ubuntu would be pretty locked down on rights. Though I guess this issue would never happen unless you happen to have a jpeg that has overviews? Which may be rare or does it have to do with the size of the ffile that it then switches to using disk?
comment:55 by , 11 years ago
From what I know, it just doesn't happen as GDAL writes to /tmp which is accessible by everyone and there really aren't other default temporary paths.
I actually don't know if it has anything to do with overviews but might have something to do with file size. Actually, I have no idea why libjpeg is involved as GDAL Warp API is passed a MEM dataset. So much voodoo.
comment:56 by , 11 years ago
| Status: | assigned → new |
|---|
comment:57 by , 4 years ago
| Resolution: | → wontfix |
|---|---|
| Status: | new → closed |
I don't think this is an issue anymore.
Yep. I know that warning. Can you post the output of gdalinfo of the source raster file and ST_Metadata() of the out-db raster?