[raster]: raster2pgsql with overview generation crashes

[robe]

I haven't tried all permutations of this, but I wanted to do some speed tests for queries using my in db aerials vs. same aerials stored out db using the same tile settings. this is testing with

r11142

My command looks like this:

raster2pgsql -I -e -F -C -Y -R -s 26986 -t 500x500 -l 2 *.jpg aerials.boston_outdb | psql

I'll need to also retry this with indb since when I did the in db it was in PostGIS 2.0.

I'm also going to try without tiling. Anyway things sort of move smoothly except for this notice for each tile:

Processing 116/116: E:/GISDATA/bostonaerials2008/jpeg/25229020.sid.jpg

and its when it finally gets to the last tile that it crashes. Presumably when trying to apply constraints and so forth.

  Fault Module Version:	6.1.7601.17725
  Fault Module Timestamp:	4ec4aa8e
  Exception Code:	c0000374
  Exception Offset:	00000000000c40f2
  OS Version:	6.1.7601.2.1.0.274.10
  Locale ID:	1033
  Additional Information 1:	6005
  Additional Information 2:	6005ee1f61854d6369e1fbaa6883c156
  Additional Information 3:	9869
  Additional Information 4:	986908455240911563ee66b8d79d92ea

which looks like gibberish. when I close the crash window — it shows this notice in console:

ERROR: diff_rastinfo: Could not run raster alignment test
ERROR:  rt_raster_from_hexwkb: Raster HEXWKB input must have an even number of c
haracters
CONTEXT:  COPY o_2_boston_outdb, line 25, column rast: "01000003006E3433333333E3
3FD73433333333E3BFBBCA487D73D80E4136BCAF0C86832B4100000000000000000000000000..."

I'm running this on Windows 2008 64-bit using

PostgreSQL 9.2.3, compiled by Visual C++ build 1600, 64-bit POSTGIS="2.1.0SVN r11142" GEOS="3.4.0dev-CAPI-1.8.0 r0" PROJ="Rel. 4.8.0, 6 March 2012" GDAL="GDAL 1.9.2, released 2012/10/08" LIBXML="2.7.8" LIBJSON="UNKNOWN" TOPOLOGY (topology procs from "2.1.0SVN r11095" need upgrade) RASTER

[robe]

More observations — as the note suggests, it is when its building the overview. So this is crazy robe2 trying to create a virtual tiled with overviews. Not sure what exactly that would mean.

Anyway further information — comparing the rows in both sets of tables. My original indb and outdb base tables both have the same number of rows: 11600

However the overview table seems to have gotten as far as loading rows: 2875 before croaking. My in db overview table which presumably should be the same accept in db has 2900 rows. So it had 25 more to go before it kicked the bucket.

[robe]

Oops cut off the notice part that shows for each tile processed:

Processing 116/116: E:/GISDATA/bostonaerials2008/jpeg/25229020.sid.jpg
ERROR: diff_rastinfo: Could not run raster alignment test

[dustymugs]

Can you try running that file through gdalinfo? And load only that file with raster2pgsql

[robe]

Okay will do. BTW I have reworded. The issue is only when building with overviews. Without overview all loads up though I haven't tried querying it.

[robe]

Driver: JPEG/JPEG JFIF
Files: 25229020.sid.jpg
       25229020.sid.jpg.aux.xml
Size is 5000, 5000
Coordinate System is:
PROJCS["NAD83 / Massachusetts Mainland",
    GEOGCS["NAD83",
        DATUM["North_American_Datum_1983",
            SPHEROID["GRS 1980",6378137,298.2572221010002,
                AUTHORITY["EPSG","7019"]],
            AUTHORITY["EPSG","6269"]],
        PRIMEM["Greenwich",0],
        UNIT["degree",0.0174532925199433],
        AUTHORITY["EPSG","4269"]],
    PROJECTION["Lambert_Conformal_Conic_2SP"],
    PARAMETER["standard_parallel_1",42.68333333333333],
    PARAMETER["standard_parallel_2",41.71666666666667],
    PARAMETER["latitude_of_origin",41],
    PARAMETER["central_meridian",-71.5],
    PARAMETER["false_easting",200000],
    PARAMETER["false_northing",750000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AUTHORITY["EPSG","26986"]]
Origin = (251486.436173995750000,902771.024778253280000)
Pixel Size = (0.300000000000017,-0.300000000000023)
Metadata:
  GEOTIFF_CHAR__GTModelTypeGeoKey=ModelTypeProjected
  GEOTIFF_CHAR__GTRasterTypeGeoKey=RasterPixelIsArea
  GEOTIFF_CHAR__ProjectedCSTypeGeoKey=PCS_NAD83_Massachusetts
  GEOTIFF_CHAR__ProjLinearUnitsGeoKey=Linear_Meter
  GEOTIFF_NUM__1024__GTModelTypeGeoKey=1
  GEOTIFF_NUM__1025__GTRasterTypeGeoKey=1
  GEOTIFF_NUM__1026__GTCitationGeoKey=IMAGINE GeoTIFF Support
Copyright 1991 - 2005 by Leica Geosystems Geospatial Imaging, LLC. All Rights R
served
@(#)$RCSfile: egtf.c $ IMAGINE 9.0 $Revision: 10.0 $ $Date: 2005/07/26 15:10:00
EST $
Projection Name = State Plane
Units = meters
GeoTIFF Units = meters
  GEOTIFF_NUM__3072__ProjectedCSTypeGeoKey=26986
  GEOTIFF_NUM__3073__PCSCitationGeoKey=IMAGINE GeoTIFF Support
Copyright 1991 - 2005 by Leica Geosystems Geospatial Imaging, LLC. All Rights R
served
@(#)$RCSfile: egtf.c $ IMAGINE 9.0 $Revision: 10.0 $ $Date: 2005/07/26 15:10:00
EST $
State Plane Zone -2001
NAD = 83
  GEOTIFF_NUM__3076__ProjLinearUnitsGeoKey=9001
  IMAGE__BITS_PER_SAMPLE=8
  IMAGE__COLOR_SCHEME=0
  IMAGE__COMPRESSION_BLOCK_SIZE=512
  IMAGE__COMPRESSION_GAMMA=2.000000
  IMAGE__COMPRESSION_NLEV=7
  IMAGE__COMPRESSION_VERSION=2,0,0
  IMAGE__COMPRESSION_WEIGHT=2.000000
  IMAGE__CREATION_DATE=Tue Nov 18 10:44:29 2008

  IMAGE__DATA_TYPE=0
  IMAGE__DYNAMIC_RANGE_LEVEL=127.500000
  IMAGE__DYNAMIC_RANGE_WINDOW=256.000000
  IMAGE__ENCODING_APPLICATION=GeoExpress 6.0.0.1331
  IMAGE__HEIGHT=5000
  IMAGE__INPUT_FILE_SIZE=76841077.000000
  IMAGE__INPUT_FORMAT=GeoTIFF
  IMAGE__INPUT_NAME=M:\sp_tiffs\25229020.tif
  IMAGE__TARGET_COMPRESSION_RATIO=14.999999
  IMAGE__WIDTH=5000
  IMAGE__WKT=PROJCS["NAD83 / Massachusetts Mainland",GEOGCS["NAD83",DATUM["Nort
_American_Datum_1983",SPHEROID["GRS 1980",6378137,298.2572221010002,AUTHORITY["
PSG","7019"]],AUTHORITY["EPSG","6269"]],PRIMEM["Greenwich",0],UNIT["degree",0.0
74532925199433],AUTHORITY["EPSG","4269"]],PROJECTION["Lambert_Conformal_Conic_2
P"],PARAMETER["standard_parallel_1",42.68333333333333],PARAMETER["standard_para
lel_2",41.71666666666667],PARAMETER["latitude_of_origin",41],PARAMETER["central
meridian",-71.5],PARAMETER["false_easting",200000],PARAMETER["false_northing",7
0000],UNIT["metre",1,AUTHORITY["EPSG","9001"]],AUTHORITY["EPSG","26986"]]
  IMAGE__X_RESOLUTION=0.300000
  IMAGE__XY_ORIGIN=251486.586174,902770.874778
  IMAGE__Y_RESOLUTION=0.300000
  VERSION=MG2
Image Structure Metadata:
  COMPRESSION=JPEG
  INTERLEAVE=PIXEL
  SOURCE_COLOR_SPACE=YCbCr
Corner Coordinates:
Upper Left  (  251486.436,  902771.025) ( 70d52'29.52"W, 42d22'25.48"N)
Lower Left  (  251486.436,  901271.025) ( 70d52'30.01"W, 42d21'36.87"N)
Upper Right (  252986.436,  902771.025) ( 70d51'23.96"W, 42d22'25.12"N)
Lower Right (  252986.436,  901271.025) ( 70d51'24.46"W, 42d21'36.51"N)
Center      (  252236.436,  902021.025) ( 70d51'56.99"W, 42d22' 1.00"N)
Band 1 Block=5000x1 Type=Byte, ColorInterp=Red
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 2 Block=5000x1 Type=Byte, ColorInterp=Green
  Image Structure Metadata:
    COMPRESSION=JPEG
Band 3 Block=5000x1 Type=Byte, ColorInterp=Blue
  Image Structure Metadata:
    COMPRESSION=JPEG

[robe]

You can download the file and metadata file from:

​http://www.bostongis.com/postgisstuff/25229020.zip

I tried doing the same process with just that single file and crashes the same way.

[dustymugs]

I can't seem to replicate the problem with the following command on Linux and OSX.

raster2pgsql -I -e -F -C -Y -R -s 26986 -t 500x500 -l 2 /path/to/25229020.sid.jpg foo_outdb | psql -d test

[dustymugs]

The error message that you're getting that I find strange is…

ERROR:  rt_raster_from_hexwkb: Raster HEXWKB input must have an even number of characters

Can you send me the output from raster2pgsql instead of piping directly to psql?

[robe]

well that's might annoying. I can't replicate the issue on my windows 7 64-bit desktop with the same build. I'll check the that server again. This time I did set teh client encoding so that can't be the issue. Though I wonder if it's some weirdness with permissions again and the JPEG.

[robe]

I think that error has nothing to do with the issue. The dump itself causes a crash. So I think that error is just because the output was partially created and did not complete.

I put in ​http://www.bostongis.com/postgisstuff/bad_out.zip

I thought it may have beeen the real admin vs. not real admin issue so I ran by "run as administrator" and got the same issue. I'll try to run later without outdb to rule out outdb. It could be still a security issue since it doesn't happen on my windows 7 and this server is pretty locked down (and on top of that has a VMWare trend micro virus scanning running on the VMhostware directly too so may have been smacked by the host itself)

[robe]

I'm tempted to just close this out and chuck it up to security paranoia. I tried outputting without the outdb switch and it crashed too, so not limited to outdb. I also switched this to a low priority. I'll try to test on another windows 2008 r2 without so much draconian security measures in place.

Even though last I tried this on this particular server the overview generation worked, I don't think it has anything to do with changes in PostGIS codebase because the trend micro is a new development which I haven't (nor do I know) if I can get any useful information from.

[dustymugs]

Looking at the bad dump, the end of the file got cut off by something so the last line to COPY in is invalid. There also is no following "\." as expected nor any of the constraint additions.

[robe]

I really think this is a GDAL issue rather than anything we can do at momemnt and will probably be a non-issue if you are going to change to not use overviews in future.

I'm just going to close this out as a won't fix because it will probably moot anyway with raster2pgsql rework.