[raster] ST_Union raster out of memory then kicks the bucket on windows 64bit

[robe]

I ran this test on my mingw32 build (on PostgreSQL windows 32-bit on windows 7 64-bit) and my mingw64bit (on PostgreSQL windows 64-bit windows 2003).

SELECT ST_AsEWKT(ST_ConvexHull(ST_Union(rast1.rast, 1))) FROM (
		 	(SELECT ST_SetSRID(ST_SetValue(ST_AddBand(ST_MakeEmptyRaster( 100, 100, (i-1)*100, (i-1)*100, 0.0005, -0.0005, 0*i, 0*i), '16BSI'), i, (i+1),-6000),4326) As rast FROM generate_series(1,10) As i) ) As rast1 LIMIT 3;

On my postgsi 32 bit it gives error

NOTICE:  Neither raster provided has a NODATA value for the specified band indices.  NODATA value set to minimum possible for 16BSI
CONTEXT:  PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1
ERROR:  out of memory
DETAIL:  Failed on request of size 373628520.
CONTEXT:  PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1

********** Error **********

ERROR: out of memory
SQL state: 53200
Detail: Failed on request of size 373628520.
Context: PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1

on the 64-bit build it gives

NOTICE:  Neither raster provided has a NODATA value for the specified band indices.  NODATA value set to minimum possible for 16BSI
CONTEXT:  PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1
********** Error **********

Then just kills the service.

[pracine]

You're trying to make a 1800100 x 1800100 pixel raster…

What is the behavior of PostGIS if you try to build a polygon having the same number of points as this raster would have pixels?

[Bborie Park]

I think what robe meant for this ticket was that the 64-bit server was not failing gracefully. It should be failing in a manner similar to the 32-bit server.

Now the question is why is it failing horribly.

[pracine]

Is the 32-bit failing so gracefully?

[Bborie Park]

robe, can you retry this?

Testing the example query in the Windows Experimental 64-bit PostGIS on PostgreSQL 9.1 in Windows 7 Pro results in the following

NOTICE:  Neither raster provided has a NODATA value for the specified band indices.  NODATA value set to minimum possible for 16BSI
CONTEXT:  PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1
ERROR:  invalid memory alloc request size 1137072672
CONTEXT:  PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1

********** Error **********

ERROR: invalid memory alloc request size 1137072672
SQL state: XX000
Context: PL/pgSQL function "_st_mapalgebra4unionstate" line 13 at RETURN
SQL function "_st_mapalgebra4unionstate" statement 1

It isn't crashing out on me.

[robe]

No still crashes. I'm testing on a windows 2003 server 64-bit box. I'll try it on a windows 7 and windows 2008 R2 64-bit tomorrow and see what happens.

How many gb do you have. I have 8GB on this box, but its running about 5 postgresql services on it though that shouldn't make a difference.

[Bborie Park]

Now that you mention it, I never changed any of the defaults in postgresql.conf after installing EDB's version of PostgreSQL. So…

shared_buffers = 32MB

The VM has 4GB RAM.

[pracine]

I must add to this that some weeks ago we did a big raster/raster intersection (ST_Clip actually) involving millions of 10x10 tiles and polygons the size of North Dakota and the memory what going up and up until it crashed. We did the same without tiling the raster and it went through it without problem. I guess there must be some kind of memory leek in ST_MapAlgebra(raster, raster).

[Bborie Park]

I just ran the example query in 64-bit Linux (VM with 2GB RAM and shared_mem = 512M, work_mem = 16M) through valgrind and have no memory leaks beyond what is normally there (PostgreSQL-specific leaks that I've always seen).

[Bborie Park]

On repeated runs of the example query, PostgreSQL 9.1 with PostGIS 64-bit in Windows 7 crashes consistently. It does run without crashing (query still fails) the first time the query is run but crashes on the second attempt.

Repeated runs of the example query in 64-bit Linux with PostgreSQL 9.1 and PostGIS 64-bit does not result in a crash.

[robe]

This query doesn't crash anymore:

SELECT ST_AsEWKT(ST_ConvexHull(ST_Union(rast1.rast, 1))) FROM (
		 	(SELECT ST_SetSRID(ST_SetValue(ST_AddBand(ST_MakeEmptyRaster( 100, 100, (i-1)*100, (i-1)*100, 0.0005, -0.0005, 0*i, 0*i), '16BSI'), i, (i+1),-6000),4326) As rast FROM generate_series(1,10) As i) ) As rast1 LIMIT 3;

It just gives an error:

ERROR:  rt_raster_new: Dimensions requested exceed the maximum (65535 x 65535) permitted for a raster

Which seems fair enough to me. So I'm closing this out.