Opened 10 years ago

Closed 10 years ago

#837 closed defect (fixed)

[raster] ST_MapAlgebra -- I think we have a memory leak

Reported by: robe Owned by: jorgearevalo
Priority: blocker Milestone: PostGIS 2.0.0
Component: raster Version: main
Keywords: Cc:

Description (last modified by robe)

I tried to do this on my test elephant

-- its a single tile -- I forget the size O

ALTER TABLE ch13.pele ADD column map_rast raster;
UPDATE ch13.pele 
SET map_rast = ST_MapAlgebra(rast,'CASE WHEN rast BETWEEN 100 and 250 THEN 1 WHEN rast = 252 THEN 2 WHEN rast BETWEEN 253 and 254 THEN 3 ELSE 0 END', '2BUI') ; 

and it crashes the server after about 13 seconds.

This is on Windows XP latest trunk.

Attachments (1)

pele.sql (1.0 MB) - added by robe 10 years ago.

Download all attachments as: .zip

Change History (21)

Changed 10 years ago by robe

Attachment: pele.sql added

comment:1 Changed 10 years ago by robe

Description: modified (diff)
Version: 1.5.Xtrunk

comment:2 Changed 10 years ago by robe

Description: modified (diff)

comment:3 Changed 10 years ago by robe

Component: postgispostgis raster
Owner: changed from pramsey to jorgearevalo
Summary: ST_MapAlgebra -- I think we have a memory leak[raster] ST_MapAlgebra -- I think we have a memory leak

comment:4 Changed 10 years ago by jorgearevalo

Server crashes when returning serialized generated raster. Debugging.

comment:5 Changed 10 years ago by dzwarg

Jorge:

I traced it back to trunk/raster/rt_core/rt_api.c#L3025. Hope this helps.

comment:6 Changed 10 years ago by pracine

Priority: mediumblocker

comment:7 Changed 10 years ago by dzwarg

The file changed since I made that last comment.

See trunk/raster/rt_core/rt_api.c@6911#L3025

comment:8 Changed 10 years ago by dzwarg

Hello Jorge,

I have found that the size of the raster triggers this bug. It happens to me when I process 100x100 rasters, but it does not happen when processing 10x10 rasters.

comment:9 Changed 10 years ago by jorgearevalo

Status: newassigned

Please, give a try to r6958. Now the server doesn't crash for me, but I had to comment SPI_finish calls. Not recommended, I think. I've found this 10-years old unresponded question, with the same problem: http://www.mail-archive.com/pgsql-general@postgresql.org/msg18558.html

I think, I'll try this: http://www.mail-archive.com/pgsql-general@postgresql.org/msg79320.html

And asking pgsql list too. Anyway, give it a try.

comment:10 in reply to:  7 Changed 10 years ago by jorgearevalo

Replying to dzwarg:

The file changed since I made that last comment.

See trunk/raster/rt_core/rt_api.c@6911#L3025

Thanks David. Really useful :-)

comment:11 Changed 10 years ago by jorgearevalo

Just for the record. Differents behaviors of the function in PostgreSQL 8.4.7 and PostgreSQL 9.0.3 when uncommenting the SPI_finish call:

  • PostgreSQL 8.4.7: Crash caused by SIGSEV, segmentation fault.
  • PostgreSQL 9.0.3: Error deserializing raster (Unknown pixeltype: 15). And incorrect number of bands reported (32639 instead of 1)

Link to SQL file with the raster table used, 14400 rows: http://dl.dropbox.com/u/6599273/srtm_35_04.tgz

Query that causes different behaviours when SPI_finish is uncommented: select st_mapalgebra(rast, 1, 'rast + 10', '-1 * rast') from srtm_35_04 where rid < 3651

The where clause is not necessary. The problem simply appears when the query processes more than 3650 rows.

comment:12 Changed 10 years ago by pracine

And what happen when SPI_finish is commented out?

comment:13 Changed 10 years ago by jorgearevalo

Without SPI_finish call you get this warning message:

WARNING: transaction left non-empty SPI stack
HINT: Check for missing "SPI_finish" calls.

But the application doesn't crash (for me at least). Now is commented.

comment:14 Changed 10 years ago by pracine

My first attemp at the C implementation of ST_MapAlgebra ever:

CREATE TYPE geomvalxy AS (
    geom geometry,
    val double precision,
    x int,
    y int
);

CREATE OR REPLACE FUNCTION ST_PixelAsPolygons(rast raster, band integer) 
    RETURNS SETOF geomvalxy AS 
    $$
    DECLARE
        rast alias for $1;
        w integer;
        h integer;
        x integer;
        y integer;
        result geomvalxy;
    BEGIN
        SELECT st_width(rast), st_height(rast)
        INTO w, h;
        FOR x IN 1..w LOOP
             FOR y IN 1..h LOOP
                 SELECT ST_PixelAsPolygon(rast, band, x, y), ST_Value(rast, band, x, y), x, y INTO result;
            RETURN NEXT result;
         END LOOP;
        END LOOP;
        RETURN;
    END;
    $$
    LANGUAGE 'plpgsql';

-- This query works
SELECT ST_MapAlgebra(ST_AddBand(ST_MakeEmptyRaster(5, 5, 0, 0, 1, 1, 0, 0, -1), '8BUI'::text, 0, -1), 'rast + 1')

-- This query crashes (certainly because SPI_finish was not done)
SELECT ST_PixelAsPolygons(ST_MapAlgebra(ST_AddBand(ST_MakeEmptyRaster(5, 5, 0, 0, 1, 1, 0, 0, -1), '8BUI'::text, 0, -1), 'rast + 1'))

with

ERROR:  SPI_connect failed: SPI_ERROR_CONNECT
CONTEXT:  SQL function "st_pixelaspolygons" statement 1

********** Error **********

ERROR: SPI_connect failed: SPI_ERROR_CONNECT
SQL state: XX000
Context: SQL function "st_pixelaspolygons" statement 1

comment:15 Changed 10 years ago by robe

Jorge,

Sorry to take so long to reply. On my windows mingw install it gives same message as yours and answers are in right ball park.

Takes 13 seconds to do the map call.

SELECT distinct (r).val
FROM (SELECT ST_DumpAsPolygons(map_rast) as r
FROM ch13.pele) As foo

Takes 359 ms.

So ST_DumpAsPolygons seems much faster and my elephant is recognizable in open jump. I haven't done any other tests though.

comment:16 in reply to:  15 Changed 10 years ago by jorgearevalo

Replying to robe:

Jorge,

Sorry to take so long to reply. On my windows mingw install it gives same message as yours and answers are in right ball park.

Takes 13 seconds to do the map call.

SELECT distinct (r).val
FROM (SELECT ST_DumpAsPolygons(map_rast) as r
FROM ch13.pele) As foo

Takes 359 ms.

So ST_DumpAsPolygons seems much faster and my elephant is recognizable in open jump. I haven't done any other tests though.

Great news :-).

comment:17 Changed 10 years ago by jorgearevalo

For the record. The query doesn't crash in Ubuntu 10.10 64bits machine. I guess is a matter of time (takes more time to full the available memory). Classic memory leak here.

Question sent to pgsql-general list:http://archives.postgresql.org/pgsql-general/2011-04/msg00130.php

comment:18 Changed 10 years ago by pracine

Are you sure the bug is not related to #851 and our own memory management with the context? I would start by understanding 851 and fix it properly. I would not be surprised to see 837 disappear with 851 fixed.

comment:19 Changed 10 years ago by jorgearevalo

Yes, they're probably related. I'm debugging #851 now.

comment:20 Changed 10 years ago by jorgearevalo

Resolution: fixed
Status: assignedclosed

Fixed in r7025. I close the ticket.

Note: See TracTickets for help on using tickets.