Buffer memory issue

[kalenik]

I am trying to compute buffer for variety of geometries and at some point program (PostgreSQL with PostGIS) crashes.

Backtrace:

Thread 1 (Thread 0x7fe788495ec0 (LWP 134001)):
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:49
#1  0x00007fe78afb7864 in __GI_abort () at abort.c:79
#2  0x00007fe78b0235ea in __malloc_assert (assertion=assertion@entry=0x7fe78b144858 "(old_top == initial_top (av) && old_size == 0) || ((unsigned long) (old_size) >= MINSIZE && prev_inuse (old_top) && ((unsigned long) old_end & (pagesize - 1)) == 0)", file=file@entry=0x7fe78b140224 "malloc.c", line=line@entry=2394, function=function@entry=0x7fe78b1450e0 <__PRETTY_FUNCTION__.3> "sysmalloc") at malloc.c:298
#3  0x00007fe78b025ce4 in sysmalloc (nb=nb@entry=8256, av=av@entry=0x7fe78b174ba0 <main_arena>) at malloc.c:2394
#4  0x00007fe78b026bf8 in _int_malloc (av=av@entry=0x7fe78b174ba0 <main_arena>, bytes=bytes@entry=8248) at malloc.c:4169
#5  0x00007fe78b0285d4 in __GI___libc_malloc (bytes=8248) at malloc.c:3078
#6  0x0000563cf2d27cfd in ?? ()
#7  0x0000563cf2d27eed in ?? ()
#8  0x0000563cf2d2d44f in repalloc ()
#9  0x0000563cf2d4b0e7 in enlargeStringInfo ()
#10 0x0000563cf2d09a00 in errcontext_msg ()
#11 0x0000563cf2d0807f in errfinish ()
#12 0x0000563cf2be6483 in quickdie ()
#13 <signal handler called>
#14 0x00007fe78b026bd7 in _int_malloc (av=av@entry=0x7fe78b174ba0 <main_arena>, bytes=bytes@entry=40) at malloc.c:4144
#15 0x00007fe78b0285d4 in __GI___libc_malloc (bytes=40) at malloc.c:3078
#16 0x00007fe78aa6cc29 in operator new(unsigned long) () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007fe73cdc4a74 in geos::geomgraph::DirectedEdgeStar::insert(geos::geomgraph::EdgeEnd*) () from /usr/local/lib/libgeos.so.3.10.0dev
#18 0x00007fe73cdce778 in geos::geomgraph::Node::add(geos::geomgraph::EdgeEnd*) () from /usr/local/lib/libgeos.so.3.10.0dev
#19 0x00007fe73cdd0bb0 in geos::geomgraph::PlanarGraph::addEdges(std::vector<geos::geomgraph::Edge*, std::allocator<geos::geomgraph::Edge*> > const&) () from /usr/local/lib/libgeos.so.3.10.0dev
#20 0x00007fe73ce36541 in geos::operation::buffer::BufferBuilder::buffer(geos::geom::Geometry const*, double) () from /usr/local/lib/libgeos.so.3.10.0dev
#21 0x00007fe73ce38a40 in geos::operation::buffer::BufferOp::bufferOriginalPrecision() () from /usr/local/lib/libgeos.so.3.10.0dev
#22 0x00007fe73ce3930d in geos::operation::buffer::BufferOp::computeGeometry() () from /usr/local/lib/libgeos.so.3.10.0dev
#23 0x00007fe73ce39362 in geos::operation::buffer::BufferOp::getResultGeometry(double) () from /usr/local/lib/libgeos.so.3.10.0dev
#24 0x00007fe73d296a9b in GEOSBufferWithParams_r () from /usr/local/lib/libgeos_c.so.1
#25 0x00007fe73d2e6f62 in buffer (fcinfo=0x563cf3f3b640) at lwgeom_geos.c:1128
#26 0x00007fe73690b080 in ?? ()
#27 0x0000563cf3f163e0 in ?? ()
#28 0x0000563cf3bd79c0 in ?? ()
#29 0x00007ffe2bea7697 in ?? ()
#30 0x0000563cf3bd7810 in ?? ()
#31 0x0000563cf3b7c4f0 in ?? ()
#32 0x0000563cf3b7ddb0 in ?? ()
#33 0x0000563cf3b7c4f0 in ?? ()
#34 0x00007ffe2bea7697 in ?? ()
#35 0x00007ffe2bea76d0 in ?? ()
#36 0x0000563cf2a91da5 in ?? ()

[kalenik]

OS: Ubuntu 20.10

[mdavis]

We need an example geometry that reproduces this crash.

[kalenik]

My case is calculation of buffers for heavy geometries in parallel and the problem is that at some point I am running out of memory and process crashes badly. I understand that with current implementation buffer calculation for heavy geometries is memory consuming process but I would prefer GEOS to throw an exception instead of crashing.

Here is the C snippet and geometry file I used to reproduce the problem. Likely you would have to limit process memory to reproduce the OOM. ​https://gist.github.com/kalenikaliaksandr/f84bc2fe3f9c5e2a01c63860477f9243 (sorry, geom is too large for attachments)

[pramsey]

Considered pre-simplifying your geometries prior to buffering if the number of vertices is particularly large?

[komzpa]

@pramsey this thingy takes down Postgres cluster from SQL level ST_Buffer. Expectation is that query will fail at most, not whole cluster.

[pramsey]

It OOMs JTS too, so "fixing" at a low level would involve radical surgery ... like a whole memory manager? or figuring out how to wrap GEOS back into the PgSQL manager? Nothing that is going to happen quickly, in other words, so I"m suggesting a workaround.

[mdavis]

There are indeed issues with the memory usage of the current buffer algorithm - see ​GEOS-344 and ​GEOS-693 for examples.

As Paul says, there is no easy fix for this. So trying to condition the inputs or the process is the only short-term solution.

I note that you are buffering with a distance of 99,999,999 and that the provided sample geometry has an extent of only about 400,000. Also, the geometry is fairly complex (420K points with 1203 holes). Computing a buffer that big on that kind of geometry is pushing the buffer algorithm way out of it's normal operating regime, and it basically is just going to produce a roundish blob anyway.

One thing you can do is to buffer at a sequence of 2 or more increasing values. In this case you could use a distance of 1,000,000, and then buffer at the 99,999,999 value.

I also wonder what exactly you are trying to do? That buffer distance of 99,999,999 looks highly artificial.

[dbaston]

Shouldn't the OOM cause a std::bad_alloc to be thrown, which should be caught by the execute wrapper in the GEOS C API?

[dbaston]

I can't reproduce a crash here. When running the attached example under limited memory (after plugging in ERROR and NOTICE handlers to initGEOS), the ERROR handler is called with the text "std::bad_alloc".

[mdavis]

Replying to dbaston:

I can't reproduce a crash here. When running the attached example under limited memory (after plugging in ERROR and NOTICE handlers to initGEOS), the ERROR handler is called with the text "std::bad_alloc".

Interesting. The OP says he encountered this in PostGIS - but wouldn't PostGIS have set those handlers as well?

In any case, we really want to try and return an actual result, rather than a crash or an error.

[mdavis]

I summarized and commented on these buffer memory issue in ​JTS-735.

For this case, the Buffer-By-Step approach should work fine. This should be fairly easy to implement.

[dbaston]

Interesting. The OP says he encountered this in PostGIS - but wouldn't PostGIS have set those handlers as well?

Yes. The problem seems to arise from the handlers that PostGIS is registering with GEOS, not with GEOS itself.

In any case, we really want to try and return an actual result, rather than a crash or an error.

Yes, although it will always be possible to exhaust memory, so it's important that the out-of-memory behavior not be a crash.