Re: [PostGIS] #4745: Possible problem with new version 2.5.5

[ezimanyi]

Dear all

Since none of the functions

void postgis_initialize_cache(FunctionCallInfo fcinfo)
const char *postgis_spatial_ref_sys()

defined in the file lwgeom_pg.c are exported, could you please instruct how to solve this issue when using call_function2 for calling the PostGIS external function transform(PG_FUNCTION_ARGS)?

If this help, here is a trace to follow the function calls

Program received signal SIGSEGV, Segmentation fault.
__strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
65      ../sysdeps/x86_64/multiarch/strlen-avx2.S: No such file or directory.
(gdb) bt
#0  __strlen_avx2 () at ../sysdeps/x86_64/multiarch/strlen-avx2.S:65
#1  0x0000556088b94ef2 in dopr (target=target@entry=0x7ffe815b9e50, format=0x7fb348b5b036 " WHERE srid = %d LIMIT 1",
    args=0x7ffe815b9ea0) at snprintf.c:443
#2  0x0000556088b95d6f in pg_vsnprintf (str=<optimized out>, count=<optimized out>, fmt=<optimized out>,
    args=args@entry=0x7ffe815b9ea0) at snprintf.c:195
#3  0x0000556088b95e43 in pg_snprintf (
    str=str@entry=0x7ffe815b9f80 "SELECT proj4text, auth_name, auth_srid, srtext FROM `U", count=count@entry=255,
    fmt=fmt@entry=0x7fb348b5b000 "SELECT proj4text, auth_name, auth_srid, srtext FROM %s WHERE srid = %d LIMIT 1")
    at snprintf.c:208
#4  0x00007fb348b0a775 in GetProj4StringSPI (srid=srid@entry=5676) at lwgeom_transform.c:345
#5  0x00007fb348b0a96f in GetProj4String (srid=5676) at lwgeom_transform.c:397
#6  AddToPROJ4SRSCache (other_srid=4326, srid=5676, PROJ4Cache=0x556089657398) at lwgeom_transform.c:496
#7  AddToPROJ4Cache (cache=cache@entry=0x556089657398, srid=srid@entry=5676, other_srid=other_srid@entry=4326)
    at lwgeom_transform.c:476
#8  0x00007fb348b0affe in GetProjectionsUsingFCInfo (fcinfo=fcinfo@entry=0x7ffe815ba270, srid1=5676,
    srid2=srid2@entry=4326, pj1=pj1@entry=0x7ffe815ba1c8, pj2=pj2@entry=0x7ffe815ba1d0) at lwgeom_transform.c:683
#9  0x00007fb348af95e9 in transform (fcinfo=0x7ffe815ba270) at lwgeom_transform.c:77
#10 0x00007fb33cf5356d in call_function2 (func=0x7fb348af9580 <transform>, arg1=93873110335448, arg2=4326)
    at /home/esteban/src/MobilityDB/src/temporal_util.c:253

Many thanks for your help.

[Algunenano]

Please don't open new tickets, comment or reopen in the old ones if necessary.

As I mentioned there, you should be passing the appropiate function context when calling the C functions. See ​https://github.com/postgis/postgis/blob/10c405f3faa48485354be22dc648daafb5a0b3f4/postgis/lwgeom_inout.c#L664 for an example.

[ezimanyi]

I am really sorry to insist but I am unable to find a solution to the problem. I have tried the following solution

PG_FUNCTION_INFO_V1(call_transform);
/**
 * Call the PostGIS transform function
 */
PGDLLEXPORT Datum
call_transform(PG_FUNCTION_ARGS)
{
	Datum result = CallerFInfoFunctionCall2(
		transform, fcinfo->flinfo, InvalidOid,
		PG_GETARG_DATUM(0), PG_GETARG_DATUM(1));
	
	/* Return null on null */
	if ( ! result )
		PG_RETURN_NULL();

	PG_RETURN_DATUM(result);
}

/**
 * Transform the geometry to the given SRID
 */
static Datum
datum_transform(Datum value, Datum srid)
{
	PG_RETURN_DATUM(DirectFunctionCall2Coll(call_transform, InvalidOid, value, srid));
}

and the problem persist

Program received signal SIGSEGV, Segmentation fault.
0x00007f6936b051f3 in GetGenericCacheCollection (fcinfo=fcinfo@entry=0x7ffdfa21a110) at lwgeom_cache.c:67
67              GenericCacheCollection* cache = fcinfo->flinfo->fn_extra;
(gdb) bt
#0  0x00007f6936b051f3 in GetGenericCacheCollection (fcinfo=fcinfo@entry=0x7ffdfa21a110) at lwgeom_cache.c:67
#1  0x00007f6936b05262 in GetPROJ4SRSCache (fcinfo=fcinfo@entry=0x7ffdfa21a110) at lwgeom_cache.c:86
#2  0x00007f6936b04ea9 in GetPROJ4Cache (fcinfo=fcinfo@entry=0x7ffdfa21a110) at lwgeom_transform.c:680
#3  0x00007f6936b04f10 in GetProjectionsUsingFCInfo (fcinfo=fcinfo@entry=0x7ffdfa21a110, srid1=4326,
    srid2=srid2@entry=5676, pj1=pj1@entry=0x7ffdfa21a0c8, pj2=pj2@entry=0x7ffdfa21a0d0) at lwgeom_transform.c:724
#4  0x00007f6936af3519 in transform (fcinfo=fcinfo@entry=0x7ffdfa21a110) at lwgeom_transform.c:77
#5  0x0000556dc42c7270 in CallerFInfoFunctionCall2 (func=0x7f6936af34b0 <transform>, flinfo=<optimized out>,
    collation=<optimized out>, arg1=<optimized out>, arg2=<optimized out>) at fmgr.c:1098
#6  0x00007f692b4d960d in call_transform (fcinfo=0x7ffdfa21a1a0)
    at /home/esteban/src/MobilityDB/point/src/tpoint_spatialfuncs.c:847
#7  0x0000556dc42c69f4 in DirectFunctionCall2Coll (func=0x7f692b4d95c9 <call_transform>, collation=<optimized out>,
    arg1=<optimized out>, arg2=<optimized out>) at fmgr.c:825
#8  0x00007f692b4d9663 in datum_transform (value=93929958568328, srid=5676)
    at /home/esteban/src/MobilityDB/point/src/tpoint_spatialfuncs.c:864

Can you please explain what I am doing wrong ? Many many thanks for you valuable help.

[Algunenano]

I think that you can't use DirectFunctionCall2Coll, as it initializes the callee fcinfo with NULL information. You need to use CallerFInfoFunctionCallXXXX all the time you might use the cache (or always if you don't want to worry about it).

[ezimanyi]

Many thanks for your prompt answer but this is exactly the problem as illustrated in the solution above. In order to call the CallerFInfoFunctionCallXXXX function I need to have a hold of the fcinfo obtained for the very first external MobilityDB function call so this should be propagated in a long chain of functions.

Wouldn't be possible to add a call

postgis_initialize_cache(fcinfo);

inside the transform(PG_FUNCTION_ARGS) function as it is done in several other external functions such as gserialized_spgist_config_2d, gserialized_spgist_config_3d, geom_from_gml, ST_AsGeoJsonRow?

[Algunenano]

In order to call the CallerFInfoFunctionCallXXXX function I need to have a hold of the fcinfo obtained for the very first external MobilityDB function call so this should be propagated in a long chain of functions.

That's what Postgis has to do. You could probably get away with caching it in a global static variable, but I would rather keep the reference when we need it. If you find a better solution I'm all ears.

Wouldn't be possible to add a call

AFAIC, that wouldn't solve anything; postgis_initialize_cache is already being called by transform via GetPJUsingFCInfo (GetProjectionsUsingFCInfo is 2.5 from what I see), which receives the fcinfo of the C function. If transform doesn't receive a "full" fcinfo it can't initialize the cache, so it doesn't matter who calls it.

Anyhow, this is all based on how I think this works, I haven't actually tested it so I'd recommend you to check it first to confirm if that's actually the issue. Also, keep in mind that this is heavily tied to how Postgis is setup and installed, as it's looking for the namespace where the function is installed, at that might be different in mobilitydb. I would strongly recommend against depending on the behaviour of Postgis C functions; an instead depend on the SQL API, which is stable and how many other projects use make use of its features.

[ezimanyi]

Dear Raúl

Many many thanks for helping me to solve the compatibility problem with PostGIS 2.5.5. I was able to succeed the regression tests again ​https://github.com/MobilityDB/MobilityDB/commits/develop Indeed, as you suggested, I needed to cache the outermost fcinfo in a global variable.

Concerning your suggestion to depend on PostGIS SQL API, this was the case of our initial MobilityDB implementation but it was impossible to use because it was too slow. A typical use case for MobilityDB is to analyze maritime data as explained in the MobilityDB workshop. ​https://github.com/MobilityDB/MobilityDB-workshop Example of such open data is provided by the Danish Maritime authority ​ftp://ftp.ais.dk/ais_data/

Each of these CSV files for a single day contains around 300 ship trips for a total of more than 10M points which means that each trip has between 3K and 4K timestamped points. We need to analyze the potential collision risks as shown next

• ​https://docs.mobilitydb.com/MobilityDB-workshop/master/ch01s08.html#imgApproach1
• ​https://docs.mobilitydb.com/MobilityDB-workshop/master/ch01s08.html#imgApproach2

We cannot afford to use the SQL API for performance reasons. Indeed, to create a single trip with 4K points we need to

• call 4K times the function ST_MakePoint
• call 4K times the function ST_SetSRID to specify that the input data is in 4326 SRID
• call 4K times the function ST_Transform to put the data in the 25832 SRID.

And all this only to load the data of a single day before analyzing every couple of trips in the data set, synchronizing (with linear interpolation) the two trips to obtain 2 trips of 4K + 4K synchronized points and

• call 4K + 4K times the function ST_Distance for each couple of trips.

As you can imagine, for such kind of manipulations we really need to access directly the liblwgeom library. The situation is much more challenging for analyzing aviation data obtained from, e.g., ​https://developer.laminardata.aero/ since in this case we need to do the computations using the PostGIS geography type and its associated functions. For this use case we needed to implement the functions ST_ClosestPoint, ST_ShortestLine, ST_LineSubstring, ST_LineInterpolatePoint, and ST_LineLocatePoint functions for geography ​https://github.com/MobilityDB/MobilityDB/blob/develop/point/src/geography_functions.c and we plan to submit a PR to PostGIS soon.

For these reasons we really need to collaborate with you in order to connect as efficiently as possible MobilityDB and PostGIS. Indeed, the overall philosophy of MobilityDB is take care of the temporal aspects and completely delegate the spatial manipulation to PostGIS. Our role is then to efficiently understand at which timestamps the current value of a spatial function may change and call PostGIS to compute the function at those timestamps.