Line Clipping MVT failure?

[pramsey]

This has been reported by numerous users of pg_tileserv, which is odd, because it seems very specific.

• Download the data file attached.
• Load it

  ogr2ogr  \
    -f PostgreSQL "PG:dbname=postgis30" \
    test_mvt2.gpkg \
    -lco SCHEMA=temp \
    -lco GEOMETRY_NAME=geom \
    -nln test_streams_mvt_2 \
    test_streams_mvt
  

• Run this tile generation SQL on it

  SELECT ST_AsMVT(mvtgeom, 'temp.test_streams_mvt_2', 4096, 'geom', 'id') FROM ( SELECT ST_AsMVTGeom(  ST_Transform(t."geom", 3857),  bounds.geom_clip,  4096,  256   ) AS "geom"      , "feature_code", "downstream_route_measure", "length_metre", "watershed_key_50k", "watershed_group_code_50k", "gradient", "stream_magnitude", "blue_line_key_50k", "linear_feature_id", "watershed_group_id", "fwa_watershed_code", "local_watershed_code", "watershed_group_code", "id", "edge_type", "feature_source", "left_right_tributary", "waterbody_key", "watershed_code_50k", "blue_line_key", "watershed_key", "gnis_id", "gnis_name", "stream_order", "id"    FROM "temp"."test_streams_mvt_2" t, (  SELECT ST_MakeEnvelope(-1.375621910642657e+07, 6.183449840157604e+06, -1.3736651227185564e+07, 6.20301771939861e+06, 3857)  AS geom_clip,   ST_MakeEnvelope(-1.3757442098879134e+07, 6.182226847705041e+06, -1.3735428234733e+07, 6.204240711851173e+06, 3857) AS geom_query  ) bounds WHERE ST_Intersects(t."geom",    ST_Transform(bounds.geom_query, 3005)) LIMIT 10000  ) mvtgeom;
  

The error we are seeing is

ERROR:  lwcollection_construct: mixed dimension geometries: 2/0

This is percolating up from GEOS2LWGEOM

#0  lwcollection_construct (type=<optimized out>, srid=3857, bbox=bbox@entry=0x0, ngeoms=2, geoms=geoms@entry=0x2523988)
    at lwcollection.c:60
#1  0x00007f01db7d7003 in GEOS2LWGEOM (geom=geom@entry=0x234a5b0, want3d=<optimized out>, want3d@entry=1 '\001') at lwgeom_geos.c:237
#2  0x00007f01db7d97c8 in lwgeom_clip_by_rect (geom1=geom1@entry=0x202da90, x1=-256, y1=-256, x2=4352, y2=4352) at lwgeom_geos.c:958
#3  0x00007f01db73f9e6 in mvt_unsafe_clip_by_box (clip_box=0x7ffe652ed650, lwg_in=0x202da90) at mvt.c:869
#4  mvt_iterate_clip_by_box_geos (basic_type=2 '\002', clip_gbox=0x7ffe652ed650, lwgeom=0x202da90) at mvt.c:966
#5  mvt_clip_and_validate_geos (clip_geom=<optimized out>, buffer=256, extent=33741456, basic_type=2 '\002', lwgeom=0x202da90)
    at mvt.c:1105
#6  mvt_clip_and_validate (clip_geom=<optimized out>, buffer=256, extent=33741456, basic_type=2 '\002', lwgeom=<optimized out>)
    at mvt.c:1140
#7  mvt_geom (lwgeom=<optimized out>, gbox=gbox@entry=0x1feca68, extent=extent@entry=4096, buffer=buffer@entry=256, 
    clip_geom=clip_geom@entry=true) at mvt.c:1223
#8  0x00007f01db74634d in ST_AsMVTGeom (fcinfo=0x1fdfcc8) at lwgeom_out_mvt.c:111

It has been observed back to 2.4.

On my home system, I cannot reproduce it. On a Centos7 system it reproduces (postgis 3.0.1, geos 3.8.1). }}}

[pramsey]

test data

[pramsey]

Other discussion and reproducers here ​https://github.com/CrunchyData/pg_tileserv/issues/39

[Algunenano]

A couple of notes:

• Line clipping uses GEOS in all versions; wagyu is only used for polygons in 3.0+
• It seems like the issue is found in lwgeom_clip_by_rect, so the first thing I would do is to extract the offending geometry and the the clipping values (those we can already see in the callstack) to try to see what's going on in there.

[Algunenano]

I've been able to reproduce it and minimize the offending query to the following:

WITH t as                     
(
    Select ST_SetSRID('01ea030000be0000009eefa7e60c2032411283c04ab383174100000000000014405839b4a80a203241c3f528dc6a83174100000000000014403f355e5a09203241068195c31e8317410000c420b07215408d976e520020324193180456c982174104003e0ad7a31740f2d24d02eb1f324139b4c8f6818217410300b6f3fd541b40d122db19de1f32414a0c02ab6f8217410100560e2d321e4077be9f5ac81f3241f2d24de2538217410000713d0a5721401f85eb91bb1f3241759318043a821741feff38b4c8b62240fed478c9ae1f3241f853e325208217410200105839b42440d34d62909a1f32418b6ce77bc881174105009eefa7c628403108ac7c8b1f3241560e2d3272811741fcff8095434b2c40be9f1aef861f32410681954345811741f2ffb3c876be2d40f4fdd438861f3241d9cef753298117410f005c8fc2b52e40068195e3851f32414a0c022be18017410400d7a3707d3040621058b9861f32418716d9ce99801741fe7f6891ed5c31402db29d6f871f324152b81e055a8017410700560e2df231404c378901881f3241ba490c021e8017410d801c5a645b324066666686861f324123dbf97ee57f17410a00d578e9a63240643bdf2f801f3241a8c64bb7b07f17410880e5d022db3240b6f3fd14701f324108ac1cda557f1741038047e17a343340bc749378681f3241ac1c5ae4207f17410d806ce7fb49334091ed7c1f611f324193180456eb7e1741f47feb51b85e334091ed7c1f581f324177be9f1a867e17410d80c420b0923340a8c64bb7561f3241e9263108527e1741f2ff03560ead3340f2d24d82601f3241508d97eeff7d174103809f1a2fdd3340560e2d32651f3241a8c64bb7c07d1741000000000000344060e5d0c2621f3241b0726811887d1741000000000000344091ed7cff5f1f324160e5d0a26f7d174100000000000034407368916d561f3241fed47869467d1741ff7fc876be1f3440273108cc491f3241dbf97eea1f7d1741ffffa7c64b3734407368912d371f3241560e2d32f97c1741020077be9f5a3440be9f1a6f2b1f32415eba490cdb7c1741ff7f703d0a7734403f355e9a261f32411f85eb51c27c17410c00dd24068134404e621038211f3241e17a14ae8d7c1741fb7f490c02ab344060e5d0c21b1f32419a999999547c17410d806ce7fbe934402731086c131f3241643bdf4f177c174109803d0ad7433540643bdfef051f3241b29defa7d87b1741ffff4f8d978e354025068115001f3241dd240601c87b1741fa7f39b4c89635409cc42050f01e3241cdcccc4cb57b1741fffff753e3e535402db29dcfdf1e3241a01a2fdd827b1741fa7f91ed7c5f3640c1caa185d71e3241b0726891397b17410080378941e03640a01a2f1dd41e32418b6ce7fbfc7a17410d801c5a645b3740560e2db2d11e324160e5d022c87a1741fcff809543ab374004560eadd11e32410e2db29d687a174104008716d90e38400ad7a330cf1e3241ae47e1fa3b7a1741fcffd8cef7533840bc749398c71e32411f85ebd10a7a1741faff09d7a3b0384052b81e05b91e32417f6abc74dc791741fb7f999999f9384096438b4cab1e32416f128340c6791741f7fff97e6a3c3940c74b37099d1e32419a999999cf791741f4ffcaa145963940448b6ca78e1e32412731082ced791741fd7f60e5d0e2394021b07288791e3241d7a370bdf17917410c80bc7493583a40aaf1d20d6e1e324191ed7c3fc8791741fbffb19defc73a40c520b092651e3241d7a370bd8e791741ff7f703d0a173b4046b6f3fd5a1e3241d7a370bd6479174108808d976e723b40d578e926471e3241fa7e6a3c4979174100005839b4283c403bdf4f2d331e3241b29def273e791741f47f438b6ce73c405c8fc2f51e1e3241378941e04e791741fffff753e3a53d40a4703daa121e32418195438b707917410880ed7c3f153e4010583914091e324191ed7c3f9f791741f3ff5b8fc2953e405839b488021e3241be9f1a2fd279174108808d976ef23e40b4c8761ef91d3241f6285c0ff8791741f3ff5b8fc2753f406f128300ee1d3241e3a59b44fa791741f4ffcaa145d63f40560e2d12e01d32413bdf4f0df07917410340e3a59b14404021b07228d21d324160e5d0a2e979174105c076be9f4a4040fed47809bb1d3241105839b4e5791741fcff285c8fb240403f355e1aa31d3241e5d022dbdd791741ff3f60e5d04241406f128320961d32417b14aec7d379174100c09bc420a04140d122db19871d3241ee7c3f35cd79174103403789411042406891ed3c7b1d3241355eba49c37917410640b6f3fd644240d34d6270631d324175931804ab791741fe7f1804565e43405eba49ac4d1d3241e7fba9718b791741068016d9ce274440aaf1d2ed3a1d3241aaf1d24d6c791741fcffd022db89444052b81e25271d3241e7fba97151791741fdbf78e926e14440c1caa165111d3241273108ac357917410000000000004540cdccccac011d32415c8fc2f516791741000000000000454023dbf93eed1c3241a245b6f3d77817410700aaf1d20d45409a9999f9db1c3241c520b0f288781741fabf9defa726454004560e6dd31c3241b29defa757781741000054e3a53b4540dbf97e0acc1c3241ba490c821a781741fcffd022db5945404e6210b8c31c32412fdd2406e1771741ff3f08ac1c8a4540cff753e3be1c3241f0a7c64bc8771741fabfa145b69345405c8fc235b31c3241e7fba9f1a17717410640b6f3fdc44540b4c8767eaa1c3241cff753e39077174101803f355eda45402fdd2466991c324152b81e058a771741fc3f355eba1946408716d92e851c324191ed7cbf9a77174100c09f1a2f6d464004560e0d711c3241f6285c0fa77717410000000000c04640000000a0651c32417b14ae47cd771741fe3fb07268f146405839b408601c3241e5d0225b00781741fe3f5c8fc2254740cff753635c1c3241f0a7c64b3b7817410300d34d62704740d578e966541c3241355eba498e78174105c0cef753b34740e5d0229b471c32414c3789c1e3781741f9bff1d24de24740c74b37a93b1c3241ae47e1fa39791741fb3fdd24062148407f6abcb42e1c32417368916d8f791741f9ffad47e16a48409a9999d9231c3241c3f5285cd579174104c0caa145964840a4703d2a1a1c3241df4f8d171c7a1741fdbf1c5a64bb4840be9f1a4f0f1c32412fdd2406627a174104c01e85ebe14840000000000d1c32410c022b877d7a1741ffffab1c5af4484008ac1c7a091c3241894160e5b07a174105400ad7a3104940d9cef7f3051c324106819543e47a17410340e3a59b14494089416045041c3241ec51b81e207b1741fc3f355eba2949404c378981001c32415839b4486f7b1741fb7f4160e5504940ba490cc2001c324125068195177c174105806abc74b34940df4f8d17041c3241621058b94f7c1741fabf4d6210d8494085eb5198081c32416de7fba9887c174105400ad7a3004a40d7a3703d091c3241643bdfcfac7c1741fdbf1c5a640b4a40448b6cc7021c3241b6f3fd54db7c1741fdbfc876be2f4a40e17a146efd1b324154e3a59bf27c1741ff3fb81e853b4a4008ac1cdaeb1b32414e621058237d1741fe7fbc7493684a4048e17a54da1b3241a8c64bb7547d1741ffffab1c5aa44a40fca9f1d2cb1b32417d3f35de797d1741fb7f999999d94a40b81e856bbc1b32410c022b879f7d174105405eba49fc4a40c74b3769b61b3241295c8f42a67d17410000000000004b4096438b4ca71b324154e3a51ba47d1741fe3f5c8fc2254b40643bdf6f991b32419cc420b0917d17410340378941504b4004560ead861b32414c3789c16e7d17410700fed478694b40713d0a37701b32411904560e377d17410000000000804b409a9999395a1b3241022b8716377d17410000000000804b40643bdf0f521b3241e7fba9f1417d17410000000000804b40273108cc4c1b3241aaf1d2cd547d174104002b8716894b40dd2406a14a1b3241e92631886c7d17410700fed478894b40e9263108491b3241f4fdd4f8a37d1741070052b81e954b40ee7c3f35441b3241ec51b89ef37d1741fe7f6891edac4b40894160853d1b3241b6f3fd542a7e174100c09bc420d04b40894160e5351b3241ee7c3fb5647e1741fe7f6891edfc4b4021b07228311b3241e7fba9f1af7e17410600a69bc4304c4060e5d0422c1b3241d578e926fb7e174100c0f3fdd4884c4054e3a55b261b3241d7a3703d527f17410000000000e04c4066666666201b3241be9f1aafa87f1741ff3fb4c8764e4d402db29d8f191b32412db29d6f03801741f9bff1d24db24d4000000080131b3241000000805a80174103c0726891fd4d40f4fdd4980d1b324125068195b18017410000000000404e407d3f351e071b3241643bdf4fdc801741fcff285c8f724e40e7fba971f81a3241333333b321811741fdffd8cef7b34e40931804b6e61a3241b07268115f81174100c04b3789e14e40cba14516d81a3241c520b0729881174103408fc2f5084f405839b408c51a32413bdf4f0df9811741fe7f6ce7fb494f400c022b47b71a32418b6ce7fb36821741fb3fdd2406614f40ae47e13aa71a324185eb513894821741f8bf45b6f37d4f40b29defa79a1a3241bc749318ca821741ff3f643bdf8f4f4017d9cef78c1a32414260e5d0ff821741fc3f39b4c8a64f403108acdc871a3241333333b312831741fc3f39b4c8a64f40ac1c5a24711a3241621058b94e8317410640ba490cc24f4017d9ce375f1a324115ae47e19b831741fdbf703d0ae74f408fc2f5c8501a3241d122db79c58317410180eb51b8ee4f401904564e431a324110583934ef83174103208716d9065040e17a14ce391a3241000000801984174102803d0ad70b5040a4703d8a361a32415a643bdf3484174103203108ac0c5040ac1c5a84331a32416de7fb2990841741ff1fb072681950408d976ed2321a324191ed7cbfd3841741ff5f1283c0225040dbf97e0a321a32416f1283c01b851741fd9f1a2fdd2c504039b4c896331a32418195430b4c85174101e0f97e6a34504033333393381a324154e3a51ba985174100000000004050408d976e52391a32413e0ad7a3cc8517410000000000405040dbf97e8a381a32413e0ad7a31486174100000000004050401d5a64bb381a32416abc74936086174100000000004050406abc74f3371a324148e17a94a88617410000000000405040ee7c3fb5371a324137894160c48617410000000000405040e3a59b04311a3241df4f8d17fb8617410000000000405040a245b6d32b1a3241ac1c5a641287174100000000004050402db29d0f161a32411904568e4e871741000000000040504077be9f9a0b1a3241a8c64bb778871741000000000040504046b6f35d091a32418fc2f5a89887174100000000004050407d3f35de091a324125068195c08717410000000000405040000000c0091a32418716d9ced48717410000000000405040c74b37a9101a32410ad7a3f035881741000000000040504091ed7c5f111a3241378941e0518817410000000000405040cff75363111a324189416065b18817410000000000405040986e12630a1a3241c976be1f0c89174100000000004050401904566e021a3241a8c64bb7668917410000000000405040c74b37e9fc19324140355e3a9e89174101a0438b6c475040bc749338f6193241e7fba9f1d4891741fdbfcaa1454e5040fa7e6abcec193241a4703d0a038a174100e0a7c64b5750403108ac9ce7193241c976be1f128a174100e0a7c64b575040f0a7c66bd919324196438bec1f8a17410240df4f8d5f50409a999959cb19324123dbf97e198a1741fe9fc64b37695040b81e856bbd1932411283c04a0f8a17410320d9cef773504017d9ced7a6193241fca9f1d2eb89174100000000008050401904560e93193241fca9f152c489174100000000008050406891ed3c8d1932419eefa746bb8917410000000000805040e3a59b2485193241b81e85ebbd8917410000000000805040ba490c426d193241aaf1d2cd058a17410000000000805040f2d24d2266193241f2d24d620c8a17410000000000805040d578e9465f1932410e2db29d078a17410000000000805040dd2406c152193241295c8fc2d5891741000000000080504008ac1cba4c193241333333b3d889174100000000008050406abc749346193241cba14536e389174100000000008050403bdf4fad331932410ad7a3702c8a1741000000000080504083c0cae1201932415eba490c758a174100000000008050409318041615193241f853e3a5be8a174100000000008050409eefa7460e193241df4f8d97098b17410000000000805040'::geometry, 3005) as geom
)
SELECT ST_AsMVTGeom(  ST_Transform(t."geom", 3857),  ST_MakeEnvelope(-1.375621910642657e+07, 6.183449840157604e+06, -1.3736651227185564e+07, 6.20301771939861e+06, 3857),  4096,  256   ) AS "geom"

Tested under Archlinux with latest REL_12_STABLE/HEAD for Postgresql, master/HEAD for Postgis but "just" GEOS 3.8:

 POSTGIS="3.1.0dev 130fdf694" [EXTENSION] PGSQL="120" GEOS="3.8.0-CAPI-1.13.1 " PROJ="6.3.1" LIBXML="2.9.10" LIBJSON="0.14" LIBPROTOBUF="1.3.3" WAGYU="0.5.0 (Internal)"

[Algunenano]

Updated to GEOS master/HEAD and the issue is still there.

I've tried extracting the geometry in the lwgeom_clip_by_rect and calling that directly but I couldn't reproduce it with that.

The issue comes from GEOS2LWGEOM. GEOSClipByRect is returning a multilinestring, and one of them has 3 coordinates (with z) and the other one has 2, so Postgis doesn't know what do to with it.

(gdb) s
lwcollection_construct (type=5 '\005', srid=3857, bbox=0x0, ngeoms=2, geoms=0x55633880f360) at lwcollection.c:53
(gdb) p lwgeom_to_wkt(geoms[0], 0, 20, 0)
$15 = 0x55633880f4a0 "LINESTRING(2542 4287,2540 4299,2537 4305,2530 4311,2519 4314,2511 4318,2504 4325,2499 4331,2497 4337,2497 4352)"
(gdb) p lwgeom_to_wkt(geoms[1], 0, 20, 0)
$16 = 0x55633880f650 "LINESTRING(2142.21276595745 4352,2138 4334,2136 4330,2131 4324,2121 4315,2103 4290,2097 4284,2090 4279,2085 4273,2076 4266,2073 4262,2072 4260,2072 4253,2072 4242,2074 4228,2075 4208,2071 4202,2064 4197,2061 4193,2061 4186,2063 4179,2064 4169,2060 4154,2056 4145,2052 4140,2047 4139,2039 4140,2023 4146,2021 4146,2013 4140,2011 4139,2009 4140,2005 4143,2003 4143,2001 4142,1990 4130,1986 4124,1984 4118)"
(gdb) p geoms[0]->flags
$17 = 1
(gdb) p geoms[1]->flags
$18 = 0

Note that geoms[0] is marked with 3D, but lwgeom_to_wkt doesn't show it (maybe I'm doing something wrong?).

[Algunenano]

Finally extracted the use case without any MVT calls:

SELECT ST_ClipByBox2D('SRID=3857;LINESTRING(2542 4287 5,2540 4299 5.361999999965,2537 4305 5.910000000033,2530 4311 6.832999999984,2519 4314 8.670000000042,2511 4318 10.351999999955,2504 4325 12.388000000035,2499 4331 14.146999999997,2497 4337 15.354999999981,2497 4363 18.652000000002,2484 4382 19.369999999995,2481 4390 19.572999999975,2480 4394 19.675999999978,2485 4406 20,2484 4410 20,2480 4415 20.123999999952,2466 4423 20.464999999967,2464 4425 20.503999999957,2458 4438 21.265000000014,2453 4443 21.55700000003,2451 4444 21.589000000036,2446 4446 21.898000000045,2441 4450 22.373000000021,2438 4455 22.876000000047,2437 4460 23.35699999996,2435 4475 24.327999999979,2432 4479 24.689999999944,2428 4483 24.974999999977,2423 4484 25.236000000033,2419 4483 25.587000000058,2414 4481 25.886000000057,2408 4480 26.34600000002,2404 4483 26.780999999959,2401 4488 27.089999999967,2398 4491 27.447000000044,2392 4493 28.158999999985,2385 4494 28.90399999998,2379 4492 29.648000000045,2375 4489 30.082999999984,2370 4481 30.947000000044,2368 4478 31.459999999963,2340 4479 34.521999999997,2320 4483 38.736999999965,2294 4491 42,2289 4493 42,2283 4498 42.108000000008,2277 4504 42.302000000025,2269 4517 43.079000000027,2263 4522 43.53899999999,2261 4523 43.706000000006,2255 4523 44.201000000001,2243 4521 45.5,2239 4517 45.886000000057,2234 4502 47.400999999954,2213 4462 49.908999999985,2211 4454 50.160999999964,2210 4443 50.631999999983,2211 4429 51.402000000002,2214 4418 52.089000000036,2212 4414 52.373000000021,2199 4404 53.283999999985,2188 4397 54,2184 4397 54.295000000042,2179 4398 54.626999999979,2166 4405 55,2159 4405 55,2155 4402 55.070999999996,2153 4390 55.351000000024,2149 4381 55.976000000024,2138 4334 60.5,2136 4330 60.895000000019,2131 4324 61.405999999959,2121 4315 62.069999999949,2103 4290 63.123999999952,2097 4284 63.302999999956,2090 4279 63.515999999945,2085 4273 63.805000000051,2076 4266 64.10699999996,2073 4262 64.185000000056,2072 4260 64.197999999975,2072 4253 64.396999999997,2072 4242 64.701000000001,2074 4228 65,2075 4208 65,2071 4202 65,2064 4197 65,2061 4193 65,2061 4186 65,2063 4179 65,2064 4169 65,2060 4154 65,2056 4145 65.222999999998,2052 4140 65.363999999943,2047 4139 65.493000000017,2039 4140 65.812000000035,2023 4146 66,2021 4146 66,2013 4140 66,2011 4139 66,2009 4140 66,2005 4143 66,2003 4143 66,2001 4142 66,1990 4130 66,1986 4124 66,1984 4118 66)'::geometry, ST_MakeEnvelope(-256,256,4352,4352));

ERROR:  lwcollection_construct: mixed dimension geometries: 2/0

Not a proper solution, but calling ST_Force2D on the input fixes the issue.

I'm leaving it there for today.

[pramsey]

So

(a) replace ClipByRect with Intersection?

(b) spelunk into GEOSClipByRect and find why it's returning 2D output in 3D input?

[Algunenano]

There is a c) option, which is ignore the extra dimension when calling from ST_AsMVTGeom (by adding an option of force2D); but neither a) nor c) fix the bug of ST_ClipByBox2D, which is supposed to never throw an exception, so I would go for b).

[robe]

push back if you think you'll get to this before EO Next week

[Algunenano]

Reported upstream: ​https://trac.osgeo.org/geos/ticket/1057

[pramsey]

Smaller yet

SELECT ST_ClipByBox2D('SRID=3857;LINESTRING(2497 4337 15.354999999981,2497 4363 18.652000000002,2484 4382 19.369999999995,2153 4390 55.351000000024,2149 4381 55.976000000024,2138 4334 60.5)'::geometry, ST_MakeEnvelope(-256,256,4352,4352));

[cbird]

Hi, I'm running into this issue (I think) now, with LINESTRING Z features. I generate MVTs dynamically for an application. Somewhat randomly, some lines disappear at different zoom levels. The error is: SQL Error [XX000]: ERROR: lwcollection_construct: mixed dimension geometries: 2/0.

ST_Force2D seems to fix this problem, but at a steep cost in time to generate MVTs.

Is this still a known problem? Thanks!

PostgreSQL 13.7 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit POSTGIS="3.2.0 0" [EXTENSION] PGSQL="130" GEOS="3.9.2-CAPI-1.14.3" PROJ="Rel. 4.9.3, 15 August 2016" LIBXML="2.9.4" LIBJSON="0.12.1" LIBPROTOBUF="1.2.1" WAGYU="0.5.0 (Internal)"