Regress Error on Bessie/Winnie 32-bit tickets full join test - 32-bit hash issue

[robe]

21:57:45 PostgreSQL 10.5 on i386-portbld-freebsd11.2, compiled by FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0), 32-bit
21:57:45   Postgis 3.0.0alpha3dev - r17557 - 2019-06-23 01:51:52
21:57:45   scripts 3.0.0alpha3dev r17557
21:57:45   GEOS: 3.6.3-CAPI-1.10.3 80c13047
21:57:45   PROJ: Rel. 5.1.0, June 1st, 2018
21:57:45 
21:57:45 Running tests
21:57:45 
21:57:45  ../loader/Point .............. ok 
21:57:47  ../loader/PointM .............. ok 
21:57:48  ../loader/PointZ .............. ok 
21:57:50  ../loader/MultiPoint .............. ok 
21:57:51  ../loader/MultiPointM .............. ok 
21:57:52  ../loader/MultiPointZ .............. ok 
21:57:53  ../loader/Arc .............. ok 
21:57:55  ../loader/ArcM .............. ok 
21:57:56  ../loader/ArcZ .............. ok 
21:57:57  ../loader/Polygon .............. ok 
21:57:59  ../loader/PolygonM .............. ok 
21:58:00  ../loader/PolygonZ .............. ok 
21:58:02  ../loader/TSTPolygon ......... ok 
21:58:03  ../loader/TSIPolygon ......... ok 
21:58:03  ../loader/TSTIPolygon ......... ok 
21:58:04  ../loader/PointWithSchema ..... ok 
21:58:05  ../loader/NoTransPoint ......... ok 
21:58:06  ../loader/NotReallyMultiPoint ......... ok 
21:58:07  ../loader/MultiToSinglePoint ......... ok 
21:58:07  ../loader/ReprojectPts ........ ok 
21:58:08  ../loader/ReprojectPtsGeog ........ ok 
21:58:09  ../loader/Latin1 .... ok 
21:58:09  ../loader/Latin1-implicit .... ok 
21:58:09  ../loader/mfile .... ok 
21:58:09  ../dumper/literalsrid ....... ok 
21:58:11  ../dumper/realtable ....... ok 
21:58:12  affine .. ok 
21:58:12  bestsrid .. ok 
21:58:12  binary .. ok 
21:58:12  boundary .. ok 
21:58:13  chaikin .. ok 
21:58:13  filterm .. ok 
21:58:13  cluster .. ok 
21:58:13  concave_hull .. ok 
21:58:15  ctors .. ok 
21:58:15  curvetoline .. ok 
21:58:15  dump .. ok 
21:58:15  dumppoints .. ok 
21:58:15  empty .. ok 
21:58:16  estimatedextent .. ok 
21:58:16  forcecurve .. ok 
21:58:16  geography .. ok 
21:58:16  geometric_median .. ok 
21:58:16  hausdorff .. ok 
21:58:17  in_geohash .. ok 
21:58:17  in_gml .. ok 
21:58:17  in_kml .. ok 
21:58:17  in_encodedpolyline .. ok 
21:58:18  iscollection .. ok 
21:58:18  legacy .. ok 
21:58:19  long_xact .. ok 
21:58:19  lwgeom_regress .. ok 
21:58:19  measures .. ok 
21:58:20  minimum_bounding_circle .. ok 
21:58:20  normalize .. ok 
21:58:20  operators .. ok 
21:58:20  orientation .. ok 
21:58:20  out_geometry .. ok 
21:58:21  out_geography .. ok 
21:58:21  polygonize .. ok 
21:58:21  polyhedralsurface .. ok 
21:58:21  postgis_type_name .. ok 
21:58:21  quantize_coordinates .. ok 
21:58:21  regress .. ok 
21:58:22  regress_bdpoly .. ok 
21:58:22  regress_buffer_params .. ok 
21:58:22  regress_gist_index_nd .. ok 
21:58:25  regress_index .. ok 
21:58:29  regress_index_nulls .. ok 
21:58:29  regress_management .. ok 
21:58:29  regress_selectivity .. ok 
21:58:30  regress_lrs .. ok 
21:58:30  regress_ogc .. ok 
21:58:30  regress_ogc_cover .. ok 
21:58:30  regress_ogc_prep .. ok 
21:58:30  regress_proj .. ok 
21:58:32  relate .. ok 
21:58:32  remove_repeated_points .. ok 
21:58:33  removepoint .. ok 
21:58:33  reverse .. ok 
21:58:33  setpoint .. ok 
21:58:33  simplify .. ok 
21:58:33  simplifyvw .. ok 
21:58:33  size .. ok 
21:58:34  snaptogrid .. ok 
21:58:34  split .. ok 
21:58:34  sql-mm-serialize .. ok 
21:58:34  sql-mm-circularstring .. ok 
21:58:34  sql-mm-compoundcurve .. ok 
21:58:35  sql-mm-curvepoly .. ok 
21:58:35  sql-mm-general .. ok 
21:58:35  sql-mm-multicurve .. ok 
21:58:35  sql-mm-multisurface .. ok 
21:58:36  swapordinates .. ok 
21:58:36  summary .. ok 
21:58:36  temporal .. ok 
21:58:36  temporal_knn .. ok 
21:58:36  tickets .. failed (diff expected obtained: /home/jenkins/tmp/pgis_reg_f9334cb0058fc5a796335d219109738a0d1e9ae5/test_98_diff)
21:58:39 -----------------------------------------------------------------------------
21:58:39 --- tickets_expected	2019-06-22 21:51:32.916336000 -0400
21:58:39 +++ /home/jenkins/tmp/pgis_reg_f9334cb0058fc5a796335d219109738a0d1e9ae5/test_98_out	2019-06-22 21:57:58.244904000 -0400
21:58:39 @@ -390,3 +390,4 @@
21:58:39  ERROR:  BOX2D_construct: args can not be empty points
21:58:39  #4176|t
21:58:39  #4394
21:58:39 +#4394
21:58:39 -----------------------------------------------------------------------------
21:58:39  twkb .. ok 
21:58:39  typmod .. ok 
21:58:48  wkb .. ok 
21:58:48  wkt .. ok 
21:58:48  wmsservers .. ok 
21:58:49  offsetcurve .. ok 
21:58:49  relatematch .. ok 
21:58:49  isvaliddetail .. ok 
21:58:50  sharedpaths .. ok 
21:58:50  snap .. ok 
21:58:50  node .. ok 
21:58:50  unaryunion .. ok 
21:58:50  clean .. ok 
21:58:51  relate_bnr .. ok 
21:58:52  delaunaytriangles .. ok 
21:58:52  clipbybox2d .. ok 
21:58:52  subdivide .. ok 
21:59:56  voronoi .. ok 
21:59:56  regress_brin_index .. ok 
21:59:57  regress_brin_index_3d .. ok 
21:59:58  regress_brin_index_geography .. ok 
21:59:58  minimum_clearance .. ok 
21:59:58  oriented_envelope .. ok 
21:59:58  in_geojson .. ok 
21:59:59  mvt .. ok 
21:59:59  geobuf .. ok 
21:59:59  mvt_jsonb .. ok 
21:59:59  uninstall .. ok (4663)
22:00:00 
22:00:00 Run tests: 126
22:00:00 Failed: 1
22:00:01 gmake[3]: *** [Makefile:215: check] Error 1
22:00:01 gmake[3]: Leaving directory '/usr/home/jenkins/workspace/PostGIS_Worker_Run/label/bessie32/f9334cb0058fc5a796335d219109738a0d1e9ae5/regress/core'
22:00:01 gmake[2]: *** [Makefile:43: check] Error 2
22:00:01 gmake[2]: Leaving directory '/usr/home/jenkins/workspace/PostGIS_Worker_Run/label/bessie32/f9334cb0058fc5a796335d219109738a0d1e9ae5/regress'
22:00:01 gmake[1]: *** [GNUmakefile:20: check] Error 1
22:00:01 gmake[1]: Leaving directory '/usr/home/jenkins/workspace/PostGIS_Worker_Run/label/bessie32/f9334cb0058fc5a796335d219109738a0d1e9ae5'
22:00:01 *** Error code 2
22:00:01 
22:00:01 Stop.
22:00:01 make: stopped in /usr/home/jenkins/workspace/PostGIS_Worker_Run/label/bessie32/f9334cb0058fc5a796335d219109738a0d1e9ae5
22:00:01 Build step 'Execute shell' marked build as failure
22:00:01 Finished: FAILURE

This is failing on new test for full join which is a new feature introduced in #4394

It looks like 2 rows are being returned instead of 1 for #4394

[robe]

okay this seems to be a 32-bit issue for sure. I turned on one of winnie's 32-bits (I wasn't testing 32-bit on winnie before and her 64-bits were passing).

Here 32-bit yields the same issue:

PostgreSQL 10.8, compiled by Visual C++ build 1800, 32-bit
  Postgis 3.0.0alpha3dev - r17557 - 2019-06-23 04:57:21
  scripts 3.0.0alpha3dev r17557
  GEOS: 3.8.0-CAPI-1.11.0 
  PROJ: Rel. 4.9.3, 15 August 2016

 tickets .. failed (diff expected obtained: /projects/postgis/tmp/3.0.0alpha3dev_pg10_geos3.8_gdal2.2.4w32/test_98_diff)
-----------------------------------------------------------------------------
--- tickets_expected	2019-06-22 19:17:09.794524200 -0400
+++ /projects/postgis/tmp/3.0.0alpha3dev_pg10_geos3.8_gdal2.2.4w32/test_98_out	2019-06-23 01:13:24.185978100 -0400
@@ -390,3 +390,4 @@
 ERROR:  BOX2D_construct: args can not be empty points
 #4176|t
 #4394
+#4394
-----------------------------------------------------------------------------

I'll try next with 9.6/32-bit on winnie to rule out PostgreSQL 10/32-bit issue

[robe]

Tested winnie 9.6/32bit and same issue.

[robe]

based on discussion on irc with Darafei

I did this:

 update pg_operator set oprcanhash = false, oprcanmerge = true where oprname = '=' and oprcode = 'geometry_eq'::regproc;

and then the test on bessie32

 with mj as (select 'POINT(0 0)'::geometry geom) select '#4394' from mj a full join mj b on a.geom = b.geom;

Yielded 1 row as it should.

Both in 3.0 are set to true.

So it seems there is something wrong with our hashing logic on 32-bit.

[Algunenano]

I see an issue with integer conversion around the hash functions:

The SQL function declares receiving an integer (32bits signed, according to the docs):

-- Availability: 2.5.0
CREATE OR REPLACE FUNCTION geometry_hash(geometry)
	RETURNS integer
	AS 'MODULE_PATHNAME','lwgeom_hash'
	LANGUAGE 'c' STRICT IMMUTABLE _PARALLEL;

-- Availability: 2.5.0
CREATE OPERATOR CLASS hash_geometry_ops
	DEFAULT FOR TYPE geometry USING hash AS
    OPERATOR    1   = ,
    FUNCTION    1   geometry_hash(geometry);

The C function returns an Int64GetDatum, which is converted directly from an uint64_t:

PG_FUNCTION_INFO_V1(lwgeom_hash);
Datum lwgeom_hash(PG_FUNCTION_ARGS)
{
	GSERIALIZED *g1 = PG_GETARG_GSERIALIZED_P(0);
	uint64_t hval = gserialized_hash(g1);
	PG_FREE_IF_COPY(g1, 0);
	PG_RETURN_DATUM(Int64GetDatum(hval));
}

So the output of gserialized_hash is doing uint64 → int64 → int32, which could be an issue.

[robe]

In 17575:

fix for 32-bit hash
Closes #4433
Closes ​https://github.com/postgis/postgis/pull/429

[Raul Marin]

In 17577:

Improve hash changes NEWS

Since the output of the function has changed, any
index using it must be reindexed

References #4433