armel: _ST_Covers() returns true for point in hole

[mwanner]

I tracked down an issue in the topology test suite (test addpolygon) to the following (on ARM, little-endian):

test=# SELECT _ST_Covers('POLYGON((9 28, 16 29, 16 23, 10 23, 9 28),
                                  (15 25, 13 27, 11 24, 15 25))'::geometry,
                         'POINT(13 25)'::geometry); 
 _st_covers 
------------
 t
(1 row)

The point in question in excluded by the hole, so _ST_Covers() should return false. It works properly if you use a small polygon, instead:

test=# SELECT _ST_Covers('POLYGON((9 28, 16 29, 16 23, 10 23, 9 28),
                                  (15 25, 13 27, 11 24, 15 25))'::geometry,
                         'POLYGON((13 25, 13.1 25, 13 25.1, 13 25))'::geometry);
 _st_covers 
------------
 f
(1 row)

[mwanner]

The corresponding _ST_CoveredBy() call fails in the same way, which is the reason for test 'tickets' failing on #2084. The following is a simplified version that mistakenly returns true:

test=# SELECT _ST_CoveredBy('POINT(13 25)'::geometry, 'POLYGON((9 28, 16 29, 16 23, 10 23, 9 28),(15 25, 13 27, 11 24, 15 25))'::geometry);
 _st_coveredby 
---------------
 t
(1 row)

However, I guess this runs through the same code path.

[mwanner]

patch to fix armel, where char is unsigned

[mwanner]

An 'int' return value gets casted to 'char' in several places. The latter isn't necessarily signed, so a '(int) -1' value may well be converted to a positive 'char' value on architectures where 'char' is unsigned.

The attached patch fixes this issue by clearly separating the 'int' and 'char' (or 'bool') values.

[robe]

strk or pramsey — see any issue with this patch?

[pramsey]

Commtited to 2.1 at r12895, trunk at r12896