SHP: Loss of precision causing wrong spatial query results

[danstoica]

The nature of the problem is the fact the bounding box is stored both in SHP and IDX files. The SpatialExtents() aggregate function is using the first one which might not be exactly the same as the other one, causing primary filtering to miss features on the boundary.

For very large numbers (say greater than 8 significant digits) or very high accuracy (like in lat/long datasets, where 6 digits accuracy is common) the tolerance used to compensate the precision loss is not enough (that is 0.001/2 or 0.0000001/2 for LL). That is, the difference between the extents stored in SHP (as doubles) and the extents stored in IDX (as floats) can be practically in any range. The solution is to compute this offset and adjust the bounding box accordingly before performing the primary filtering.