SetIgnoredFields and Set(Spatial/Attribute)Filter not work together

[bishop]

If I set some fields to ignore including geometry, no features returned. I.e. if I execute SQL query in any DB like

SELECT aaa from some_table WHERE bbb="ccc";

I'll receive some results, but in GDAL, if I set ignore field "bbb" and Attribute filter 'bbb="ccc"' I'll receive no features.

All the same if I add "GEOMETRY" and SetSpatialFilter I'll receive no features.

The practical case: for fast creating spatial tree I set ignoring fields for all, except geometry, but if user set any attribute filter the tree will be empty.

Another case user set ignore fields to filter them out from attributes table, but if some attribute filter is set, the table will be empty.

[Even Rouault]

Dmitry, could you provide a reproducable way for those issues ? My attempts do not show any problem.

$ ogr2ogr -f sqlite poly.sqlite ../autotest/ogr/data/poly.shp -dsco spatialite=yes

$ python
>>> from osgeo import ogr
>>> ds = ogr.Open('poly.sqlite')
>>> lyr = ds.GetLayer(0)
>>> lyr.SetIgnoredFields(['eas_id'])
0
>>> lyr.SetAttributeFilter('eas_id = 170')
0
>>> feat = lyr.GetNextFeature()
>>> feat.DumpReadable()
OGRFeature(poly):10
  area (Real) = 5268.813
  eas_id (Real) = (null)
  prfedea (String) = 35043413
  POLYGON ((479750.6875 4764702.0,479658.59375 4764670.0,479640.09375 4764721.0,479735.90625 4764752.0,479750.6875 4764702.0))

>>> sql_lyr = ds.ExecuteSQL('SELECT prfedea FROM poly WHERE eas_id = 170')
>>> feat = sql_lyr.GetNextFeature()
>>> feat.DumpReadable()
OGRFeature(SELECT):0
  prfedea (String) = 35043413

[bishop]

I faced this problem on ESRI shp.

~$ python
Python 2.7.5+ (default, Feb 27 2014, 19:37:08) 
[GCC 4.8.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from osgeo import ogr
>>> ds = ogr.Open('../autotest/ogr/data/poly.shp')
>>> lyr = ds.GetLayer(0)
>>> lyr.SetIgnoredFields(['area','eas_id','prfedea'])
0
>>> lyr.SetAttributeFilter('eas_id = 170')
0
>>> feat = lyr.GetNextFeature()
>>> feat.DumpReadable()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'DumpReadable'
>>> 

Maybe this issue not affected on all drivers.

[Even Rouault]

Fixing that doesn't seem trivial. The issue must affect all drivers that have not a SQL engine, but rely on the OGR sqlite engine, and that take into account ignored fields. Basically when you set a field as ignored, it will not be retrieved in the returned feature. But as filtering happens after retrieving the feature, the filter operates on a null field.

A potential not-so-hard fix could be to undo an ignored field if it is used in attribute filter. But the field would appear in the feature fields, which isn't presumably the expected behaviour (although probably OK). One issue with that approach being that if you unset the filter, the field will remain non-ignored.

I'd say for now: do not set an ignored field that is in an attribute filter.

Perhaps something to discuss with Martin Dobias, the author of http://trac.osgeo.org/gdal/wiki/rfc29_desired_fields

[wonder]

Is there an easy way for OGR to find out which fields will be used in the SQL query? Knowing them would allow OGR to override the ignored fields settings...

I understand the "ignored fields" functionality just as a hint for OGR that some fields will not be used by the client and therefore can be ignored - most of the drivers do not have support for this functionality anyway. So if a field appears within feature fields even if it was set as ignored, that is completely fine.

[bishop]

Replying to wonder:

Is there an easy way for OGR to find out which fields will be used in the SQL query? Knowing them would allow OGR to override the ignored fields settings...

Yes, it can be easily get from swq parser of attribute filter. But if we ignore such ignored fields, the SetIgnoredFields will not work as expected. Or we have to make this fields empty in result feature?

[wonder]

Replying to bishop:

But if we ignore such ignored fields, the SetIgnoredFields will not work as expected. Or we have to make this fields empty in result feature?

The idea behind SetIgnoredFields was to tell OGR that it does not need to load those fields. It does not say that it must not load them. The only reason is to speed up the fetching if possible. I don't think the fields need to be explicitly emptied if set to be ignored but loaded because of the filter.

[Even Rouault]

This ticket has been automatically closed because Trac is no longer used for GDAL bug tracking, since the project has migrated to GitHub. If you believe this ticket is still valid, you may file it to ​https://github.com/OSGeo/gdal/issues if it is not already reported there.