MAP EXTENT truncates GetFeature requests for data outside bounds

[tomkralidis]

If data in a given layer is outside the bounds of the MAP/EXTENT object, this
data is not included in a blanket GetFeature request, i.e.:

http://[host]/mapserv?version=1.0.0&service=WFS&request=GetFeature&typename=roads

It should be.

[tomkralidis]

I can confirm that this behaviour still exists.

Thinking about this again, I wonder what our best approach is here. Even though the layer bbox does not overlap MAP/EXTENT, there _is_ data there, in accordance to what a WFS client thinks should be returned (i.e. all data).

I guess this leads us to the question: what relevance does MAP/EXTENT have in WFS Server (mapwfs.c)?

Comments?

[sdlime]

Depends on the query type. For attributes the extent is used as a pre-filter before any attribute-based filtering is done. Does WFS just wrap the msQueryUsing... functions or does it have it's own processing logic? I assume the latter since it supports spatial filters that the CGI-based query modes do not.

What sorts of OGC filters are we talking about here?

Steve

[sdlime]

One workaround would be to set the map->extent based on the layers data extent before doing any queries...

Steve

[assefa]

The user has the ability to use a bbox as part of a wfs query and that would be used to set the extents before doing a query. For me I think it is an acceptable workaround.

[tomkralidis]

Replying to assefa:

The user has the ability to use a bbox as part of a wfs query and that would be used to set the extents before doing a query. For me I think it is an acceptable workaround.

Right. What about the following use case: the user invokes the URL as per:

​http://[host]/mapserv?version=1.0.0&service=WFS&request=GetFeature&typename=roads

...in this case, the client is expecting all the data to return (i.e. no filters have been applied)

?

[assefa]

ok convinced :) In that case, what we should I think would be to use the bbox extents that were advertised for the layer (in the getcapabilities) and set the map extent with it. Any bbox settings that are part of the query would overwrite this default behavior.

[sdlime]

I wish we could do this (Assefa's) logic for the CGI-based attribute query but there's no way to know if the map->extent was modified or if it is the original. Anyway, I think Assefa's idea to use the layer extent(s) is sufficient.

Steve

[tomkralidis]

Fixed in r7674. Tested with:

$ ~/foss4g/mapserver/trunk/mapserver/mapserv QUERY_STRING="map=config.map&service=WFS&version=1.0.0&request=GetFeature

against:

MAP
 NAME "mstrac-1287"
 STATUS ON
 SIZE 400 300
 #EXTENT -180 -90 180 90
 EXTENT 10 10 20 20  # (does not overlap data)
 IMAGECOLOR 255 255 255
 WEB
  IMAGEPATH "/tmp/"
  IMAGEURL "/ms_tmp/"
  METADATA
   "wfs_onlineresource" "http://foo"
   "wfs_srs"               "EPSG:4326"
  END
 END
 PROJECTION
  "init=epsg:42304"
 END
 LAYER
  NAME obs
  STATUS ON
  TYPE POINT
  DATA "../../trunk/msautotest/wxs/data/obs.shp"
  DUMP TRUE
  METADATA
    "wfs_title"         "obs"
    "wfs_featureid"     "stn_id"
  END
  PROJECTION
    "init=epsg:4326"
  END
  CLASS
   NAME "obs"
   COLOR 200 255 0
   OUTLINECOLOR 120 120 120
  END
 END
 LAYER
  NAME prov
  STATUS ON
  TYPE POINT
  DATA "../../trunk/msautotest/wxs/data/province.shp"
  DUMP TRUE
  METADATA
    "wfs_title"         "province"
    "wfs_featureid"     "name"
  END
  PROJECTION
    "init=epsg:42304"
  END
  CLASS
   NAME "obs"
   COLOR 200 255 0
   OUTLINECOLOR 120 120 120
  END
 END
END

..testing and editing in the mapfile:

• same projections
• different projections
• absense of projection defs

If anyone can give this a once over or test, that would be great.

[bartvde]

Changeset r7674 introduced a new problem, changing the sequence of queries in the WFS GetFeature request can now yield different results, since the map->extent is set to the last query layer's extent. This is done in a separate loop before the queries are actually run. So this can result in one query sequence resulting in 1 result (the expected), whereas changing the sequence of the Query elements may yield 0 results.

[tomkralidis]

Bart: thanks for the info. Can you post an example of some sort to clarify?

[bartvde]

Hey Tom, I've uploaded a testcase to my website:

​http://www.osgis.nl/download/mapserver/bug1287.tgz

it contains 2 shapefiles, a MAP file and 2 Python scripts, as well as the output which I am getting from the two scripts. As you can see the sequence of the Query tags will change the output (this is the only difference between the two Python scripts).

If I remove the following line in mapwfs.c all is fine:

bbox = map->extent = ext;

[tomkralidis]

Bart: if I comment out that line in mapwfs.c, the problem as originally specified in this ticket comes back.

I also looked your testcase. The desired behaviour would be to set the bbox on a per layer basis, as the case of querying against > 1 typename in a GetFeature case, per your testcase.

Assefa: any comments here on approach? Thinking out loud:

• aggregate all typename bbox's into one which encompasses them all, then run the query. BUT, what happens if different bbox's are specified for each?

[bartvde]

Hey Tom, indeed I was thinking the same, aggregate the bbox's of the several layers from the request. Though it's kind of a shame that a WFS attribute query will end up with an unnecessary spatial query as well, but I guess there is too much to be done to change that ....

If different BBOX's are specified in the request, does the code not go through msQueryByRect with the bbox of the layer?

if(msQueryByRect(map, -1, bbox) != MS_SUCCESS) {

[assefa]

Sorry to be late on this. Couple of notes on this:

I feel the extent value we advertise in the capabilities document for each layer should be able to read a map level metadata (ows/wfs_extent) that takes precedence on the layer->extent values being used. This would allow users to limit queries on all the layers to a certain extent. Of course each layer can override this by specifying a layer level ows_extent metadata.

Aggregating could be one way of doing it. Another possibility is to do the msQueryByRect on each layer (using the advertised extents).

Here are the diifrent scenarios:

• getfeature request has a bbox element: the bbox is used to do do the queries
• the request has a Filter element (bbox and filter are mutually exclusive): if the filter contains a spatial bbox, we use it in the query. If no spatial extent is given, we use the advertises extent of the layer (instead of the current behavior of using the map->extent). Changes here should be trivial.
• no bbox, no filter: either aggregate the advertised extents or do a query by rect on each feature type using it's extent.

Comments?

[assefa]

patch against mapserver 5.6

[assefa]

Added a patch that I believe fixes the initial problem mentioned by tom. It should also work fix the bug reported by Bart. Will commit/add more tests after 5.6

[assefa]

test case to use for msautotest: results for hits are different when the layer's order is inverted (which should not happen). Note that there is a bug in the hits count #3224 to be fixed before testing this.

​http://127.0.0.1/cgi-bin/mapserv.exe?map=f:/projects/mapserver-buildkit-2008/mapserver/msautotest/wxs/wfs_filter.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=popplace,province&resulttype=hits

​http://127.0.0.1/cgi-bin/mapserv.exe?map=f:/projects/mapserver-buildkit-2008/mapserver/msautotest/wxs/wfs_filter.map&SERVICE=WFS&VERSION=1.1.0&REQUEST=GetFeature&TYPENAME=province,popplace&resulttype=hits

[assefa]

patch added in trunk (r9597) msautotest added r9599

[bartvde]

Verified, works fine.