Opened 13 years ago

Last modified 12 years ago

#4011 new defect

WFS maxFeatures support broken in many cases.

Reported by: sdlime Owned by: sdlime
Priority: high Milestone: 6.2 release
Component: MapServer C Library Version: 6.0
Severity: major Keywords:
Cc: warmerdam, assefa

Description

Passing a value for maxFeatures in a WFS request doesn't work in some cases. This is because of how that filtering is done relative to FILTER processing. Since FILTERs are used increasingly for both attribute and some spatial filters (e.g. intersect) this becomes a problem. The layerObj maxfeatures property is used...

Presently it's up to a driver to apply maxfeatures, for example, PostGIS sets a limit on the SQL statement. Not all drivers implement this so it's impossible to use maxFeatures in some cases. A better solution would be apply feature counting in the query and drawing functions that request geometries from drivers. This allows for all drivers to be supported and makes it possible to count results after all requisite filters have been applied.

Steve

Attachments (2)

4011.patch (6.3 KB ) - added by sdlime 13 years ago.
Patch against branch-6-0 for query side of things…
4011-2.patch (7.2 KB ) - added by aboudreault 12 years ago.

Download all attachments as: .zip

Change History (21)

by sdlime, 13 years ago

Attachment: 4011.patch added

Patch against branch-6-0 for query side of things...

comment:1 by myOpenLayersUName, 13 years ago

I have compiled a version of MapServer for Linux using the patches attached to this ticket and tested the WFS queries I have been previously unable to perform against Shapefiles.

I was able to successfully perform the queries with filters and maxFeatures.

Yay!!!! You can't see it, but I'm doing a 'very' 'happy dance' in my cubicle!

comment:2 by warmerdam, 13 years ago

Cc: warmerdam added

comment:3 by assefa, 13 years ago

Cc: assefa added

comment:4 by adube, 13 years ago

I just tried this patch and it solved my issue as well : the filters of my queries are performed before the maxFeatures, thus now behaves as expected. Thanks sdlime for the patch.

comment:5 by assefa, 13 years ago

Hi Steve,

Pagination support with WFS: the wfs request can send a startindex and a maxfeatures. The startindex is an integer value the allows users to paginate through a set of results.

Current support is done at 2 levels:

  • for the oracle driver, the layer->startindex element is taken into account to do the queries.
  • for other drivers: msGMLWriteWFSQuery is where we pass through the elements until we get to the start index and output from there to the max features (if set).

With the patch proposed in this bug, we also need to take into account the startindex and apply the logic in the query and draw functions. Note also that removing the logic at the driver side and put it higher up might end up not being efficient when you deal with a big set of results (which is usually the case for clients using pagination)

comment:6 by sdlime, 13 years ago

I'd appreciate any ideas on how to efficiently support both. Pagination support is more uneven than max results. Kinda seems like if both pagination AND max results are set then the drivers that support pagination would produce a result set with max results or less and would be unaffected by my patch. I mean, in PostGIS if you set a start value the local result set still starts at 0 doesn't it?

If no pagination is necessary then the drivers wouldn't limit things on their side and would let MapServer do it (as in the patch).

Steve

comment:7 by aboudreault, 12 years ago

Steve, we tried the patch on a development server and it seems that it introduced a new bug. If the startIndex is >0, the maxfeature is not taken into account.

in reply to:  7 comment:8 by sdlime, 12 years ago

Replying to aboudreault:

Steve, we tried the patch on a development server and it seems that it introduced a new bug. If the startIndex is >0, the maxfeature is not taken into account.

Which driver? I may have been a bit overzealous in my gutting of the Oracle and/or PostGIS code. My thinking now is that we'd only use maxfeatures IF start index is set as well.

Steve

comment:9 by aboudreault, 12 years ago

We are using postgis.

comment:10 by rouault, 12 years ago

(in case it might help, or confuse ;-), I've read a possibly related discussion on GeoServer side : see the "Paging" paragraph at http://geoserver.org/display/GEOS/GSIP+61+-+WFS+2.0 )

in reply to:  9 comment:11 by sdlime, 12 years ago

Replying to aboudreault:

We are using postgis.

Try backing out the patch for mappostgis.c and test again. It rally shouldn't matter if the driver tries to honor maxfeatures.

Steve

comment:12 by aboudreault, 12 years ago

Steve, the performance impact is considerable. Oracle and PostGIS both support a maxfeature and index parameters. It looks to me that it's a way better to keep those drivers using the native method of doing this...

comment:13 by aboudreault, 12 years ago

well, I'm not aware of the other issues... so I will assume that we *really* can't use those features from the db. I'll attach my modified patch that fixed my issue (maxfeatures ignored if startindex is set but the driver doesn't support it) Please, review and let me know if you and I should commit.

by aboudreault, 12 years ago

Attachment: 4011-2.patch added

comment:14 by aboudreault, 12 years ago

note that I wasn't really sure of the use of the 3 if statements (maxfeatures>0 && startindex<0, etc.).. that's what I've modified/simplified. Take a look at it and let me know if you see something wrong for the desired behavior.

comment:15 by sdlime, 12 years ago

This looks good at first glance. A better solution than my original patch. I missed the layer supports paging API function originally. Have you been able to test across providers with both start index and max features?

Steve

comment:16 by aboudreault, 12 years ago

I've tested postgis and shapefiles providers. I do not have any other providers here.

comment:17 by aboudreault, 12 years ago

Steve, I see that the startindex is handled in the drivers... shouldn't it be handled in mapserver too? I guess we will experience issues when using maxfeature and startindex if they are not applied at the same level.. I would suggest to put this in 6.0.3... to be sure to not break 6.0.2.

comment:18 by aboudreault, 12 years ago

Milestone: 6.0.2 release6.2 release

comment:19 by aboudreault, 12 years ago

for the record, here is how pagination should be done in ORACLE (provided by mdsmith):

select * 
  from ( select a.*, rownum rnum
           from ( YOUR_QUERY_GOES_HERE -- including the order by ) a
          where rownum <= MAX_ROWS )
 where rnum >= MIN_ROWS
Note: See TracTickets for help on using tickets.