Support paging for ArcGIS REST API

[jpalmer]

The ArcGIS REST API 10.3 now supports paging using resultOffset and resultRecordCount request parameters. Would be good to provide the option to use this paging functionality when reading a HTTP JSON endpoint like the WFS driver.

Recently I've had issues requesting large datasets using the JSON feature request, probably due to server side memory issues. Using this new paging API would probably solve this problem.

Here's an example URL that using the paging request parameters:

​http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=pgeojson&resultOffset=0&resultRecordCount=100

Esri supporting docs:

​http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Query_Feature_Service_Layer/02r3000000r1000000/

supportsPagination: ​http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Layer/02r3000000w6000000/

[Even Rouault]

trunk r29111 "GeoJSON: implement automatic scrolling through result sets of ArcGIS GeoServices Feature Service (#5943)"

[jpalmer]

Wow thanks Even. Good incredible!

I came across a few issue in my first test:

1/ Setting the page size via the resultRecordCount KVP didn't work for me as the offset page size still had 1000 increments. e.g:

ogr2ogr -f GPKG geodetic.gpkg "http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=geojson&resultRecordCount=10000" --debug on

HTTP: Fetch(http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=geojson&resultRecordCount=10000)
HTTP: These HTTP headers were set: Accept: text/plain, application/json
GDAL: GDALOpen(http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=geojson&resultRecordCount=10000, this=0x7ff2ada223a0) succeeds as GeoJSON.
GDAL: QuietDelete(geodetic.gpkg) invoking Delete()
GDAL: GDALDriver::Create(GPKG,geodetic.gpkg,0,0,0,Unknown,0x0)
HTTP: Fetch(http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=geojson&resultRecordCount=10000&resultOffset=1000)
HTTP: These HTTP headers were set: Accept: text/plain, application/json
HTTP: Fetch(http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=geojson&resultRecordCount=10000&resultOffset=2000)
HTTP: These HTTP headers were set: Accept: text/plain, application/json

2/ Maybe it's a good idea to provide a config/open option to set the page size?

3/ If I use JSON as the request output format from the server I get index errors. I have to use geojson or pgeojson to avoid this. Under a 10.1 service it's ok to use f=json.

ogr2ogr -f GPKG geodetic.gpkg "http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=json&resultRecordCount=10000" --debug on 2>&1 | more
HTTP: Fetch(http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/query?where=1%3D1&outFields=&returnGeometry=true&returnZ=false&returnM=false&f=json&resultRecordCount=10000)
HTTP: These HTTP headers were set: Accept: text/plain, application/json
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1
ERROR 1: Invalid index : -1

etc..

This older 10.1 query works:

ogr2ogr -f GPKG ta2.gpkg "http://maps.stats.govt.nz/wss/service/arcgis1/guest/Boundaries/2013_Geographies/MapServer/9/query?where=1=1&text=&objectIds=&time=&geometry=&geometryType=esriGeometryEnvelope&inSR=&spatialRel=esriSpatialRelIntersects&relationParam=&outFields=&maxAllowableOffset=&outSR=&outSR=&outFields=&outFields=&geometry=&returnZ=true&returnM=true&gdbVersion=&f=json&resultRecordCount=10" -nln ta_bdys -nlt PROMOTE_TO_MULTI -lco GEOMETRY_NAME=shape --debug on

[Even Rouault]

Replying to jpalmer:

Wow thanks Even. Good incredible!

I came across a few issue in my first test:

1/ Setting the page size via the resultRecordCount KVP didn't work for me as the offset page size still had 1000 increments. e.g:

Yes, the value is limited by the server to a maximum value ("The maximum value for this parameter is the value of the layer's maxRecordCount property." in ​http://resources.arcgis.com/en/help/arcgis-rest-api/index.html#/Query_Feature_Service_Layer/02r3000000r1000000/). Word added in the driver doc now

2/ Maybe it's a good idea to provide a config/open option to set the page size?

Perhaps, but you're already obliged to write a lot of parameters in the URL, so...

3/ If I use JSON as the request output format from the server I get index errors. I have to use geojson or pgeojson to avoid this. Under a 10.1 service it's ok to use f=json.

Hum, with outFields= (empty), there's a "current_mark_name" attribute in features[], but not declared in fields[]. Strange behaviour from the server... If you specify, outFields=current_mark_name or outFields=*, you'll get consitant declaration. I've done a fix to avoid the error to be emitted, but with outFields=, you'll get only the geometry.

I've also fixed detection of paging in f=json.

trunk r29120 "GeoJSON: avoid errors when reading ESRIJson that has properties in features, but not declared. Fix detection of pagin in p=json format (#5943)"

[jpalmer]

Thanks Even. Ah yes the serverside constraint. Maybe a warning could be raised to the user if the set page size is greater than the supported max record count? Guess could be done with check of exceededTransferLimit flag or capabilities URL e.g ​http://services.arcgis.com/XTtANUDT8Va4DLwI/arcgis/rest/services/LINZ_Geodetic_Marks/FeatureServer/0/?f=pjson

Thanks for the fix on the f=json.

[Even Rouault]

trunk r29124 "GeoJSON: select maximum page size for ArcGIS Feature Service scrolling instead of hard-coded 1000 limit, and issue a warning if the user specified a resultRecordCount value greater than the one allowed by the server (#5943)"

[Even Rouault]

Milestone renamed