Add retry after HTTP 503 or 504

[jpalmer]

Some WFS services are now run using reverse-proxy or load balancing services such as Nginx. In the case that a WFS request takes too long to return a result from a backend service, services such as Nginx are typically configured to return a gateway timeout error (HTTP 504). Also (maybe due to maintenance) load balancing services might return service unavailable errors (HTTP 503).

This patch makes a change to the GDAL HTTP request code to check for HTTP 503 and 504 errors and if defined by the GDAL_HTTP_MAX_RETRY and GDAL_HTTP_RETRY_DELAY config options will retry the HTTP request.

[Even Rouault]

The proposed patch enables retries by default. I'm aware of GDAL deployments where a more sophisticated retry logic (with exponential retry delay plus some random value) has been implemented on the client code to deal with particular situations, so it might not be desirable to have the retry logic enabled by default in GDAL itself. Would that be OK if we change GDAL_HTTP_MAX_RETRY default to 0 ?

[jpalmer]

Yip that sound fine to me.

I should also note that this functionality is required because large WFS paging can not be retried client side without completely restarting the whole request, which in the case of large datasets is very costly.

[Even Rouault]

trunk r28893 "CPLHTTPFetch(): add retry logic in case of 503 and 504 errors with the GDAL_HTTP_MAX_RETRY (default: 0) and GDAL_HTTP_RETRY_DELAY (default: 30 s) config options (derived from patch by jpalmer, #5920)"

In addition to the change from 5 to 0 retry by default, I've added doc for the options, fixed compiler warnings and also added freeing of some allocated memory. I couldn't easily test those situations, so it would be good if you could check it still works.

[jpalmer]

Thanks! Good point on the deallocation of memory - very sloppy of me!

I will test tomorrow.

[Even Rouault]

Replying to jpalmer:

Thanks! Good point on the deallocation of memory - very sloppy of me!

I'm not sure if any/all are really needed (depends on which code paths are taken), but hopefully they shouldn't hurt.

[Even Rouault]

Milestone renamed