vsis3: AWS HTTP 307 redirect responses fails to authenticate

[tveastman]

We've found that many buckets fail to be listed using /vsis3/

Amazon AWS will sometimes return a 'HTTP/1.1 307 Temporary Redirect', requiring the client to make the same request to a different endpoint. This is common with newly created buckets.

For example, a call to list the contents of 'bucket-name.s3.amazonaws.com' can return a 307 redirect to 'kx-test-s3-scan-import.s3-ap-southeast-2.amazonaws.com'.

The /vsis3/ handler follows this redirect, but it does not seem to update the 'Authorization' header for the request.

Because the 'Host' header has changed, the Authorization header is no longer valid, and the request fails with '403 Forbidden'.

It seems that what's required is for the Authorization: header to be rebuilt before making the follow-up (redirected) request.

AWS docs on Temporary Request Redirection are at: ​https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html

I haven't found AWS docs yet that *explicitly* state the Authorization header needs to be recalculated, but I note that 'host' is in the list of 'SignedHeaders', and the Host header *does* change after redirect, while the signature does not.

[tveastman]

Here's a transcript of the two curl requests made after an attempt to read the directory, I have redacted my AWS_ACCESS_KEY_ID

The smoking gun, I think, is that the Host header changes between the two requests, the Authorization header specifies 'SignedHeaders=host', and yet the signature has not been updated.

In [33]: gdal.ReadDir('/vsis3/kx-test-s3-scan-import/')

> GET /?delimiter=/ HTTP/1.1
Host: kx-test-s3-scan-import.s3.amazonaws.com
Accept: */*
x-amz-date: 20171031T013051Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20171031/ap-southeast-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=973a3c2db94867c18b19f808290608fc7f54b0b359d33a50980b8d58b0c3ed73

< HTTP/1.1 307 Temporary Redirect
< x-amz-bucket-region: ap-southeast-2
< x-amz-request-id: 5A712999630381D5
< x-amz-id-2: JfErqIqhkOEhXAaSxMylOKGEhUhLhsM+r8EtI3tkf1G3DuFyWRPC4FXxuIUmzc5GmN4CzPluZh8=
< Location: https://kx-test-s3-scan-import.s3-ap-southeast-2.amazonaws.com/?delimiter=/
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 31 Oct 2017 01:30:51 GMT
* Server AmazonS3 is not blacklisted
< Server: AmazonS3


> GET /?delimiter=/ HTTP/1.1
Host: kx-test-s3-scan-import.s3-ap-southeast-2.amazonaws.com
Accept: */*
x-amz-date: 20171031T013051Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20171031/ap-southeast-2/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=973a3c2db94867c18b19f808290608fc7f54b0b359d33a50980b8d58b0c3ed73

< HTTP/1.1 403 Forbidden
< x-amz-bucket-region: ap-southeast-2
< x-amz-request-id: 8E0DA2DD9CF89026
< x-amz-id-2: 8YfrNVucdfN1+WDV6ZHkF0Gf28MxodR5vR6SpMzJlTJWiKfUF7uKMKUHoEzDfKryBQme1xK5z5g=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 31 Oct 2017 01:30:51 GMT
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
<
* Connection #12 to host kx-test-s3-scan-import.s3-ap-southeast-2.amazonaws.com left intact

[Even Rouault]

In 40591:

/vsis3/: properly handle 307 TemporaryRedirection (fixes #7116)

[Even Rouault]

Would you mind retrying with latest trunk ? I didn't try to reproduce with the "real" S3 servers, but tested against the stub used for autotest suite.

[tveastman]

I can't test the patch yet, but when I can I'll update this ticket with my results.

[tveastman]

I've tested a 2.3 build, and the signatures are now updated on redirect, and the call succeeds.

However, there's a 400 BAD REQUEST response in the request logs, I wanted to have you look and check if that's expected behaviour or not.

Here's an example call to a test bucket I created, I've redacted my AWS_ACCESS_KEY_ID. You'll see a couple calls, including a 400 response, and a successful result.

In other words, looks like the bug is fixed, but is there something else going wrong?

> GET /?delimiter=/ HTTP/1.1
Host: an-irish-bucket.s3.amazonaws.com
Accept: */*
x-amz-date: 20171105T214501Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6fd339cbe15000c65447219bbd3894b086a5de8f7acd75ab0b782b501160386a

< HTTP/1.1 307 Temporary Redirect
< x-amz-bucket-region: eu-west-1
< x-amz-request-id: B44BE218EBCA093A
< x-amz-id-2: BXi5XGEqC/CeTm/q123XmV5aye3E1/pORy6WedOJlEBKYvtl0MuYMOeXB4yUIP43pSn2UdaFFvA=
< Location: https://an-irish-bucket.s3-eu-west-1.amazonaws.com/?delimiter=/
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Sun, 05 Nov 2017 21:45:02 GMT
< Server: AmazonS3

> GET /?delimiter=/ HTTP/1.1
Host: an-irish-bucket.s3-eu-west-1.amazonaws.com
Accept: */*
x-amz-date: 20171105T214502Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=81a0717fb17dd5db8ac422c774ca8b3a94b5cd249621be822d1ff746de0896b1

< HTTP/1.1 400 Bad Request
< x-amz-bucket-region: eu-west-1
< x-amz-request-id: 1E8E55536F81A634
< x-amz-id-2: c475qXUrpdJfH0h997Es3x9GIeoqSEMlkSeowMIvwEBAFBLnfGalQ07ZlkIirbyoA8M9+UAbYFo=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Sun, 05 Nov 2017 21:45:04 GMT
< Connection: close
< Server: AmazonS3


> GET /?delimiter=/ HTTP/1.1
Host: an-irish-bucket.s3-eu-west-1.amazonaws.com
Accept: */*
x-amz-date: 20171105T214504Z
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f7af380f40e5de098880cbab5e40a599c1075f74c4d65253f2c746eebf191c78

< HTTP/1.1 200 OK
< x-amz-id-2: 204lMzM7t5dEOEOq+NMkuSlPnxOnV2nvuTlsWJ+eDouEeTM0h/QbjdGrBEq/Z06FmY18wzeArII=
< x-amz-request-id: 5CA4CA211D857B47
< Date: Sun, 05 Nov 2017 21:45:06 GMT
< x-amz-bucket-region: eu-west-1
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Server: AmazonS3
['a-directory']

[Even Rouault]

This is more or less expected. The code doesn't handle the "x-amz-bucket-region: eu-west-1" response header that indicates the change of region, so it first retry with the currrent/default region (us-east-1) and the 400 response will give it the new region, so there's some potential for optimization there