#7116 closed defect (fixed)
vsis3: AWS HTTP 307 redirect responses fails to authenticate
Reported by: | tveastman | Owned by: | warmerdam |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | default | Version: | 2.2.2 |
Severity: | normal | Keywords: | VSIS3 S3 AWS 307 redirect signature |
Cc: | robert.coup@… |
Description
We've found that many buckets fail to be listed using /vsis3/
Amazon AWS will sometimes return a 'HTTP/1.1 307 Temporary Redirect', requiring the client to make the same request to a different endpoint. This is common with newly created buckets.
For example, a call to list the contents of 'bucket-name.s3.amazonaws.com' can return a 307 redirect to 'kx-test-s3-scan-import.s3-ap-southeast-2.amazonaws.com'.
The /vsis3/ handler follows this redirect, but it does not seem to update the 'Authorization' header for the request.
Because the 'Host' header has changed, the Authorization header is no longer valid, and the request fails with '403 Forbidden'.
It seems that what's required is for the Authorization: header to be rebuilt before making the follow-up (redirected) request.
AWS docs on Temporary Request Redirection are at: https://docs.aws.amazon.com/AmazonS3/latest/dev/Redirects.html
I haven't found AWS docs yet that *explicitly* state the Authorization header needs to be recalculated, but I note that 'host' is in the list of 'SignedHeaders', and the Host header *does* change after redirect, while the signature does not.
Change History (6)
comment:3 by , 7 years ago
Would you mind retrying with latest trunk ? I didn't try to reproduce with the "real" S3 servers, but tested against the stub used for autotest suite.
comment:4 by , 7 years ago
I can't test the patch yet, but when I can I'll update this ticket with my results.
comment:5 by , 7 years ago
I've tested a 2.3 build, and the signatures are now updated on redirect, and the call succeeds.
However, there's a 400 BAD REQUEST response in the request logs, I wanted to have you look and check if that's expected behaviour or not.
Here's an example call to a test bucket I created, I've redacted my AWS_ACCESS_KEY_ID. You'll see a couple calls, including a 400 response, and a successful result.
In other words, looks like the bug is fixed, but is there something else going wrong?
> GET /?delimiter=/ HTTP/1.1 Host: an-irish-bucket.s3.amazonaws.com Accept: */* x-amz-date: 20171105T214501Z x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=6fd339cbe15000c65447219bbd3894b086a5de8f7acd75ab0b782b501160386a < HTTP/1.1 307 Temporary Redirect < x-amz-bucket-region: eu-west-1 < x-amz-request-id: B44BE218EBCA093A < x-amz-id-2: BXi5XGEqC/CeTm/q123XmV5aye3E1/pORy6WedOJlEBKYvtl0MuYMOeXB4yUIP43pSn2UdaFFvA= < Location: https://an-irish-bucket.s3-eu-west-1.amazonaws.com/?delimiter=/ < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Sun, 05 Nov 2017 21:45:02 GMT < Server: AmazonS3 > GET /?delimiter=/ HTTP/1.1 Host: an-irish-bucket.s3-eu-west-1.amazonaws.com Accept: */* x-amz-date: 20171105T214502Z x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=81a0717fb17dd5db8ac422c774ca8b3a94b5cd249621be822d1ff746de0896b1 < HTTP/1.1 400 Bad Request < x-amz-bucket-region: eu-west-1 < x-amz-request-id: 1E8E55536F81A634 < x-amz-id-2: c475qXUrpdJfH0h997Es3x9GIeoqSEMlkSeowMIvwEBAFBLnfGalQ07ZlkIirbyoA8M9+UAbYFo= < Content-Type: application/xml < Transfer-Encoding: chunked < Date: Sun, 05 Nov 2017 21:45:04 GMT < Connection: close < Server: AmazonS3 > GET /?delimiter=/ HTTP/1.1 Host: an-irish-bucket.s3-eu-west-1.amazonaws.com Accept: */* x-amz-date: 20171105T214504Z x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 Authorization: AWS4-HMAC-SHA256 Credential=AKIAREDACTED/20171105/eu-west-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=f7af380f40e5de098880cbab5e40a599c1075f74c4d65253f2c746eebf191c78 < HTTP/1.1 200 OK < x-amz-id-2: 204lMzM7t5dEOEOq+NMkuSlPnxOnV2nvuTlsWJ+eDouEeTM0h/QbjdGrBEq/Z06FmY18wzeArII= < x-amz-request-id: 5CA4CA211D857B47 < Date: Sun, 05 Nov 2017 21:45:06 GMT < x-amz-bucket-region: eu-west-1 < Content-Type: application/xml < Transfer-Encoding: chunked < Server: AmazonS3 ['a-directory']
comment:6 by , 7 years ago
This is more or less expected. The code doesn't handle the "x-amz-bucket-region: eu-west-1" response header that indicates the change of region, so it first retry with the currrent/default region (us-east-1) and the 400 response will give it the new region, so there's some potential for optimization there
Here's a transcript of the two curl requests made after an attempt to read the directory, I have redacted my AWS_ACCESS_KEY_ID
The smoking gun, I think, is that the Host header changes between the two requests, the Authorization header specifies 'SignedHeaders=host', and yet the signature has not been updated.