vsis3 requires AWS Tokens, doesn't support other authentication methods

[perrygeo]

Amazon distributes a number of public datasets on S3, most notably Landsat (​https://aws.amazon.com/public-data-sets/landsat/)

These are accessible by unauthenticated/public users

$ echo $AWS_SECRET_ACCESS_KEY

$ aws s3 ls s3://landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B2.TIF
2015-03-14 19:20:21   51934592 LC81390452014295LGN00_B2.TIF
2015-03-14 19:20:18    6863541 LC81390452014295LGN00_B2.TIF.ovr

But the vsis3 virtual file system requires credentials

$ gdalinfo /vsis3/landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B2.TIF
ERROR 1: AWS_SECRET_ACCESS_KEY configuration option not defined
gdalinfo failed - unable to open '/vsis3/landsat-pds/L8/139/045/LC81390452014295LGN00/LC81390452014295LGN00_B2.TIF'.

This is also a problem for private s3 objects where access is configured by other mechanisms such as Access Control Lists (​http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html?tag=viglink123524-20#CannedACL) which don't rely on user env vars.

I have not looked too deeply into the vsis3 code but would it possible to avoid a hard check for access key, instead passing these requests through and catching any authentication errors reported by the service?

[Even Rouault]

For public datasets, I would just use /vsicurl/​https://landsat-pds.s3.amazonaws.com/... . If /vsis3/ could be used for public datasets, that would probably mean we should do a first request to figure out if credentials are needed or not.

Regarding ACL, I couldn't really understand how that works when skimming quickly through the URL you provided, particularly how the "grantee" is supposed to advertize itself in the request so that the server figures out if he has rights or not.

[perrygeo]

For public datasets, agreed that https is a viable alternative.

For authentication methods from within the AWS infrastructure, access is granted through IAM roles that are tied to the instances. I don't pretend to understand the details fully but this might be a more informative link:

​http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

[perrygeo]

For authenticating within AWS, it looks like the keys and tokens are obtained using an additional network request[1]. Docs seem to suggest that AWS client applications should be responsible for this.

[1] ​http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials

[Even Rouault]

Implmented per r39470 and following commits