Skip to content

Use newer S3 hostname style

Erwan Rouchet requested to merge s3-addressing-style into master

Closes #772 (closed)

We had an issue while deploying %Data export to preprod where the export bucket would fail due to signature issues. The /export/<id>/download/ endpoint was correctly doing a HTTP 302 to preprod-arkindex-export.s3.amazonaws.com, but then we would get a HTTP 307 to preprod-arkindex-export.s3.eu-west-3.amazonaws.com. The change in hostname breaks the signature and causes the error.

The docs are not that clear about it, but it seems that referring to buckets as name.s3.amazonaws.com is deprecated because it was always sent to us-east-1, which then had to proxy to the correct region. On newer buckets, it might do a HTTP 307 to name.s3.eu-west-3.amazonaws.com, breaking the signature. On some buckets created in recently-created regions, it would do a HTTP 400.

botocore is even less explicit about this situation. While it has a configuration option specifically for us-east-1 to cause it to always use name.s3.us-east-1.amazonaws.com, but nothing to explicitly enable this new behavior.

Turns out the s3.addressing_style option enables this:

  • auto (default): name.s3.amazonaws.com. Deal with it.
  • virtual: name.s3.region.amazonaws.com
  • path: s3.region.amazonaws.com/path

The behavior set by auto is deprecated while the other two are fully supported by AWS according to its docs.

Using virtual with a custom endpoint URL will however cause another behavior to occur: With an endpoint set to s3.lol, you get URLs to http://bucketname.s3.lol/path. Therefore, to avoid breaking everything when using MinIO, I use the default addressing style when an endpoint URL is set, and otherwise (when the real S3 will be used) use virtual.

Edited by Erwan Rouchet

Merge request reports