Use object redirects for AWS S3 file alt locations.

Description

S3 website redirection was added in GUISE-88. It used S3 routing rules, a special bucket-level configuration that allowed redirects to be specified for certain paths.

This seems to work well, but there's one is one huge roadblock: the number of redirects configured using routing rules is apparently limited to 50. If you go over that, the AWS SDK gives this error:

software.amazon.awssdk.services.s3.model.S3Exception: … routing rules provided, the number of routing rules in a website configuration is limited to 50. (Service: S3, Status Code: 400, Request ID: …)

Others have ran into this problem. However as someone pointed out on Stack Overflow, there was another redirect feature announced in 2012: object-level redirects.

With this approach one must store a zero-byte (or any content, apparently) object with a special x-amz-website-redirect-location metadata value that either indicates the path to another S3 object or a URL. We'll need to update the Guise Mummy S3 website deployment to use this object-based redirect, at least for files.

S3 websites now have a redirectMeans configuration setting, with these options:

  • optimal: (default) Functions like object except that if the total number of redirects is below redirectCountOptimalThreshold (which currently defaults to 30 but can be configured), functions like routing-rule.

  • object: All redirects use objects, except for redirects from collections, which must use routing rules.

  • routing-rule: All redirects use routing rules.

Environment

None

Activity

Show:
Garret Wilson
May 9, 2020, 5:04 PM

The documentation isn't very clear on this point, but it appears that I can't just set normal metadata with a key of x-amz-website-redirect-location, or S3 will think it is custom metadata and give it a prefix, yielding x-amz-meta-x-amz-website-redirect-location. As pointed out on Stack Overflow, it look like I need to use a special setting when I put the object.

Garret Wilson
May 9, 2020, 9:09 PM
Edited

Despite all the thinking of when to use routing rules for collection redirections, it turns out that S3 doesn't even seem to support "normal" objects in collection form. That is if we try to store an object at foo/bar/, it winds up as some sort of folder on S3 without any metadata and no way to turn it into a redirect object. (Redirect information sent with the object seems to be ignored.) So we'll need to use routing rules for all redirects that have a collection as the source, not just those going to another collection.

Note that this will preclude implementation of GUISE-91.

Actually considering this a little more, the question of whether to use a routing rule or an object redirect; and the question of whether to use a base replacement or a full path replacement; are two separate questions. Even though we must use a routing rule to route foo/ to bar or to bar/, we can still decide to use a routing rule of <ReplaceKeyPrefixWith> instead of <ReplaceKeyWith> based upon whether both are collections.

Garret Wilson
May 9, 2020, 10:31 PM

The fact that we cannot redirect from a collection using an object means that even when we have object mode turned on, we will have to still use routing rules for collections. And that is a basically the auto setting. Which means there is no point in having a separate auto redirect means; it will now be the same as object.

Garret Wilson
May 9, 2020, 11:30 PM
Edited

The good news is that with the usual CloudFront distribution using Route 53 and Amazon certificates, the object redirects will redirect to the CloudFront domain, not to the underlying bucket as redirect rules do. The bad news is that for some reason there is an intermediate HTTP redirect; that is:

  1. https://example.com/foo.html redirects to

  2. http://example.com/bar.html redirects to

  3. https://example.com/bar.html

Interestingly this happens even using routing rules (e.g. for sites deployed using the original implementation)! To bypass the HTTP step we may want to build the entire redirect URL for CloudFront distributions, even for routing-rule based redirects, to force the use of HTTPS in the initial redirect.

I opened a Stack Overflow question to try to understand this more, and then opened up a separate ticket since this affects routing rules as well and was an existing issue.

Garret Wilson
May 10, 2020, 6:17 PM

It turns out that the object redirect value must be URI-encoded in addition to being an absolute path. (See AWS Java SDK v2 Issue #1828.) Presumably S3 doesn't even interpret the object redirect value and just sends it back as the HTTP Location header value.

Assignee

Garret Wilson

Reporter

Garret Wilson

Labels

Components

Fix versions

Affects versions

Priority

Blocker
Configure