Nginx as origin for S3 with authentication, with CDN on top

We recently had a small problem with our content delivery system. The setup is as follows:

– AWS S3 bucket that requires authentication
– EC2 instance, running Nginx with ngx_aws_auth
– ELB load balancer in front of the EC2 instance
– CDN configured to use the ELB as the origin

The problem was that the content could be loaded directly from the Nginx, or from the load balancer, but when trying to load the content through the CDN, we’d get an AWS error message about a mismatching signature:

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>
The request signature we calculated does not match the signature you provided. Check your key and signing method.
</Message>
<AWSAccessKeyId>REDACTED</AWSAccessKeyId>
<StringToSign>
GET x-amz-date:Tue, 05 Apr 2016 12:15:35 GMT /REDACTED_URL_TO_ASSET
</StringToSign>
<SignatureProvided>xHLCuT9r7sAUbONvHaDeWDigCGs=</SignatureProvided>
<StringToSignBytes>
REDACTED
</StringToSignBytes>
<RequestId>AACF04EB41AB61DC</RequestId>
<HostId>
REDACTED
</HostId>
</Error>

The culprit ended up being the “Date”-header added to the request by the CDN. This caused ngx_aws_auth to use an incorrect date value in calculating the AWS signature. Simply unsetting the header with:

proxy_set_header Date ""; # Does not work

..does not work, since ngx_aws_auth uses the incoming header for the signature calculation.

The solution was not particularly pretty, but it works:

server {
    root /usr/share/nginx/html;
    listen 81;
    server_name "127.0.0.1:81";

    location / {
        proxy_set_header x-amz-cf-id "";
        proxy_pass "https://S3_BUCKET_NAME.s3.amazonaws.com";
        aws_access_key 'REDACTED';
        aws_secret_key 'REDACTED';
        s3_bucket S3_BUCKET_NAME;
        proxy_set_header Authorization $s3_auth_token;
        proxy_set_header x-amz-date $aws_date;
        resolver               8.8.8.8 valid=300s;
        resolver_timeout       10s;
    }
}

server {
    root /usr/share/nginx/html;
    listen 80;
    server_name  "SERVER_HOSTNAME";
    server_name  "LOAD_BALANCER_HOSTNAME";
    server_name  "LOAD_BALANCER_ALIAS";
    server_name  "CDN_HOSTNAME";

    location / {
        proxy_set_header Date "";
        proxy_pass "http://127.0.0.1:81";
    }
}

Essentially, Nginx first receives the request and strips the Date-header. It then proxy_passes the request to itself, in another port, where ngx_aws_auth now can use the correct date value to calculate the signature.

Another option would’ve been to strip the Date-header at the CDN end, before the request is passed to the LB, but we decided we’d rather have a solution that works with any stock CDN setup, in case we change CDN service providers.