Elastic Beanstalk applications using two ports and Websockets

This blog post describes how to set up and deploy an AWS Elastic Beanstalk application written in Node.js that listens to both regular HTTP requests and Websocket connections.

TL;DR: Download the sample application, and deploy it into an EB environment created using eb create alb-test-app-dev1 --elb-type application.

By default, applications in Elastic Beanstalk only listen to one port, and that is reflected in settings of the Nginx proxy, the Elastic Load Balancer, and the ELB listeners. We must change Elastic Beanstalk’s default settings to make the dual port setup work, which is done with .ebextensions. Also, Websockets don’t seem to work too well with a “Classic” ELB. An “Application Load Balancer” must be used instead.

A test server

To test the deployment, I created a simple test server by copypasting code from a couple of Node.js tutorials:

/**
 * Minimal test server for debugging simultaneous use of regular HTTP requests
 * and websockets on Elastic Beanstalk.
 *
 * The regular server responds with a "200 OK" to requests at /healthcheck,
 * and with a JSON containing the current timestamp at /time. The socket
 * server responds with a JSON containing the current timestamp.
 *
 * Application regular server port: 8081
 * Application socket server port: 8088
 *
 * Traffic routes:
 * Regular requests: ELB:80   -> Nginx:8080 -> NodeJS:8081
 * Socket requests:  ELB:8082 -> Nginx:8082 -> NodeJS:8088
 *
 * See http://blog.ampli.fi/elastic-beanstalk-applications-using-two-ports-websockets
 *
 * Note the .ebextensions directory, which your OS may hide by default.
 *
 */

"use strict";

var http = require('http')
var port = 8081
var webSocketsServerPort = 8088;
var webSocketServer = require('websocket').server;


// Regular server
const requestHandler = (request, response) => {
    var date = new Date();
    var datestring = date.toISOString().substr(0, 19);

    if (request.url == "/time") {
        response.statusCode = 200;
        var string = '{"time":"' + datestring + '"}';
    }
    else if (request.url == "/healthcheck") {
        response.statusCode = 200;
        var string = 'OK';
    }
    else {
        var string = 'ERROR';
        response.statusCode = 400;
    }

    response.end(string)
}

var myServer = http.createServer(requestHandler)

myServer.listen(port, (err) => {
    if (err) {
        return console.log('something bad happened', err)
    }

    console.log(`server is listening on ${port}`)
})


// Websocket server
var server = http.createServer(function(request, response) {});

server.listen(webSocketsServerPort, function() {
    console.log("WS server is listening on port " + webSocketsServerPort);
});

var wsServer = new webSocketServer({
    httpServer: server
});

wsServer.on('request', function(request) {
    var connection = request.accept(null, request.origin);
    var date = new Date();
    var datestring = date.toISOString().substr(0, 19);

    connection.on('message', function(message) {
        console.log("message");
        connection.sendUTF(
            JSON.stringify(
                { time: datestring }
            )
        );
    });
});

The configuration

Custom configuration on Elastic Beanstalk is done with “.ebextensions” configuration files. These files must be saved in your app directory, in a subdirectory called “.ebextensions”, and named with a “.config” extension.

Nginx

Elastic Beanstalk installs a proxy server on the EC2 instance, and by default that proxy server is Nginx. We need to customize the Nginx configuration to make it listen to both ports we intend to use.

This set of commands causes Elastic Beanstalk to replace the default Nginx configuration on the EC2 instances it creates with our custom configuration:

files:
  /etc/nginx/conf.d/proxy.conf:
    mode: "000644"
    owner: root
    group: root
    content: |
      upstream nodejs {
        server 127.0.0.1:8081;
        keepalive 256;
      }
      upstream nodejssocket {
        server 127.0.0.1:8088;
        keepalive 256;
      }

      server {
        listen 8082;

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
            set $year $1;
            set $month $2;
            set $day $3;
            set $hour $4;
        }
        access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
        access_log  /var/log/nginx/access.log  main;

        location /healthcheck {
          # Healthcheck checks the node process from the other port, so we can
          # always return a 200 from the socket port.
          return 200;
        }

        location / {
            proxy_pass  http://nodejssocket;
            proxy_set_header   Connection "";
            proxy_http_version 1.1;
            proxy_set_header        Host            $host;
            proxy_set_header        X-Real-IP       $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

        gzip on;
        gzip_comp_level 4;
        gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

      }

      server {
        listen 8080;

        if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
            set $year $1;
            set $month $2;
            set $day $3;
            set $hour $4;
        }
        access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd;
        access_log  /var/log/nginx/access.log  main;

        location / {
            proxy_pass  http://nodejs;
            proxy_set_header   Connection "";
            proxy_http_version 1.1;
            proxy_set_header        Host            $host;
            proxy_set_header        X-Real-IP       $remote_addr;
            proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header Upgrade $http_upgrade;
            proxy_set_header Connection "upgrade";
        }

        gzip on;
        gzip_comp_level 4;
        gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        location /static {
            alias /var/app/current/static;
        }

      }

  /opt/elasticbeanstalk/hooks/configdeploy/post/99_kill_default_nginx.sh:
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/bash -xe
      rm -f /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf
      service nginx stop
      service nginx start

container_commands:
 removeconfig:
    command: "rm -f /tmp/deployment/config/#etc#nginx#conf.d#00_elastic_beanstalk_proxy.conf /etc/nginx/conf.d/00_elastic_beanstalk_proxy.conf"

Elastic Load Balancer

The Elastic Load Balancer also needs to be configured to pass traffic from our public facing ports to the ports that the Nginx proxy is listening to. This is accomplished with another .ebextensions file:

option_settings:
  aws:elbv2:listener:80:
    DefaultProcess: regular
    ListenerEnabled: 'true'
    Protocol: HTTP
  aws:elasticbeanstalk:environment:process:regular:
    HealthCheckPath: /healthcheck
    Port: '8080'
    Protocol: HTTP
  aws:elbv2:listener:8082:
    DefaultProcess: socket
    ListenerEnabled: 'true'
  aws:elasticbeanstalk:environment:process:socket:
    HealthCheckPath: /healthcheck
    Port: '8082'
    Protocol: HTTP

Deployment

The sample application is available in this tar archive. To use an Application Load Balancer, the Elastic Beanstalk environment must be created with the EB CLI. It is not possible, at the time of this writing, to create ELBs other than “classic” ones using the web console.

To install and deploy the test application, download the tar archive, open a terminal, and navigate to the directory you’ve saved the archive to. Then:


$ tar -xzf alb-test-app.tgz
$ cd alb-test-app
$ eb init
$ eb create alb-test-app-dev1 --elb-type application


Testing

To test your deployed application, start by finding your new environment’s hostname in the AWS web console.

Then, to test the regular requests, simply point your browser or cURL to http://your-eb-environment-hostname-here/time . You should get a JSON response with the current UTC timestamp.

To test the socket connection, first edit “index.html” from the tar archive, and set your environment hostname in the file. Then open the file with File->Open or similar functionality in your browser.