Securing Docker Containers on AWS
By Robb Dempsey

In 2015 the Docker team set out to consolidate all the information around best practices for securing docker containers. You can find references to the CIS Docker Community Edition Benchmark version 1.1.0 and Docker’s own white paper Introduction to Docker Security on the topic here. The links above get you more directly to the assets. Just an FYI, the CIS benchmark will cost you an email address to access the download.

On most projects at nearForm we are deploying our solutions within Docker containers. There are tasks that are repeated on each project to secure and harden off those deployments and we built this packer template to produce a quick and easy way for you to spin up an AWS AMI that passes the Docker-Bench-Security script. The Docker-Bench-Security repo is a work product of the above mentioned consolidation efforts by the Docker team.

To accomplish the building of this AMI we use Packer, which is an easy way to automate the creation of your images. It supports multiple providers (i.e. AWS, Digital Ocean) and allows you to document in a repo how your images were built and modifications that were done to them over time.

The work to get this AMI passing the Docker-Bench-Security was not a small task, but a critical task of any development organization. Minimizing the attack surface of an application we deploy into the wild will always return on investment. Whether that return comes from keeping an active attacker from escalating privileges and damaging our craft or simply letting you sleep at night knowing that you’ve done everything you can to secure your environments. Give it a spin and feel free to raise issues or PRs if you find ways to improve upon the work.

TL;DR

Just give me an AMI I can work with

The Work

To get things started we are using Packer to create an ebs-backed AMI with the underlying OS being Ubuntu 17.0.4. We are also installing Docker CE for Ubuntu with the supporting packages suggested by the Docker Team. Out of the box, our base Ubuntu AMI is failing the following tests in the Docker-Bench-Security. We’ll go through each and document how we passed each test.

[WARN] 1.1  - Ensure a separate partition for containers has been created
[NOTE] 1.2  - Ensure the container host has been Hardened
[WARN] 1.5  - Ensure auditing is configured for the Docker daemon
[WARN] 1.6  - Ensure auditing is configured for Docker files and directories - /var/lib/docker
[WARN] 1.7  - Ensure auditing is configured for Docker files and directories - /etc/docker
[WARN] 1.8  - Ensure auditing is configured for Docker files and directories - docker.service
[WARN] 1.9  - Ensure auditing is configured for Docker files and directories - docker.socket
[WARN] 1.10 - Ensure auditing is configured for Docker files and directories - /etc/default/docker
[WARN] 1.12 - Ensure auditing is configured for Docker files and directories - /usr/bin/docker-containerd
[WARN] 1.13 - Ensure auditing is configured for Docker files and directories - /usr/bin/docker-runc
[WARN] 2.1  - Ensure network traffic is restricted between containers on the default bridge
[WARN] 2.5  - Ensure aufs storage driver is not used
[WARN] 2.8  - Enable user namespace support
[WARN] 2.11 - Ensure that authorization for Docker client commands is enabled
[WARN] 2.12 - Ensure centralized and remote logging is configured
[WARN] 2.13 - Ensure operations on legacy registry (v1) are Disabled
[WARN] 2.14 - Ensure live restore is Enabled
[WARN] 2.15 - Ensure Userland Proxy is Disabled
[WARN] 4.1  - Ensure a user for the container has been created
[WARN] 4.5  - Ensure Content trust for Docker is Enabled
[WARN] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image
[INFO] 4.7  - Ensure update instructions are not use alone in the Dockerfile
[INFO]      \\\\\\\* Update instruction found: [node:8]
[INFO] 4.9  - Ensure COPY is used instead of ADD in Dockerfile
[INFO]      \\\\\\\* ADD instruction found: [node:8]

[INFO] 5 - Container Runtime
[WARN] 5.2  - Ensure SELinux security options are set, if applicable
[WARN] 5.10 - Ensure memory usage for container is limited
[WARN] 5.11 - Ensure CPU priority is set appropriately on the container
[WARN] 5.14 - Ensure 'on-failure' container restart policy is set to '5'
[WARN] 5.25 - Ensure the container is restricted from acquiring additional privileges
[WARN] 5.28 - Ensure PIDs cgroup limit is used
[INFO] 5.29 - Ensure Docker's default bridge docker0 is not used
1.1 - Ensure a separate partition for containers has been created

To pass 1.1 we had to do a few tasks that included modifying our underlying packer template and then scripting a few changes for the AMI build. The volume that is defined in the ami_block_device_mappings will be setup and used as the mount for all things Docker. (i.e. images, containers…). Few notes on this block device: - its currently set for 1GB in size, which is not ideal for anything but testing. (i.e. "volume_size": 1) - "device_name" : "/dev/sdf" gets renamed hence the different name in our ./scripts/volumn.sh From my research this appeared to be because of the specific AMI we are using, but to dig deeper on the warning from AWS and reasoning behind them read more

The actual work of mounting and configuring the disk for operation is completed in ./scripts/volumn.sh.

1.2 Ensure the container host has been Hardened

The underlying host of the AMI we create IS NOT hardened and the Docker-Bench-Security DOES NOT verify the host’s settings. You could easily switch the AMI ID that we are using in our example with one from CIS Benchmarks and start fresh with a hardened host and a secure docker implementation.

1.5 - 1.13 Auditing

These tests are passed with the installation and configuration of auditd, which is all done within auditd.sh. With our settings we are watching for writes and changes to the attributes of each item. [more info] Each test is listed above the setting for reference.

2.1 - 2.15 Docker daemon configuration

We accomplish passing these test with the creation of /etc/docker/daemon.json from ./files/daemon.json.

2.1 - Ensure network traffic is restricted between containers on the default bridge

This warning seems to contradict 5.29, which we implment a solution for alter on in this post.

2.5 - Ensure aufs storage driver is not used

Since we created our new parition with ext4 we are able to use the preferred storage-driver which is overlay2. - Docker CE on Ubuntu storage drivers - supported backing filesystems

    "storage-driver": "overlay2"
2.8 - Enable user namespace support

“The best way to prevent privilege-escalation attacks from within a container is to configure your container’s applications to run as unprivileged users.” source Using default means our user will be dockremap versus something that may make more sense in your environment.

    "userns-remap": "default"
2.11 - Ensure that authorization for Docker client commands is enabled

We do not implement changes (i.e we still get a WARN result) for this since it requires generation of keys and certificates that will be unique to your environment. You will cand read more on the details of how to configure the docker daemon and how to protect the docker socket

2.12 - Ensure centralized and remote logging is configured

We are writing everything to syslog, but you could easily configure aws-logs or another end point that you actively work with for logging.

    "log-driver": "syslog", 
    "log-opts": { "syslog-address": "udp://1.2.3.4:1111" }
2.13 - Ensure operations on legacy registry (v1) are Disabled

This is possibly a legacy configuation since support “for the v1 protocol to the public registry was removed in 1.13”.

    "disable-legacy-registry": true`
2.14 - Ensure live restore is Enabled

This setting allows your containers to continue running when there are issues that cause the docker daemon to be down. more info

    "live-restore": true
2.15 - Ensure Userland Proxy is Disabled
    "userland-proxy": false
4.1 - Ensure a user for the container has been created

This warning is looking for you to run your processes within the container as a seperate users. With userns-remap enabled for our containers this seems a bit excessive since the container is restricted to a lower privileged user. This could be a wrong assumption, but restricting users that run your process can inside the container could have other adverse effects. This should be evaluated and configured based on your applications needs.

This test was passed by modifying the Dockerfile to include adding a new user and group. You can confirm your settings are in place with the following.

$ docker inspect --format 'User={ {.Config.User} }' flamboyant_yonath
User=app:app

$ docker exec -it flamboyant_yonath /bin/bash

app@188099606d58:~/my-project$
4.5 - Ensure Content trust for Docker is Enabled

We take care of this by setting the environment variable DOCKER_CONTENT_TRUST for all users in ./scripts/docker-setup.sh . Note that if you build an image locally there are known issues with Docker Content Trust disallowing your action with an authorization failure (i.e. 401). You can use the --disable-content-trust flag to bypass the setting we created in our docker setup script

$ docker run --disable-content-trust example:1.0.0

Docker Content Trust has no concept of trusting something built locally #25852

4.6 - Ensure HEALTHCHECK instructions have been added to the container image

Adding the following lines to the Dockerfile gets us past this test. This checks the url every 20s and fails after 3s.

HEALTHCHECK --interval=20s --timeout=3s \
  CMD curl -f http://localhost:8000/ || exit 1
example:1.5.1   Up 2 seconds (healthy)    8000/tcp    fervent_kilby
example:1.5.0   Up 7 minutes (unhealthy)  8000/tcp    wonderful_tesla
4.7, 4.9 Commands UPDATE && ADD

It appears the node:8 image has a few items that docker-bench-security does not like. Running docker history on the image we build with our example will show that those commands are getting into our images based on FROM node:8

5.10, 5.11 Memory && CPU

By default, a docker container has no resource constraints. You can set specific runtime flags for both Memory and CPU based on your applications needs. more info I had no success with the proposed flags and passing the tests that docker-bench-security runs for them.

5.14 - Ensure ‘on-failure’ container restart policy is set to ‘5’

This can be accomplished by add the flag --restart to your docker run command as below:

docker run -d --disable-content-trust --restart on-failure:5  example:1.5.1
5.25 - Ensure the container is restricted from acquiring additional privileges

This can be accomplished by adding the flag --security-opt to your docker run command

docker run -d –disable-content-trust –restart on-failure:5 –security-opt="no-new-privileges:true" example:1.5.1
5.28 - Ensure PIDs cgroup limit is used

I’ve yet to find a solution that works and passes the test. The runtime flag --pids-limit does not appear to have any effect on the configuration of a container.

5.29 - Ensure Docker’s default bridge docker0 is not used

If you are interested in using a bridge other than docker0 you can Build your own bridge

Requirements

Setup

  1. Setup packer
  2. $ git clone git@github.com:nearform/devops.git
  3. Create a file named variables.json in the root of the example repo from #2. Add your access and secret keys. If you want to use a different region in AWS change the aws_region and find replace the ubuntu_source_ami with the AMI ID that has an instance type of hvm-ssd from Ubuntu. Your File should look something like this:
{
  "aws_secret_key": "< INSERT YOUR SECRET KEY >",
  "aws_access_key": "< INSERT YOUR ACCESS KEY >",
  "aws_region": "us-east-1",
  "ubuntu_source_ami": "ami-cb1d41b0"
}

Build the AMI After executing the command below you should see an AMI in the AWS Console

    $ packer build -var-file=variables.json base-image-us-east-template.json

Our example template uses packer’s amazon-ebs builder. We use a builder and two provisioners in our template to accomplish the following:

  1. Create /etc/docker/daemon.json
  2. Create and mount the volume that will be used to store our Docker assets
  3. Install Docker CE
  4. Clone Docker-Bench-Security
  5. Install and configure auditd

Run an Instance of Your AMI

Once packer has finished building your AMI, you can log into the AWS Console and launch an instance from the AMI. You will only need to configure network related items (i.e. subnet) and the rest you should be able to take the defaults. Once your instance is up and running ssh into it and you should find docker-bench-security in the home directory of the ubuntu user. Do the following to run and verify the AMI passes the test we’ve documented above.

  1. $ cd docker-bench-security
  2. $ sudo sh ./docker-bench-security.sh

note: sudo is used in step 2 due to checks against files and directories that require elevated privileges.

Run a Container in your Instance

If you would like to build a docker image and run a container based on that image you can include ./files/app/ in your AMI build, which will give you a simple hapi app to run the Docker-Bench-Security tests against.

To include these files add the following to the provisioners section of the template

    {
      "type": "shell",
      "inline": ["mkdir ./app/"]
    },
    {
      "type": "file",
      "source": "./files/app/",
      "destination": "./app/"
    }
Subscribe to our monthly newsletter!
join the discussion