Building a Transparent Proxy in AWS VPC with Terraform and Squid

In this article I’m going to be setting up an example network and deploying a transparent proxy to it.

To make this repeatable and to show exactly how it can be deployed in AWS VPC, I am using Terraform.

Terraform is an excellent tool for describing and automating cloud infrastructure.

All of the terraform code in this project can be found here:  https://github.com/nearform/aws-proxy-pattern I have chosen to use AWS for this article and the associated terraform code, but the concept is very general and terraform code could be written for any supported IaaS platform (e.g. Google Cloud, Azure, etc.)

Let me start by defining some terms and explaining what a proxy is good for.

A transparent proxy is one where there is no configuration required on the applications using the proxy. The hosts may have some routing configuration to make it work, but applications are unaware.

This is in contrast to an explicit proxy, where applications are made aware and direct their traffic to the proxy’s IP address e.g. via a browser PAC (proxy autoconfiguration) file.

In large deployments, systems like Windows Group Policy and WPAD (PAC file discovery via DHCP or DNS) are used to configure a large number of hosts automatically.

Using a proxy can provide a degree of control over outbound web traffic.

For example, the proxy can monitor and keep audit logs of that traffic, or intervene and block traffic that is likely to pose a threat to the network.

There is also a very significant non-security related benefit in the form of caching, where repeated downloads of the same content can be served from the proxy instead.

However by intentionally making a proxy the sole means of internet access, a bottleneck and single point of failure is introduced, so typically some kind of high availability setup would be used, such as a load balancer and proxy group.

Historically, web proxies have been most useful to system administrators in corporate networks, but even a network hosting something like a SaaS product in a microservices architecture can benefit.

One of the first things a piece of malware will do (perhaps after attaining persistence on a host) will be to contact command and control infrastructure over the internet.

A proxy can make you aware of this, block the attempt and give you a chance to identify the affected machine.

I chose Squid for the proxy software. Squid is a caching web proxy and is a very popular and mature project. It has been around for more than 20 years and is easy to install and configure.

The diagram above shows the key components - a private subnet with an example host that will be trying to make requests to the internet.

The proxy receives traffic via a network default route, logging requests and filtering based on domain name.

A management host is used just to provide a means to SSH to the example host, which only has a private IP address.

Management networks are a topic for another article - but this simulates something which would normally happen over a VPN, if at all.

There is a private subnet with a default route to the proxy’s network interface. This means all internet-bound traffic will arrive at the proxy server first.

Disabling source and destination checks on the proxy instance is also necessary to allow it to receive traffic not destined for the instance’s own IP address:

Finally there are some iptables rules on the proxy instance which redirects the packets into the squid server (which is listening on ports 3129 for HTTP and 3130 for HTTPS).

From there, the fact that the proxy instance has a public IP address and the default route for the public subnet is the VPC’s internet gateway is sufficient to send the traffic out to the internet:

There were many ways I could have provisioned the EC2 instance used for the proxy.

In this  excellent article  on the AWS security blog, the proxy was built from source on the running instance.

Many package managers like apt or yum have a squid package available, though you often don’t get to run the latest version or with particular compiled-in features.

There are Chef and Puppet modules for squid to follow a more ‘declarative’ paradigm, where I specify the machine configuration I need and let the provisioning layer translate that into ‘imperative’ commands to achieve the goal.

Whichever provisioning method is used, a machine imaging solution like Packer could be used to build an AWS AMI ahead of time. This could be particularly useful to get an instance serving traffic quickly after boot, which is very useful in a load balanced proxy group setup.

In my case, I opted to use my own docker image, based on Alpine Linux and using the apk package manager to install squid. This produced an image around 15mb in size, and along with terraform, contributed to making this repeatable for others.

Just pulling a docker image and running it is a lot easier and repeatable in a variety of environments compared to the alternatives.

It also allowed me to try out some interesting solutions that NearForm are working on for vulnerability scanning of docker images, and it was good to see that the low attack surface of Alpine fed through to a clean bill of health on my image (at least for now).

The server still needed some provisioning to get to the point where it can run the squid container, and I used AWS user data scripts for this, which installs Docker, creates the squid configuration file, sets up an x509 certificate that squid needs for SSL inspection, runs the squid container and creates the aforementioned iptables rules.

Which runs  this script  on instance boot.

The squid configuration looks like this:

Most of the configuration is straightforward and sets up ports and domain whitelists. The interesting part is the ssl_bump directives.

Intercepting HTTPS traffic is basically a form of Man-in-the-Middle attack so to avoid certificate warnings and client rejections, a proxy that wants to decrypt HTTPS traffic usually works by having clients install a root certificate (owned by the proxy) in advance, and issuing new certificates signed by this root for HTTPS domains on the fly.

This is a complex topic and is a bit of a security ‘can of worms’ which you can read more about  here .

For this article, I’m doing something less controversial and just doing domain name filtering during the TLS handshake rather than decrypting traffic.

This is where the ssl_bump directives come in:

By asking Squid to peek, peek and splice respectively, I’m instructing it to open an onward TCP connection, forward the ClientHello and make a note of the SNI domain (if any), extract the domain from the ServerHello and then make a decision.

Either it will splice the rest of the connection (pass through without decryption), or it will terminate the connection based on domain name.

Squid did a great job of explaining this peek and splice method in  their online documentation .

So that was plenty of theory! Let’s see if it actually works.

I bring up the infrastructure using  terraform plan  and  terraform apply , ensuring that I have AWS credentials set up at  ~/.aws/credentials  with a profile called  default .

Once the infrastructure is created, I should see some IP addresses in the output:

Then I open an SSH connection through the management host to the example host, ensuring that I avoid the common pitfalls:

If you have done all that, you should be able to get through to the example host:

In a separate window, you can SSH to the proxy instance to monitor the logs:

The sha512.badssl.com domain is one of the test endpoints on the BadSSL service which offers up a valid SSL connection - the proxy is allowing through connections on both HTTP and HTTPS for domains in the whitelist.

The proxy should have logged these requests in the other window:

Requests to a non-whitelisted domain are prevented, as is a connection to a whitelisted domain where the certificate has expired. The access.log and cache.log record look like this:

access.log

cache.log

Don’t forget to  terraform destroy  at the end so you don’t rack up the costs in your AWS account for EC2 servers you don’t need!

Also check out our post about writing reusable terraform modules for an AWS based infrastructure .