Zed Attack Proxy in a CI Pipeline?

Testing, particularly around security, is a core part of the ethos of all NearForm development teams.

In many organisations, penetration testing can often happen just before a product first pushes to production, and periodically thereafter.

Penetration testing is performed by external teams and is focused on finding vulnerabilities that may exist within the infrastructure or code base. It can be a slow and expensive process.

This brings up two questions:

Open Web Application Security Project - OWASP  is the gold standard of tools, advice and security best practices.

We will focus on using  ZED Attack Proxy  - ZAP - and show how to integrate it into our Continuous Integration (CI) pipeline. The goal is to automate ZAP with as little configuration as possible.

Jenkins users (and Jira ones also) may benefit from the ready-to-use Jenkins  plugin  for your CI needs.

If the Jenkins plugin is not an option, ZAP has  Docker support  and a  wiki  full of interesting and useful information and instructions.

Note: The ZAP project also has a desktop multi-platform  tool  that can help test and prepare for the CI integration (or manual analyses) and also helps to get a feeling for the tools capabilities and configuration options.

Although ZAP excels at crawling and scanning frontend (SPA) applications, in this example we integrate ZAP with  Udaru , an open source access manager for Node.js written by NearForm.

Udaru uses  Swagger  to document the endpoints and this is very useful when testing it with ZAP.

ZAP has two scripts which can help us accomplish this goal.

Note: ZAP docker images are quite big and download can be slow. You may notice there is a smaller and more CI optimized image called owasp/zap2docker-bare, however avoid it for now as it does not have necessary scripts to perform the scans needed.

The following scripts help to perform a  Baseline Scan  aand an  API scan .

First, pull the weekly docker image

Run the baseline scan against a local http server, for example running on 8000 port

Run the API scan against the local server running on 8080 port based on the  Swagger  definition of the API.

Note: The command  $(ifconfig en0 | grep "inet " | cut -d " " -f2)  is used to get the correct IP address to be able to access the Site/API outside the docker container.

If you are on a MAC OSX this may not work as Docker has some difference in networking. Then you can use  docker.for.mac.localhost  for the host part of the  -t  argument.

Neither the baseline-scan.conf or the api-scan.conf files exist - the first time you run the commands use  -g instead of  -c , which will produce the configuration files with levels of alerts set to  warn .

With these files generated, you can set which tests should fail/ignore/warn as you run the scripts.

The first few lines of the file describe the contents and their configuration.

Both of these scripts will test a front-end or back-end application. However, problems can arise with authenticating a back-end API request as this is a common case for testing REST APIs; this is usually the  Authorization  header.

This part is described in the  ZAP blog  and basically boils down to adding some extra configuration for the ZAP’s replacer add-on.

Example using ZAP API scan with Authorization header:

This technique can be used to add additional headers if needed.

The beauty of being able to pass a Swagger json to ZAP is that it significantly helps with things like identifying all endpoints to be tested, saves from building elaborate contexts for testing and makes it extremely easy to use thanks to the OpenAPI standardization (Swagger).

If there is no similar setup on the project that needs testing then this considerably adds to the time needed to do the configuration of a  context  needed to execute the scans and get some meaningful results.

If we think of ZAP pen-testing as a step in the CI execution pipeline (alongside checks, test runs and deploys) then baseline scanning fits here perfectly.

It is fast (length controllable with  -d /  --duration  options and defaults to 1 minute) safe and can provide basic insight during regular CI execution.

API scan (also referred to as  attack  in the documentation), is very comprehensive (with out of the box generated configuration), but it is CPU intensive and, depending on your API size (number of endpoints to test), very lengthy.

Also you need to factor in the fact that a big Docker image for ZAP needs to be downloaded (it is recommended to use the weekly one).

This does not really make it a good candidate for a step in the regular CI pipeline, but instead it is more suited to a dedicated CI job that runs nightly against a development or staging environment.

The scan is not considered  safe  so it would be prudent to run it on environments that can afford this kind of pressure (usually pre-production ones).

Reports generated from ZAP usually need manual overview until the issues detected are fixed.

It is prudent to do a manual execution of the scans and fix the initial issues prior to putting in any CI system.

ZAP can serve as  man-in-the-middle  .

This is very useful in the context of end-to-end tests or even integration tests that a project might already have.

Those tests might expose some vulnerabilities that cannot be crawled or found otherwise with baseline ZAP scan.

To make this work, we need to make sure ZAP is running as a proxy server, with network traffic from the tests runs through it and using a headless mode that CI systems can work with.

ZAP can be run via docker in headless mode on localhost like this:

In tests that have the capability to use a proxy server, set the proxy option and let the test run as normal.

ZAP will analyse the requests in its separate threads without affecting the tests and communication with them.

So far so good; we can put all of this in the CI pipeline since it is all headless and automated.

The final phase is to actually get some data or alerts or even reports from the ZAP based on the test run.

This is the hard part, there is nothing that works out of the box and our data is stuck in the ZAP docker container!

There are two approaches here to access the report data:

First, use the  command line options  and use the  session  and  last_report  flags to create a usable report, with one extra docker run like:

Unfortunately, this means that it is necessary to preserve session information outside of the container and reuse it.

This is not very CI friendly, but doable if the end goal is the report.

Session data should also be preserved across CI steps and subsequent runs which needs manual setup and testing on most modern cloud CI providers.

A second approach is to use the  ZAP API  to get some data outside of ZAP’s database, which means the container needs to be running until data has been extracted.

We did not fully try these options out, but to help get some actionable result, we could use a Node.js client on NPM called  zaproxy .

It’s a generated API client so some additional development work would be needed to get through to the report data.

There seems to be one more NPM option and it’s a Grunt runner called  grunt-zaproxy .

This looks a bit more promising based on the example in the NPM readme. It should allow you to run scans and get a report, via Grunt. However it requires a bit of an update as it does not support latest Node.js versions.

It’s still worth noting that setting ZAP as proxy for tests is a valid approach, just maybe not for CI.

After the test runs, the results could be gathered and reviewed manually using the ZAP desktop application.

With all these known pros and cons how did we integrate ZAP in our CI pipeline on  Udaru ?

The short answer is that we did not. Not in the fully automated CI way.

We did create some  npm scripts  to run the baseline and API scans and we store the reports within our project  documentation .

Anyone working on the project is free to run the scans to see if their work has disclosed a security vulnerability.

The flexibility of npm scripts allows us to easily integrate this into a CI pipeline, should we choose to perform a round of penetration testing (for example as a pre-release step).

The npm scripts basically start the Udaru server on a specified port, run the docker commands to initiate the scans and store the reports.

For more in depth information on the Udaru implementation, please see this  pull request .

OWASP ZAP is a powerful tool in the battlefield of secure web applications.

The toolset developed around it is powerful, modern and is the cornerstone of moving to a fully automated penetration testing state in the CI Pipeline.

The Jenkins plugin is highly recommended for baseline scans.

For API scans, be prepared to invest a bit more into tooling environments, issue/alert analysis and fixes as this is something that is hard to fully automate.

Having peace of mind around the security of the product/project is definitely worth the investment of having this nice tool in your security belt.

For more information about our open source project Udaru see our article on dynamic intrusion detection for authorisation systems like Udaru .

Image: Jonatan Pie