Container security is (or should be) a concern to anyone running software on Docker Containers. Gone are the days when running random Images found on the internet was common place. Security guides for Containers are common now: examples from Microsoft and others can be found easily online.
The two leading Container Orchestrators also offer their own security guides: Kubernetes Security Best Practices and Docker security.
Container Image Origin
One of the single biggest factors in Container security is determined by the origin of container Images:
- It is recommended to run your own private Registry to distribute Images
- It is recommended to scan these Images against known vulnerabilities.
Running a private Registry is easy these days (Azure Container Registry for instance).
I will concentrate on the scanning of Images in the remainder of this post and show how to look for common vulnerabilities using Core OS Clair. Clair is probably the most advanced non commercial scanning solution for Containers at the moment, though it requires some elbow grease to run this way. It’s important to note that the GUI and Enterprise features are not free and are sold under the Quay brand.
As security scanning is recommended as part of the build process through your favorite CI/CD pipeline, we will see how to configure Visual Studio Team Services (VSTS) to leverage Clair.
Installing CoreOS Clair
First we are going to run Clair in minikube for the sake of experimenting. I have used Fedora 26 and Ubuntu. I will assume you have minikube, kubectl and docker installed (follow the respective links to install each piece of software) and that you have initiated a minikube cluster by issuing the “minikube start” command.
Clair is distributed through a docker image or you can also compile it yourself by cloning the following Github repository: https://github.com/coreos/clair.
In any case, we will run the following commands to clone the repository, and make sure we are on the release 2.0 branch to benefit from the latest features (tested on Fedora 26):
~/Documents/github|⇒ git clone https://github.com/coreos/clair Cloning into 'clair'... remote: Counting objects: 8445, done. remote: Compressing objects: 100% (5/5), done. remote: Total 8445 (delta 0), reused 2 (delta 0), pack-reused 8440 Receiving objects: 100% (8445/8445), 20.25 MiB | 2.96 MiB/s, done. Resolving deltas: 100% (3206/3206), done. Checking out files: 100% (2630/2630), done. rafb@overdrive:~/Documents/github|⇒ cd clair rafb@overdrive:~/Documents/github/clair|master ⇒ git fetch rafb@overdrive:~/Documents/github/clair|master ⇒ git checkout -b release-2.0 origin/release-2.0 Branch release-2.0 set up to track remote branch release-2.0 from origin. Switched to a new branch 'release-2.0'
The Clair repository comes with a Kubernetes deployment found in the contrib/k8s subdirectory as shown below. It’s the only thing we are after in the repository as we will run the Container Image distributed by Quay:
rafb@overdrive:~/Documents/github/clair|release-2.0 ⇒ ls -l contrib/k8s total 8 -rw-r--r-- 1 rafb staff 1443 Aug 15 14:18 clair-kubernetes.yaml -rw-r--r-- 1 rafb staff 2781 Aug 15 14:18 config.yaml
We will modify these two files slightly to run Clair version 2.0 (for some reason the github master branch carries an older version of the configuration file syntax – as highlighted in this github issue).
In the config.yaml, we will change the postgresql source from:
source: postgres://postgres:password@postgres:5432/postgres?sslmode=disable
to
source: host=postgres port=5432 user=postgres password=password sslmode=disable
In config.yaml, we will change the version of the Clair image from latest to 2.0.1:
From:
image: quay.io/coreos/clair
to
image: quay.io/coreos/clair:v2.0.1
Once these changes have been made, we can deploy Clair to our minikube cluster by running those two commands back to back:
kubectl create secret generic clairsecret --from-file=./config.yaml kubectl create -f clair-kubernetes.yaml
By looking at the startup logs for Clair, we can see it fetches a vulnerability list at startup time:
[rbigeard@flanger ~]$ kubectl get pods NAME READY STATUS RESTARTS AGE
clair-postgres-l3vmn 1/1 Running 1 7d
clair-snmp2 1/1 Running 4 7d
[rbigeard@flanger ~]$ kubectl logs clair-snmp2
...
{"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:213","Time":"2017-08-14 06:37:33.069829"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:88","Time":"2017-08-14 06:37:33.069960","package":"Ubuntu"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2017-08-14 06:37:33.092898","package":"Oracle Linux"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2017-08-14 06:37:33.094731","package":"RHEL"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2017-08-14 06:37:33.097375","package":"Alpine"}
Scanning Images through Clair Integrations
Clair is just a backend and we therefore need a frontend to “feed” Images to it. There are a number of frontends listed on this page. They range from full Enterprise-ready GUI frontends to simple command line utilities.
I have chosen to use “klar” for this post. It is a simple command line tool that can be easily integrated into a CI/CD pipeline (more on this in the next section). To install klar, you can compile it yourself or download a release.
Once installed, it’s very easy to use and parameters are passed using environment variables. In the following example, CLAIR_OUTPUT is set to “High” so that we only see the most dangerous vulnerabilities. CLAIR_ADDRESS is the address of Clair running on my minikube cluster.
Note that since I am pulling an image from an Azure Container Registry instance and I have specified a DOCKER_USER and DOCKER_PASSWORD variable in my environment.
# CLAIR_OUTPUT=High CLAIR_ADDR=http://192.168.99.100:30060 \ klar-1.4.1-linux-amd64 romainregistry1.azurecr.io/samples/nginx Analysing 3 layers Found 26 vulnerabilities CVE-2017-8804: [High] The xdr_bytes and xdr_string functions in the GNU C Library (aka glibc or libc6) 2.25 mishandle failures of buffer deserialization, which allows remote attackers to cause a denial of service (virtual memory allocation, or memory consumption if an overcommit setting is not used) via a crafted UDP packet to port 111, a related issue to CVE-2017-8779. https://security-tracker.debian.org/tracker/CVE-2017-8804 ----------------------------------------- CVE-2017-10685: [High] In ncurses 6.0, there is a format string vulnerability in the fmt_entry function. A crafted input will lead to a remote arbitrary code execution attack. https://security-tracker.debian.org/tracker/CVE-2017-10685 ----------------------------------------- CVE-2017-10684: [High] In ncurses 6.0, there is a stack-based buffer overflow in the fmt_entry function. A crafted input will lead to a remote arbitrary code execution attack. https://security-tracker.debian.org/tracker/CVE-2017-10684 ----------------------------------------- CVE-2016-2779: [High] runuser in util-linux allows local users to escape to the parent session via a crafted TIOCSTI ioctl call, which pushes characters to the terminal's input buffer. https://security-tracker.debian.org/tracker/CVE-2016-2779 ----------------------------------------- Unknown: 2 Negligible: 15 Low: 1 Medium: 4 High: 4
So Clair is showing us the four “High” level common vulnerabilities found in the nginx image that I pulled from Docker Hub. At times of writing, this is consistent with the details listed on docker hub. It’s not necessarily a deal breaker as those vulnerabilities are only potentially exploitable by local users on the Container host which mean we would need to protect the VMs that are running Containers well!
Automating the Scanning of Images in Azure using a CI/CD pipeline
As a proof-of-concept, I created a “vulnerability scanning” Task in a build pipeline in VSTS.
Conceptually, the chain is as follows:
I created an Ubuntu Linux VM and built my own VSTS agent following published instructions after which I installed klar.
I then built a Kubernetes cluster in Azure Container Service (ACS) (see my previous post on the subject which includes a script to automate the deployment of Kubernetes on ACS), and deployed Clair to it, as shown in the previous section.
Little gotcha here: my Linux VSTS agent and my Kubernetes cluster in ACS ran in two different VNets so I had to enable VNet peering between them.
Once those elements are in place, we can create a git repo with a shell script that calls klar and a build process in VSTS with a task that will execute the script in question:
The content of scan.sh is very simple (This would have to be improved for a production environment obviously, but you get the idea):
#!/bin/bash CLAIR_ADDR=http://X.X.X.X:30060 klar Ubuntu
Once we run this task in VSTS, we get the list of vulnerabilities in the output which can be used to “break” the build based on certain vulnerabilities being discovered.
Summary
Hopefully you have picked up some ideas around how you can ensure Container Images you run in your environments are secure, or at least you know what potential issues you are having to mitigate, and that a build task similar to the one described here could very well be part of a broader build process you use to build Containers Images from scratch.