This article will dive into different aspects of container security and walk through creating a secure container image. To do so, we will cover the following topics:
- Author secure
Dockerfilesaccording to best practices checked by dockle
- Scanning and eliminating vulnerabilities in the inner- and outer-loop using container vulnerability scanning
- Part 1: Code, Container, Cluster, Cloud: The 4C’s of Cloud-Native Security
- Part 2: Container Security: The 4C’s of Cloud-Native Security ⬅
- Part 3 Cluster Security: The 4C’s of Cloud-Native Security
- Part 4: Code Security: The 4C’s of Cloud-Native Security
- Part 5: Cloud Security: The 4C’s of Cloud-Native Security
That said, let’s briefly talk about the example application. We will containerize a simple web API build with Go for demonstration purposes. The API itself consists of three exposed HTTP endpoints:
POST: /echo– Responds the entire payload sent to the endpoint as the body of an HTTP 200 response
GET: /healthz/ready– Responds with an HTTP 200
GET: /healthz/alive– Responds with an HTTP 200
Nothing fancy because it does not matter what happens inside the app as we’re dealing with container security for the sake of this article. Although I use Go here, you can use all concepts and tools no matter which programming language or framework you are using. You can find the source code of the application in the cloud-native-security repository (See container-security subfolder).
Containerizing the Sample application
Containerizing existing applications is super easy with Visual Studio Code (VSCode) and its Docker extension. If you have not yet installed the Docker extension, go ahead and install it now! To install the Docker extension for VSCode, you can search for Docker (published by Microsoft itself) using the extension browser (
CMD-or-Ctrl+Shift+X) you execute
code --install-extension ms-azuretools.vscode-docker in the terminal of your choice.
If you haven’t cloned the sample repository, it is now the time to do that:
# move to the directory where you keep your repos cd repos # clone the repo git clone firstname.lastname@example.org:thinktecture-labs/cloud-native-security.git # move into the cloned repo cd cloud-native-security/container-security # fire up VSCode code .
The Docker extension adds several commands to your VSCode installation. Execute the Docker: Add Docker Files to Workspace command now. This will guide you through the process of generating a tailored
Dockerfile. When asked by VSCode, provide the following answers:
- Application Platform: Go
- Application Port: 3000
- Include optional Docker Compose files? No
By providing the answers above, VSCode can generate the following
Dockerfile for you:
#build stage FROM golang:alpine AS builder RUN apk add --no-cache git WORKDIR /go/src/app COPY . . RUN go get -d -v ./... RUN go build -o /go/bin/app -v ./... #final stage FROM alpine:latest RUN apk --no-cache add ca-certificates COPY --from=builder /go/bin/app /app ENTRYPOINT /app LABEL Name=containersecurity Version=0.0.1 EXPOSE 3000
Check Dockerfile with dockle
container-security:0.0.1 image in place (you can check the list of all your container images using
docker images), we can check if the
Dockerfile meets best practices. To get our
Dockerfile checked, we will use
dockle. If you have not installed
dockle on your machine, check the detailed installation instructions here on GitHub.
To get our image checked, invoke
dockle container-security:0.0.1 and you should see the following incidents being displayed:
WARN - CIS-DI-0001: Create a user for the container * Last user should not be root INFO - CIS-DI-0005: Enable Content trust for Docker * export DOCKER_CONTENT_TRUST=1 before docker pull/build INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image * not found HEALTHCHECK statement
For the scope of this article, we will fix
CIS-DI-0006. We will not enable content trust for Docker (DCT) in this article because not all managed services offerings for running containerized workloads support DCT as of today. This also applies for Azure Kubernetes Service (See corresponding discussion on GitHub).
CIS-DI-0001: Last user should not be root
Dockerfile generated by VSCode does not add a custom, non-privileged user. This means our container will run as
root, which should be prevented is possible. We can fix
CIS-DI-0001 by adding a new user and switching to the user context with the
USER command. So let’s quickly update our
Dockerfile, add a dedicated user called
bob, and switch in his context:
#build stage FROM golang:alpine AS builder RUN apk add --no-cache git WORKDIR /go/src/app COPY . . RUN go get -d -v ./... RUN go build -o /go/bin/app -v ./... #final stage FROM alpine:latest RUN apk --no-cache add ca-certificates COPY --from=builder /go/bin/app /app # non-root RUN adduser -D bob && chown -R bob /app USER bob # end-non-root ENTRYPOINT /app LABEL Name=containersecurity Version=0.0.2 EXPOSE 3000
With those changes in place, let’s create a new image from the
0.0.2 tag. When
docker has finished the build, we can again invoke
dockle to check the image:
docker build . -t container-security:0.0.2 # ... # => => naming to docker.io/library/container-security:0.0.2 dockle container-security:0.0.2 INFO - CIS-DI-0005: Enable Content trust for Docker * export DOCKER_CONTENT_TRUST=1 before docker pull/build INFO - CIS-DI-0006: Add HEALTHCHECK instruction to the container image * not found HEALTHCHECK statement
dockle reports, we’ve successfully optimized our
CIS-DI-0001 is gone.
CIS-DI-0006: Add health check instruction to the container image
CIS-DI-0006, we have to provide instructions for Docker on how to perform health checks for this container. Luckily, our sample API exposes several HTTP endpoints. For continuously checking the health of the API, we want Docker to issue an HTTP request to
Consult the official
Dockerfile reference here, to see which options and arguments can be used in combination with
Remember that health check commands are executed inside the container. Because our base image is tiny, we’ve to add a package to issue an HTTP request. For demonstration purposes, let’s take
curl. Let’s update the
Dockerfile to add
curl and configure our custom health check:
#build stage FROM golang:alpine AS builder RUN apk add --no-cache git WORKDIR /go/src/app COPY . . RUN go get -d -v ./... RUN go build -o /go/bin/app -v ./... #final stage FROM alpine:latest ## also install curl for healthcheck RUN apk --no-cache add ca-certificates curl COPY --from=builder /go/bin/app /app # non-root RUN adduser -D bob && chown -R bob /app USER bob # end-non-root # healthcheck HEALTHCHECK CMD curl --fail http://localhost:3000/healthz/alive || exit 1 # end healthcheck ENTRYPOINT /app LABEL Name=containersecurity Version=0.0.3 EXPOSE 3000
Although we used
curl in this example, keep in mind that
curl will impact the overall size of your container image, and the overall attack surface of your container image increases because of the added software. Sometimes it is wiser to write a small (self-contained) CLI for Docker health checks. Although this can be achieved with any programming language, I would prefer writing it in Rust because of robustness, safety, and the final size of the distributable.
Inner-loop vulnerability scanning with Snyk
Now that our
Dockerfile meets best practices (except
CIS-DI-0005), we must check our container images for known vulnerabilities. Vulnerability scanning checks all software inside the container (Software bill of materials SBOM) for known vulnerabilities. This means that all its dependencies and even Alpine Linux are scanned on top of our executable.
Plenty of tools are available for scanning container images in the inner loop. I decided to demonstrate vulnerability scanning using Snyk (
snyk CLI). I’ve chosen Snyk because it is also integrated with Docker Desktop, which most developers use these days on Windows and macOS. If you want to use the standalone
snyk CLI or scan your container images for vulnerabilities directly with
docker scan, a Snyk account is required. For further information, check out the corresponding parts of the Snyk documentation. Once installed, you have to authenticate using
To scan our container image with
sync container test container-security:0.0.3. Snyk finish the scan and report 0 vulnerabilities
Testing container-security:0.0.3... Organization: thorsten-hans Package manager: apk Project name: docker-image|container-security Docker image: container-security:0.0.3 Platform: linux/amd64 Base image: alpine:3.15.0 Licenses: enabled ✔ Tested 19 dependencies for known issues, no vulnerable paths found. According to our scan, you are currently using the most secure version of the selected base image.
However, there may be situations where your vulnerability scanner will report a list of discovered vulnerabilities. If this happens, enough information is included to understand which dependency introduces the vulnerability, and links to detailed information about how to remediate it if a fix for that particular vulnerability is available:
✗ Medium severity vulnerability found in glibc/libc-bin Description: NULL Pointer Dereference Info: https://snyk.io/vuln/SNYK-UBUNTU2004-GLIBC-1564900 Introduced through: email@example.com, meta-common-packages@meta From: firstname.lastname@example.org From: meta-common-packages@meta > email@example.com Image layer: Introduced by your base image (ubuntu:20.04)
Outer-loop vulnerability scanning with Azure Container Registry and Azure Defender for Containers
Although we successfully scanned our image for vulnerabilities in the inner-loop using Snyk, we should always enforce vulnerability scanning in the outer-loop too. Popular container registries such as Azure Container Registry (ACR) provide seamless integration with vulnerability scanners. ACR is no exception here. By enabling Azure Defender for Containers, your container images will automatically scanned when pushed to your ACR instance.
Microsoft does not have its own vulnerability scanner. Instead, Azure Defender for Containers leverages the vulnerability scanner created and provided by Qualys. When writing this article, customers using Azure Defender for Containers are charged $0.29 per image scan.
To enable Azure Defender for Containers, Azure Defender for Cloud must be activated for the desired Azure subscription: You can enable Azure Defender for Cloud by navigating to Azure Defender for Cloud | Environment settings. From the list of subscriptions, select the desired Azure subscription and ensure Azure Defender for Cloud is activated.
From the list of features, ensure Azure Defender for Containers is set to ON as shown in the following figure:
Once you’ve activated Azure Defender for Containers, Azure will scan container images in the following situations:
- new image / tag get pushed to ACR
- new images get imported to ACR
On top of that, all container images that are pulled from ACR during the last 30 days will be scanned once a week from Azure Defender for Containers for free.
If the scanner detects vulnerabilities, all information about the vulnerabilities of an image will be aggregated and displayed in Azure Defender for Cloud, along with recommendations on how to remediate and further background information about each vulnerability.
Containers are perhaps the most popular distribution format of this time. Managed Kubernetes offerings such as Azure Kubernetes Service (AKS) and others drove container adoption to an even higher level in the past years. Every developer must have a common understanding of what containers are, how they work, and how they should be authored with security in mind.
In addition to authoring robust and secure Dockerfiles, development teams regularly have to check their custom container images for vulnerabilities. Fortunately, there are plenty of tools available to integrate our software development process seamlessly.