Everyone’s talking about Kubernetes these days because of the role it plays in deploying highly scalable and dynamic application environments.
Unfortunately, what people are not talking about as much is Kubernetes security. While there are a few Kubernetes security resources out there (such as some basic security tips in the Kubernetes documentation), they’re few and far between.
In this post, I’d like to offer some additional Kubernetes security best practices. They reflect my real-world experience using Kubernetes in production, and include consideration of securing both Kubernetes itself and an application that is deployed using Kubernetes.
Kubernetes Security: Host
The foundation of any good security profile is a secure operating system running on the host. The best practice is to use a proven operating system and stay current on its patches, and apply recommended security hardening through the use of tools like OpenSCAP, or follow the checklists manually which are produced by organizations like NIST. Ideally, the operating system that underpins the running containers is optimized for a container environment by stripping out all non-essential elements. A great example is RancherOS. It was designed from the ground up to run containers and nothing else.
In reality, for most enterprises to follow best practices around the host, they are implementing their Kubernetes environments on top of full-featured, general-purpose operating systems like Microsoft Windows DataCenter and Red Hat Enterprise Linux, as they have existing procedures and tools to both patch and monitor the operating systems.
Kubernetes Security: Network
One of the main reasons that Kubernetes was such an innovative product was how it handled the network layer. As opposed to using NAT on each host like the first-generation Docker containers, Kubernetes exposed a unique address IP to each pod that was accessible from anywhere in the Kubernetes container. As this was accessible to anyone, it introduced the ability to have more fine-grained security, but initially caused some headaches as any wide-open network does.
Best practices for network security in Kubernetes are to go beyond the basic networking available and leverage the CNI (container network interface), which Kubernetes is standardized on to implement a more robust networking layer that includes either multi-tenant support, network policies, or both. Kubernetes covers the four types of networking in their documentation.
With a multi-tenant network, each namespace within the Kubernetes cluster gets a private addressable subnet and can only access pods in other namespaces that are specifically exposed as services. Open vSwitch is a great example of a networking layer for Kubernetes that has multi-tenant support.
Network policies in most next-generation networking layers are available to be deployed via Kubernetes. Project Calico is an example of a widely used networking layer that has been built from the ground up to support network policies with containers. Network policies allow cluster administrators to predefine network access control lists which control what ports are accessible and which services can be accessed. These policies are installed into the cluster almost as a set of primitives which an application developer can then assign to their application during deployment and use as the basis for the network access they feel their application will need to successfully operate.
Kubernetes Security: Code to Image Repository (CI/CD Pipeline)
There are several types of scanning that are common to any application, like static code analysis and fuzz testing, which are good practices to provide better quality code, but are not specific to containers or the modern applications which are being deployed on top of Kubernetes. Therefore, specific security best practices for containerized apps deployed with Kubernetes are required
And while Kubernetes isn’t necessarily involved during the build and packaging, having a secure environment starts with having a real assessment of the risk associated with anything being deployed. Application development has changed over time, from everything being built in-house to how current popular design patterns like microservices and all the modern front-end frameworks like Vue.js leverage open source libraries wherever possible to increase productivity and reduce time to deliver functionality to the clients. Containers are the poster child for this phenomenon, as they are built-in layers, and there is routinely no canonical list of what projects have been included in the final build of the container image.
To combat this, there are products which specialize in finding which versions of which open source libraries are in play inside an application, including going through any dependencies that are brought in through its package manager (like Maven or npm). By discovering and identifying all the projects in play, these products can identify any known vulnerabilities, identify which images are at risk as new vulnerabilities are discovered, and track the license that comes with that specific library. Tracking license compliance may not seem like a security risk. But it can have downstream effects like having to expose proprietary code later on if an incompatible license is included. For an example, see the current dispute between MongoDB and Debian.
Kubernetes Security: Provisioning (Deployment) and Running Containers
As I said in the previous section, Kubernetes is technically only concerned with an application from the point of deployment and under its management. For this reason, it’s important to add other tools to your security toolset that can address other potential vulnerabilities in your overall stack. These tools should be obvious to most experienced admins — They include things like firewalls and anomaly detection systems.
The few tools that do operate in this space that aren’t underpinning technology do more around monitoring which images are running where, and comparing that to the list of versions that were identified as part of the scanning mentioned above to allow for things like “critical vulnerabilities” to be potentially shut down immediately upon discovery.
Security is absolutely paramount in today’s modern computing environment as bad actors are increasing in both their numbers and their sophistication in attack execution. With this as the baseline of operations, best practices in Kubernetes security will continue to evolve and be refined to cover the ever-expanding attack surface that the exponential growth in containers and microservices are creating.