Kubernetes hardening: Drilling down on the NSA/CISA guidance

Kubernetes has become the de facto choice for container orchestration. Some studies report that up to 88% of organizations are using Kubernetes for their container orchestration needs and 74% of that occurring in production environments. That said, security remains a critical concern with as many as 94% of organizations reporting at least one security incident in their Kubernetes environments in the last 12 months.

As organizations continue to adopt Kubernetes, it is crucial that they follow industry best practices and guidance to ensure they are using Kubernetes securely. One great resource is the Kubernetes Hardening Guidance recently published by the US National Security Agency (NSA) and Cybersecurity and Infrastructure Security Agency (CISA).

Other valuable security guides and books for Kubernetes security include the Center for Internet Security (CIS) Kubernetes Benchmark, Department of Defense (DoD) Kubernetes Security Technical Implementation Guide (STIG), and the Kubernetes Security book by Liz Rice and Michael Hausenblas of Aqua Security.

The Kubernetes Security Hardening Guidance identifies three common sources for Kubernetes security risk: supply chain risks, malicious threat actors, and insider threats. The guidance also breaks down its coverage across key areas such as threat modeling, Kubernetes pod security, network separation and hardening, authentication and authorization, log auditing, and upgrading and application security best practices.

The guidance starts by explaining the core architectural components of Kubernetes. This includes the overarching cluster and its core components such as the control plane, nodes, and pods, which reside on nodes. Understanding these core components is key to understanding how Kubernetes functions, how the components interact, and ultimately how to secure them.

NSA/CISA

Control plane

One of the most important aspects of Kubernetes clusters is the control plane. It makes global decisions about the cluster and can have devastating effect if it is compromised by malice or simply incompetence. This investigation uncovered 2,284 etcd servers that malicious actors could access through the internet.

The components that make up the control plane include the controller management, cloud controller manager, Kubernetes API server, etcd, and the scheduler. Each operates on unique ports within the cluster and has specific functions. Your ability to exert control over these components can vary, especially if you are using a managed Kubernetes service from a cloud service provider (CSP) such as AWS’s Elastic Kubernetes Service (EKS). While this may seem like a drawback, managing the Kubernetes control plane is a complex task. Depending on your workforce skill level, using a managed Kubernetes offering and focus on your applications is probably the most secure option. A managed Kubernetes offering like EKS handles activities related to the control plane and underlying infrastructure like upgrades, fixes, and security patches. 

NSA/CISA

Pod security

Kubernetes pods are the smallest deployable unit in the Kubernetes ecosystem. This is where organizations’ containerized applications reside, and pods are a primary entry point for malicious actors. For this reason, it is critical to follow best practices. This includes not allowing containers to run as root, using immutable containers with writable directories, and ensuring you are deploying secure container images, free of known vulnerabilities, misconfigurations, poor access control, protecting pod service account tokens, and more. You can also take advantage of emerging guidance on container hardening best practices.

Network separation and hardening

Your security efforts shouldn’t stop at the pods. Networking within the cluster is also key to ensuring that malicious activities can’t occur, and if they do, they can be isolated to mitigate their impact. In addition to securing the control plane, key recommendations include using network policies and firewalls to both separate and isolate resources and encrypting traffic in motion and protecting sensitive data such as secrets at rest.

One core way of doing this is taking advantage of the Kubernetes namespace native functionality. While three namespaces are built-in by default, you can create additional namespaces for your applications. Not only does the namespace construct provide isolation, but it can help use resource policies to limit storage and compute resources at the namespace level as well. This can prevent resource exhaustion, either by accident or maliciously, which can have cascading effect on the entire cluster and all its supported applications.

While namespaces can help provide resource isolation, leveraging network policies can control the flow of traffic between the various components including pods, namespaces and external IP addresses.

Cloud infrastructure

While much of the guidance focuses on securing Kubernetes clusters and their core components, it points out that you must still protect the cloud infrastructure itself. Kubernetes is most often deployed on virtual machines running in cloud environments. If you fail to secure those machines, all the security rigor employed on the underlying clusters and applications will be irrelevant if a malicious actor can just take down, take over, or compromise the infrastructure your clusters and applications reside on.

Administrators should adhere to industry best practices from the CSPs to harden the infrastructure and leverage either cloud-native or third-party tooling to both secure and monitor the hosting infrastructure for malicious activities. A great place to start is security guidance from the leading CSPs such as AWS, Azure and Google Cloud Platform.

Authentication and authorization

Much like other aspects of IT, authentication and authorization are a fundamental part of securing access to Kubernetes. This is further complicated by the fact that user authentication is not a built-in feature of Kubernetes.

In Kubernetes there are two types of users: service accounts and normal user accounts. Kubernetes uses bearer tokens to support authentication and these tokens are often stored in well-known locations, which can be targeted from outside the cluster if not secured.

Administrators can implement role-based access control (RBAC) to assign roles to individuals and grant access accordingly. This must be enabled in the cluster and then the administrator can use the native permission types of roles and clusterroles. Administrators can leverage roles to set permissions for specific namespaces and clusterroles to set permissions for all cluster resources.

Log auditing

Despite the best of efforts to prevent and mitigate all security incidents and concerns, incidents will happen. When they do, logging is absolutely critical to determining what happened and how. The guidance emphasizes using log auditing for both verifying that things are operating as intended and being able to investigate when they aren’t. The guidance lays out specific activities that should be logged, which includes API requests, performance metrics, deployments, network traffic, and scaling. All these metrics provide key insights into how clusters are operating and potentially malicious activities, should they occur.

The guidance points out that auditing capabilities aren’t enabled by default. It also emphasizes the need to use next-gen SIEM platforms that support Kubernetes environments due to their complex and dynamic nature.

Upgrading and application security practices

The guidance points out that while hardening clusters and the various components is important, security doesn’t stop there. Security practices and activities also apply to applications running on the servers and hosted in the Kubernetes environments. This is where industry guidance such as CIS benchmarks and others come into play. While taking the measures laid out in this guide can help, as well as using managed Kubernetes service offerings from CSPs, you ultimately still own and are responsible for the applications running in the clusters. You must secure them appropriately.

Securing a Kubernetes cluster, its various components, residing applications, and the underlying cloud infrastructure is an arduous and complex task. That said, industry guidance like the NSA/CISA Kubernetes Hardening Guide provide a great starting point. Misconfigurations pose the greatest threat to cloud and Kubernetes environments, and teams must address them accordingly to ensure secure adoption and use.