DEVOPSdigest

We all wish we could build, deploy, and run our applications without the stress of security concerns. However, the reality is that most of us will run into serious security or compliance issues at one time or another. When that happens, an organization is likely to experience the frustration of delayed application deployments and stifled agility. Containers and Kubernetes promise faster development cycles, quicker bug fixes, and increased velocity, but when security is an afterthought, organizations risk the very gains that containerization promises, particularly agility.

The recent StackRox State of Container and Kubernetes Security Survey found that 44% of survey respondents had to delay an application rollout due to container or Kubernetes security issues. But the right combination of policies, processes and tooling will help mitigate these security and compliance concerns so that teams are more confident pushing container and Kubernetes deployments into production, which should be the culmination of any successful container strategy.

It's interesting to note how DevOps and security teams tend to deploy Kubernetes — roughly 50% of users self-manage their deployments, according to the most recent survey. In this model, these teams must be aware of the security pitfalls and the necessary requirements needed to meet compliance standards. Not that self-managed Kubernetes is less-secure, rather, it simply requires more diligence with updates and patching and more fine tuning of Kubernetes-native security controls. The alternative is tapping into the declarative power of a Kubernetes-native security platform that integrates with the existing tooling of a self-managed environment. Whether provider-managed or self-managed, Kubernetes users must take care to protect their applications and infrastructure from vulnerabilities and misconfigurations, ensure compliance with external and internal policies, and detect and stop runtime threats.

Two-thirds of survey respondents referenced security and compliance as their largest source of concern with their organizations' container strategies. Inadequate investment in security led the list (34%) while others felt their organizations did not take threats seriously enough (15%). There were additional concerns that compliance needs were not being accounted for (17%) as well.

Despite security concerns, organizations are increasingly adopting DevSecOp models that enable security to be tightly integrated with application development and operations starting in the build phase of the software development lifecycle. The survey found that the vast majority of respondents have some form of DevSecOps initiative underway. Only 17% continue to operate DevOps separately from security, proving that DevSecOps is more than a buzzword. This shows promise for the ability of organizations to continually shed concerns about security in order to optimize container and Kubernetes deployments for production.

However, organizational development and IT operations are only part of the equation. There are major container and Kubernetes security risks that exist beyond the realm of container strategies that must be addressed. According to the survey, 90% of respondents claimed to have experienced a security incident in their container and Kubernetes environment in the last 12 months. The most common types of security incidents as reported by respondents include misconfigurations and exposures, Kubernetes vulnerabilities, runtime threats, and failed compliance audits. As such, it's critical that organizations take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments further.

Exposures due to misconfigurations were some of the most worrisome security risks in container and Kubernetes environments, and for good reason as 67% of the survey respondents detected a serious misconfiguration in the last 12 months. These risks fall under the banner of configuration management, which pose a unique security challenge when using Kubernetes to orchestrate containerized applications. While a host of tools are available for vulnerability scanning of container images, configuration management requires more consideration.

Teams generally know not to expose Kubernetes dashboards to the Internet but configuring a pod’s security context or implementing Kubernetes RBAC are just two examples of more challenging settings DevOps teams need to get right. Because of this, configuration management should be automated to control the risk of human error due to error-prone manual processes. The survey indicated that most misconfiguration-related security incidents in Kubernetes are caused by human error. Development and security teams must look closely at how they configure images, secrets, namespaces, runtime privileges, network policies, persistent storage, and the control-plane - especially for self-managed Kubernetes deployments.

Kubernetes vulnerabilities are another major source of risk, and one that continues to increase given the mounting frequency of CVEs impacting the ecosystem, for example CVE-2020-8554, a man-in-the-middle vulnerability announced in December 2020. Common exploits of known vulnerabilities include crypto mining and other malware installation as well as privilege escalation and host access. The problem is pervasive and while image scanning at the build stage is necessary, vulnerabilities also pose a security risk to deployments in production. Effective vulnerability management spans the entire container lifecycle, and should:

■ Identify vulnerabilities in images

■ Prevent images with risky vulnerabilities from getting pushed to production-accessible container registries

■ Fail builds containing fixable vulnerabilities above a certain severity

■ Leverage custom or third-party admission controllers in Kubernetes clusters to prevent the scheduling of vulnerable container images

The runtime phase introduces additional threats for Kubernetes users from external adversaries. In this phase, software is in the wild, so there are additional tactics teams should use to defend against these risks. It starts with monitoring runtime activity and establishing a baseline of known application behavior, particularly the most security-relevant container activities such as process activity and network communications among and between containerized services and external clients and servers. Kubernetes’ declarative data is also critical in this phase. Using date from the build and deploy phases can help evaluate observed versus expected activity during runtime in order to identify suspicious activity. Limiting unnecessary network communication is another best practice. Runtime is when you can see what kind of network traffic is needed for the application to function, as opposed to what’s allowed, offering the opportunity to remove unnecessary connections and reduce complexity that can introduce risks. Finally, users should leverage process allow-lists after observing an application for a period of time, identifying processes that are executed in the normal course of its behavior, then using this as an allow list against future application behavior.

Compliance, being one of the primary drivers behind container and Kubernetes security initiatives, is one final area that must be taken very seriously. Sixteen percent of StackRox’s survey respondents admitted failing a compliance audit in the past year - often this can be attributed to security becoming an afterthought in the journey towards container adoption. There are several compliance standards specific to containers and Kubernetes that apply to all organizations — CIS Benchmark for Docker, CIS Benchmark for Kubernetes, NIST SP 800-190. Industry specific compliance standards include PCI-DSS, HIPAA, and SOC 2. A too-common mistake is to wait until containers are in production before considering their compliance requirements, or only focusing on compliance at the runtime phase. The best way to mitigate risk from failing a compliance audit is to implement security controls as early as possible in the container life cycle. Automating compliance checks and evidence reporting as much as possible will help to reduce operational overhead.

Organizations need to take the necessary steps to mitigate these risks so they don’t resort to delaying application deployments more than they already are. Given the lack of confidence in security and compliance being felt within DevOps and security teams, it’s natural to hold back on production deployments, but it will limit the benefits of containerization and Kubernetes adoption. Start small by looking for ways to gain visibility into cloud-native environments. Then try to understand how images are built and whether they contain any vulnerabilities, how the workloads and infrastructure are configured to operate, and where compliance gaps exist. With this information, security and DevOps can begin to enforce policies to reduce risks to an acceptable level and work towards fulfilling their container and Kubernetes adoption goals.