Keeping secrets in a devsecops cloud-native world

Devsecops adoption is widely underway, with many organizations looking to break down silos among development, security and operations while leveraging cloud-native architectures to drive secure software outcomes for organizations. That said, one problem is still pervasive throughout this devsecops journey and that’s the case of secrets management.

Secrets consist of information organizations desire to keep private, most notably credentials, access keys or certificates of some form. As evident from previous data breach reports, such as IBM’s, credentials and secrets remain among the top factors involved in incidents and data breaches, often giving malicious actors initial or lateral access in environments.

Poor secret management practices have been tied to or exploited in several breaches, including Codecov and Twitch. Recent activity around the Samsung data breach that exposed portions of Samsung’s source code included over 6,000 secret keys.

Secrets management has been a challenge in years past and the problem only seems to be escalating. In a recent State of Secrets Sprawl 2022 report from GitGuardian, over 6 million secrets were detected in scans of public GitHub repositories in 2021, which is twice as many compared to 2020.

Some of the problems are that organizations lack maturity around secrets management, particularly in cloud-native devsecops oriented environments. This includes gaps around inventorying secrets, centralized approaches, controlling access, and preparing to respond to secret incidents when, not if, they occur.

Other factors exacerbating the secret sprawl problem include the growing popularity of open-source solutions and associated repositories. Source code, images and infrastructure-as-code (IaC) manifests are but a few sources where secrets can inadvertently or unknowingly be committed and subsequently exposed.

Organizations’ widespread use of open source-technologies container images and IaC templatized manifests can save a lot of time and expedite innovation, but it also can serve as an avenue of secret exposure, particularly when committed back to the community in public repositories.

Public repositories aren’t the only source of concern. As organizations continue to experience data breaches that compromise their internal private repositories or even continuous integration/continuous delivery (CI/CD) build systems and pipelines, secrets can also be exposed to malicious actors. Application security teams are struggling to keep pace with the activity of development teams and others when it comes to implementing secret management practices and maturity.

Secrets management best practices

Organizations need to mature their secret management practices. Failing to do so can impact not only a single organization but have a cascading impact across the supply chain and ecosystem if the exposed secrets belong to a software vendor or cloud/managed service provider.

Part of the challenge are the diverse locations and methods in which secrets can be inadvertently shared, stored or exposed. In modern technology environments this includes in source-code repositories, CI/CD pipelines, container images, IaC templates and even a developer’s local workstation. Let’s discuss methods to mitigate some of these concerns.

Open-source and commercial tools

When it comes to repositories and CI/CD pipelines, you have both open-source and proprietary vendor solutions to choose from. Some of the most popular open-source options include Gitleaks and Trufflehog. Both can help support scanning Git repositories for stored secrets and be integrated with popular CI platforms like GitHub and GitLab to catch secrets flowing through the pipeline before they ever make it to a repository or worse, runtime environments. They’re able to search for secrets such as passwords, API keys and personal access tokens, all of which can be abused by malicious actors to impact not just your organization but also any customers, business partners or others whom you may interact with from a digital perspective.

These open-source solutions also support the use of pre-commit hooks, which can be run prior to something actually being committed, for example. While both options support the use of built-in rules, they also can create custom secret detection rules, which can be useful depending on the type of data your organization may be responsible for safeguarding.

It is important to tune these tools, because while they are extremely helpful in identifying potential exposed secrets, they also can be very noisy and frustrate the development team as they try to tackle the slew of alerts and findings to determine false positives versus real issues.

Emerging leaders in the vendor space include GitGuardian, which was cited earlier in some of their associated studies and findings on secret sprawl. GitGuard provides a code security policy engine which can be integrated with version control systems, DevOps tooling and IaC configurations to prevent secrets from being exposed. They also actively monitor every public GitHub commit and scan them, which currently has over 70 million active users, albeit not all of them are using public repositories. However, GitGuardian’s activity of doing this monitoring on public repositories helps inform some of the metrics cited earlier regarding the state of secret sprawl and just how big of a problem it actually is.

Developer education

Another recommendation is just developer education from the AppSec team. While tooling can go a long way and is a must to ensure you catch what you can, in the name of shifting left, starting with the developer is a great place to begin. Educating developers on the perils of leaked secrets and the ramifications it can have on not just your organization but others associated with you, either as customers or partners is also critical.

Include leaked secrets scenarios in your incident response plan

As with much of cybersecurity, it is not a matter of if, but when secrets are compromised. This is why your incident response plan and associated playbooks should cover scenarios of leaked secrets. If secrets are exposed, how does the organization respond? Who is involved? How do you invalidate those secrets to minimize the impact? How do you communicate with customers or partners who are impacted? These are key considerations when it comes to the problem of secrets sprawl and associated security incidents.

Taking a multi-faceted approach as discussed above through tools, processes and people can help your organization mitigate the risks associated with leaked secrets and the subsequent fallout. Implementing the items discussed above will be key for developers, AppSec engineers and cloud-native organizations to mitigate the risks associated with secrets. Keeping a secret is no trivial matter, particularly in the age of cloud and devsecops.