As 2022 draws to a close, recent incidents are no exception: the risk of secrets leaking is high.
In mid-September, Uber, the ride-hailing company, confirmed reports of an attacker who managed to break into several critical internal systems after finding hard-coded administrator credentials in a Powershell script. The secrets allowed logging into Uber’s privileged access management platform, where many other credentials were stored: the attacker was thus able to take control of several administrative accounts of AWS, GCP, Google Drive , Slack, SentinelOne, HackerOne, and more.
A few weeks later, Toyota, the automotive giant, revealed that it had accidentally exposed a credential key for accessing customer data in a public GitHub repository for nearly five years. Although they assured that the key had been invalidated, such a long exposure could mean that multiple malicious actors had already accessed it.
Millions of secrets exposed
Secrets (usernames and passwords, API tokens, encryption keys, etc.) are the cornerstone of security because they allow the right identities (whether they belong to humans or machines) ) to access the right thing at the right time. When security was about securing the perimeter of an organization, secrets were highly protected and kept out of reach of most developers and other IT professionals. Keeping the “crown jewels” safe was relatively easy as long as development and operations were completely separate domains.
But with the advent of DevOps and agile practices, hyperconnected systems have completely upended the notion of perimeter security. As the number of interconnected software components increases, the number of secrets needed to make them communicate also increases. Software components include databases, third-party components used during build or runtime, microservices, etc. Now that everyone and every asset in the software development lifecycle uses secrets, secrets are no longer a scarce resource. Mechanically, the probability of having a secret leak somewhere is also higher.
So in 2021, more than six million secrets were exposed on GitHub, the place “where the world builds software”. This number doubled compared to 2020. On average, 3 out of 1,000 commits contained a claim, which is 50% more than last year. Many of these secrets relate directly to corporate resources like cloud accounts, web servers, or public certificates, and many more could be used to impersonate an employee or escalate privileges.
This situation is clearly a boon for hackers. The secrets are there, ready to be exploited.
Various reports have highlighted the problem: “using stolen credentials” is the most common way to break into web applications, according to the latest DBIR report. It accounts for more than 80% of breaches, while “vulnerability exploits” account for less than 20% of cases. The report calls it “a low-cost, high-paying strategy that appeals to a range of attackers.”
The problem of secrets scattering across code bases is growing exponentially. Yet application security has lagged behind in responding to this threat. Let’s try to understand why.
Secrets in the Application Security Landscape
The focus of application security has traditionally been to find vulnerabilities such as Cross-Site Request Forgery (CSRF), Server-Side Request Forgery (SSRF), or logic flaws in source code. These vulnerabilities always manifest in the current and deployed state of an application.
With secrets, the story is different. A mistakenly hard-coded secret does not require the software to be running in order to be exploited. In fact, this flaw will remain dangerous even if it is not present in the running code!
When developers collaborate on code, they use a version control system – today almost always git. This tool keeps track of every source code revision – including errors – which means that a valid secret can be hard-coded into a version of the code that has never been deployed, and replicated as often as the base. of code is cloned. This is how credentials can expand much faster than application security can keep up, inside or outside the corporate IT perimeter.
Hard-coded credentials have another characteristic: they accumulate over time. The longer developers have worked (or will work) on a project, the higher the likelihood that secrets have been hard-coded at some point. These probabilities also increase with the size of the codebase and the total number of repositories. Therefore, hard-coded secrets accumulate much like “technical debt” does, except that technical debt shows up in code reviews, while secrets won’t. This is because code reviews only consider the net difference between the current state and the proposed change, and not all intermediate changes where secrets might have been inadvertently added.
The consequence is that security debt accumulates over time, and past a certain point it can become unmanageable for an application security team.
When searches were performed on the Twitch and Samsung codebases last year, between 6,500 and 7,000 hard-coded secrets were uncovered: internal company email passwords, API keys cloud services, third-party tokens and other authentication credentials, etc.
On average, it can be estimated that a typical company with 400 developers and four AppSec engineers would discover an average of 1,050 unique hard-coded secrets in their code repositories. More worryingly, each secret is hard-coded multiple times – 13 on average – which multiplies the effort required to properly “clean” the code (or at least investigate each case) for the application security team.
How to avoid hard-coded credentials?
The first step is to implement the controls now to prevent credentials from being hard-coded month after month and to “stop the bleeding”. Real-time monitoring is the best way to curb the proliferation of secrets because it catches errors as early as possible.
If the developer is alerted just before entering a secret into the VCS, the unit cost of remediation is less than a minute. The identifier was only exposed on the local workstation, and the developer can run a few commands to modify their contribution without consequences for their team.
On the other hand, if the faulty commit reaches the remote repository, the secret should be considered compromised: indeed, anyone with read access, today or tomorrow, could exploit it. This means it is a vulnerability. To remedy this, the secret must be revoked, regenerated and redistributed wherever it was used. This remediation procedure can take minutes, hours, or even longer, disrupting the workflows of multiple people.
On average, the cost of remediation often amounts to at least two man-hours. Applying this to the previously mentioned numbers, at least 2,100 man-hours would be required to achieve zero secrets in code in the average case (2 hours x 1,050 secrets). And this is a minimum, since it assumes that no other secrets will be hard-coded in the future.
Investigating thousands of incidents related to hard-coded secrets, prioritizing them, and remediating them can seem overwhelming for an application security team without a prior strategy. Yet this is not a fatality. By starting small and gradually curbing the accumulation of new vulnerabilities, in 6 or 12 months, the gains will be evident, as security engineers will have seen their mean time to remediation steadily improve.
Don’t overlook your secrets
In the world of software development, as it is today, no one can ignore the fact that hard-coded credentials represent a significant risk that threatens enterprise security. If you really want to obtain long-term results, you need to start today to regain control over your security debt. For hard-coded secrets, this means leveraging incremental source code scans to curb their accumulation over time. This will prove to be an indispensable asset in helping developers and security engineers solve this problem together.