Why deal with cleartext secrets?
Although only developers have access to their own repositories, most of them are cloned locally, and cleartext secrets (if any) can be found both on workstations and in source code managers (SCM). Therefore, compromising one of these computers or a developer account can enable an attacker to easily get his hands on secrets that could be very critical (especially infrastructure secrets).
We should never assume that this type of attack will never happen but rather prepare for the moment it does. It has become very popular in recent years and can target even the most cautious of people.
When best practices regarding credentials were not followed from the very beginning, it can become quite complex to deal with all those that have been added since then.
Prevent developers from introducing new secrets
The very first step is to prevent developers and tech members from introducing new secrets while working on this issue to make sure that they do not make the situation worse while you work at resolving it.
However, in some cases, you may want to start with detecting and dealing with credentials first to remove all occurrences of critical secrets before taking time to work on these approaches.
There are three ways to make sure that no new secrets are added:
- Use pre-commit hooks to detect potential credentials by running secret detectors even before they are actually added to the repository. This approach has the downside of requiring all developers to install these pre-commit hooks in the first place. Note that some of them may try to bypass them, as they can see this as a regression in their developer experience.
- Scan for secrets in pipelines, preferably on push events (not to miss standalone commits that do not belong to a PR, for instance, but that can still be retrieved from your SCM by an attacker). In this approach, developers do not require any particular setup, but new cleartext credentials can still be pushed in the SCM and thus require performing a secret rotation.
- Use SCM servers’ pre-receive hooks, which can actually perform the secret scan while receiving a commit and reject it if credentials are detected. This approach is the best one, as it does not require developers to install anything on their computers, and secrets are actually never stored in the SCM. However, installing these hooks requires hosting your own SCM server.
In order to follow the principle of defense in depth, it is even better to actually implement these three solutions at the same time.
Detect cleartext secrets in your code base
The next step is to assess the extent of the problem. To do this simply and effectively, the steps are to:
- Detect all cleartext credentials in the scope of your organization in your SCM: you can do this thanks to open-source tools like Gitleaks, TruffleHog or git-secret-scanner. The latter combines the first two tools to pre-process (to know if they are still valid) and detect more secrets.
- Sort all detected credentials by removing false positives and putting aside those that are no longer valid.
- Prioritize secrets depending on their use case in order to take care of critical credentials first. For instance, the compromise of infrastructure secrets like GCP service account keys or AWS secrets can have way greater impacts than the compromise of OAuth credentials.
Deal with detected cleartext secrets
Once the previous step is done, all that remains to be done is to process all these credentials, starting from secrets with the highest priority to make sure they no longer appear in your SCM.
Two approaches can be followed: either clearing the git history from all secret occurrences or rotating secrets. The latter is to be preferred as the former involves modifying the git history, which can have undesirable consequences (like losing parts of the code).
In addition, it is not sufficient in case parts of the code have already been leaked. In both cases, a secret manager (like Vault) is required to move cleartext secrets to a secure place where they are encrypted and where fine access control can be implemented.
The principle of secret rotation is the following:
- Finding out where the secret is used: it is a bad practice, but several applications can use the same credentials to access the same service. When a secret is rotated, its previous value is invalidated, so all the applications that were using it can encounter errors. In addition, the secret can already be stored and retrieved securely from secret managers by correctly designed applications.
- Creating one new secret for each application that uses the secret: to avoid interrupting applications, you likely want to have several valid credentials at the same time to access the service they originate from. If the same secret is used by multiple applications, it is preferable to generate one new secret per application and per environment. This facilitates future rotations in case of a compromise (since there would only be one instance of the secret to rotate) and auditing (many services allow identifying which credentials were used to access them).
- Replacing the occurrences of the old secret with the new secret(s): all the occurrences of the old secret can be removed, and the deployment of applications can be modified to retrieve the new secret(s) from the secret manager.
- Redeploying applications: all applications must be redeployed with the new secret instead of the old one.
- Invalidation of the old secret: once all the applications have been redeployed with the new secret(s), the old secret can be deactivated (when possible) and then deleted so that it can no longer be used. Deactivating it first allows to quickly rollback in case some applications were not detected in the first step.
Unfortunately, some services only allow having one secret at a time. In this case, it is only possible to delete the old value, create a new secret, and then redeploy applications, which can cause downtimes.
Conclusion
While it is quite obvious that it is important to deal with cleartext credentials, there are other good practices to follow with secrets. Especially:
- Do not share the same secret between applications: each application should have its own secret to access a service. It makes it easier to rotate application secrets in case of a compromise, as there is only one occurrence of secrets to rotate. It also reduces the impact of credentials leaks since each secret can be scoped strictly to what is necessary for the application (least privilege). With distinct identities, it is way easier to understand the path followed by an attacker in your infrastructure.
- Do not use long-living credentials at all when possible: Workload Identity Federation and OpenID Connect (OIDC) allow CI runners and other services to interconnect without sharing any secrets. For instance, GitHub runners can authenticate to GCP to deploy applications only using their identity in GitHub.
- Do not use (shared) secrets for humans to access other services: humans should be managed with Single Sign-On (SSO). SSO allows managing all identities in one place and greatly reduces the risk that a secret may leak as it removes the need for humans to store these credentials on their computers.