TL;DR
- Secrets regularly appear in storage services, environment variables, virtual machine user data, infrastructure code, logs and source code repositories.
- During one assessment, a Terraform state file in an overly permissive Amazon S3 bucket exposed credentials for an automated GitHub application.
- Those credentials led us into Microsoft Azure and then back to administrator access in another Amazon Web Services account.
- Secret managers help, but the wider fix is to use short lived identities, tightly scoped access, runtime retrieval, regular scanning and effective rotation.
A secret is only as safe as the route it takes
Secrets are supposed to let systems authenticate without leaving credentials sitting in plain view. However, a secret still has to be created, stored, retrieved by a workload, used and eventually replaced. Every stage creates another opportunity for it to appear somewhere it shouldn’t.
Cloud service providers offer dedicated tools such as AWS Secrets Manager, Azure Key Vault and Google Cloud Secret Manager. These are useful, but moving a secret into a vault does not solve the whole problem. What is often forgotten is the identities that can read it, the way an application retrieves it and any copies that are left behind.
During cloud penetration tests, we often find that the location of the first secret is only the start. Once we can use it, the permissions and trust relationships attached to that identity determine how far the compromise can travel.
The usual hiding places
We regularly find secrets in storage services, environment variables, virtual machine user data, infrastructure code, logs and source code repositories. These locations are not automatically insecure. The problem is that access to them is often broader than the team expects and the data tends to remain long after its original purpose has gone.
Hardcoded credentials are an obvious example, but they are not the only one. A deployment process might write a secret to a log. A build pipeline might place it in an environment variable. A developer might put it in a private repository and assume that private means safe. The decision can look reasonable in isolation while still creating a useful path for an attacker.
Terraform state deserves particular attention. It records the resources Terraform manages and can contain sensitive values. Encrypting it at rest is sensible, but encryption does little to help if too many identities can read the bucket or storage account containing it.
An S3 bucket to administrator access elsewhere
On an assessment, we found an Amazon S3 bucket containing Terraform state files. Its access controls were too permissive, which allowed us to retrieve a state file and extract credentials used by an automated GitHub application.
The credentials worked against a private GitHub repository. Inside it, we found more stored secrets, including credentials for a privileged Azure service principal. That identity allowed us to interact with resources in Azure and retrieve further secrets from Azure Key Vault.
Among those secrets were AWS access keys with administrator access to another AWS account! What began with one readable state file had become a route through GitHub, into Azure and back into a separate AWS environment with administrative privileges.
Lessons learnt
There was no dramatic cloud escape in this chain. The S3 bucket allowed the state file to be read. The GitHub credentials authenticated successfully. The Azure service principal had broad permissions. Key Vault returned secrets that identity was allowed to access. The AWS keys then provided the privileges assigned to them.
Each platform largely did what it had been configured to do. The weakness was in the way secrets and permissions connected them. Cloud accounts that looked isolated on an architecture diagram were linked in practice by repositories, deployment tooling and reusable credentials.
That is why exposed secrets should not be treated as isolated housekeeping findings. The useful question is not only where the secret was found. It is what the secret can reach and which other credentials become available from there.
Why exposed secrets remain common
Many exposed secrets began as a convenient way to get a deployment working. The deployment changes, the person who created it moves on and the credential remains. A temporary workaround then becomes part of the environment.
Long lived access keys make the problem worse. They give an attacker a larger window than short lived tokens or workload identities, are easier to reuse outside the intended environment and can be difficult to trace back to one application.
Access to the storage location is another common issue. A secret can sit in a sensible vault but still be exposed through an identity with excessive permissions. If every workload can open the cupboard, the cupboard is not doing much for you.
Make a stolen secret less useful
The most useful improvements reduce both the chance of exposure and the value of anything that is exposed. Practical steps include:
- Use AWS Secrets Manager, Azure Key Vault or Google Cloud Secret Manager as the controlled source for secrets rather than storing them in repositories or static configuration.
- Prefer managed identities, workload identities and short lived credentials so applications do not need reusable access keys wherever the platform supports this.
- Retrieve secrets at runtime instead of building them into images, infrastructure code or deployment packages.
- Scan repositories, logs, cloud storage and infrastructure code for exposed credentials. Run this continuously, not only before a release.
- Apply least privilege to the identities that can read or change secrets and monitor for unusual access, such as one identity retrieving many values.
- Rotate secrets automatically where possible and make sure the old value is revoked. Creating a replacement does not always end existing sessions or remove an attacker’s access.
The aim is to break the chain
It is not realistic to assume that no secret will ever appear in the wrong place. Cloud estates change quickly and deployment processes involve a lot of moving parts. The better objective is containment.
One exposed Terraform state file should not unlock a private repository, a privileged identity in another provider and administrator access somewhere else. Short lived identities, tightly scoped permissions, controlled retrieval and effective monitoring reduce the chance that one mistake becomes several.