This article was co-written by Zach Rice and Joe Leon, both at Aikido Security.
tl;dr Some credentials are meant to be public, but secret scanners still flag them as generic secrets. We wrote suppression rules for the most common ones and reduced false positives by ~2%. These rules now ship by default in Betterleaks.
Secrets scanners are built on regular expressions. Each pattern targets a specific credential type, like an AWS secret access key, a GitHub PAT, or a Stripe token. Once the scanner identifies a potential secret, it performs a liveness check by, for example, sending an HTTP request to verify whether the credential is live.
This workflow breaks down when secret scanners attempt to identify generic secrets. Generic secrets are the API keys and credentials that don't have a unique or known signature. The scanners can still use a regular expression to find them, but instead of looking to match a known credential format, it uses a generic pattern to scoop up anything that looks like it could be a secret.
Reducing noise in secrets scanning and the problem with regular expressions
It’s rather easy to write a fancy regular expression to surface strings in code (and other places) that look like secrets. But without a predefined liveness check, the scanner can’t definitively say whether a generic secret is actually a live secret.
This approach leaves users to manually sift through false positives. But that’s no fun. So over the next several months, we’ll be shipping a series of noise-reduction layers:
- Token efficiency filtering (last post)
- Publishable credentials denylist (this post)
- Public secrets denylist
- Splitting generic rules into two types: tokens and username/passwords
- ML-assisted triage
We want to do things in this particular sequence because we’re looking to remove as many false positives as cheaply as possible, and then use expensive ML operations for the leftover ambiguity. It won’t ever be perfect, but we believe generic secrets are where the biggest opportunity exists in secrets scanning. It’s the best place for researchers to uncover novel findings and defenders to outpace threat actors. The goal of this series is to make Betterleaks’ default rules good enough to enable researchers and analysts to triage generic secret results at scale.
Why some keys are considered publishable (not-so-secret) keys
Many SaaS and cloud providers mint credentials for users that are designed to be public. Stripe, the payment processor, provides users with multiple key types, including a “Publishable API key”. These keys are designed to be “put in front-end code.”
A generic secret detection rule would likely surface this as a candidate secret. It’s high entropy, located near keywords like “api_key”, and just generally looks like a credential. But it’s not an actual secret. It’s meant to be public, and it’s not something that a researcher or analyst should waste their time manually reviewing.
So we wrote a regular expression to remove this key type from Betterleak’s generic secret results.
But Stripe is just one key type. We needed to scale this process up. We researched the most popular public-by-design credential types and manually authored detection signatures for each. Yes, manually. AI was surprisingly useless for this task.
Reducing false positive rates among generic secrets
After hours of manually adding these expressions, we wanted to see if it was worth it. To measure the change, we downloaded 100GB from Common Crawl and scanned it twice with Betterleaks. Once with the new signatures, once without.
The new publishable-credential signatures cut false positives by 2.37%. In our dataset, that was 27,886 fewer secrets to review manually.
It costs about 5% more scanning time, but avoiding that many false positives is well worth it.
Your reduction rate will vary with the data you scan. Common Crawl skews toward front-end HTML and JavaScript, exactly where publishable credentials live, so it likely inflates our results. Either way, every publishable key the filter removes is one a researcher or analyst never has to triage.
Public doesn’t always mean safe
The challenge with taking this approach is that publishable credentials don’t necessarily mean there is no risk. Each key type requires manually reviewing the shape of the key and the access granted to it within the context of the SaaS platform. While building this list, we encountered five cases that are worth reviewing, especially for anyone considering submitting a PR to add more (please do!).
Unique shape: easy
These are easy. The publishable credential has a unique, unmistakable shape. We write a unique regular expression and are confident that this won’t exclude any potentially valid generic secrets.
No unique shape: needs context
These are strings that literally could show up anywhere. Maybe it’s a UUID, or maybe just any 32 hex characters. The format alone does not identify this string as a publishable credential type. In these cases, we rely on a nearby keyword confirming that it belongs to a particular SaaS platform or cloud provider.
This approach isn’t new. We do this all the time for secret credentials.
Same shape as a secret: needs a liveness check
These are tough. Some SaaS providers decide to mint secret and publishable credentials with the exact same format. Honestly, we wish they wouldn’t do this. It makes secret detection harder and confusing users. But we encountered many of these.
In these cases, the only way to safely remove the punishable key type is to add a detection signature (with an HTTP liveness check request) for the secret credential.
Since the publishable and secret keys are the same shape, the secret key type detection signature would pick up both, and after a liveness check determine that the publishable key is not a secret, and discard it.
Sensitive only if misconfigured: needs a liveness check
There are some key types that SaaS and cloud providers mint where, depending on whether a user added a particular permission, the key might be either sensitive or without risk. I’ve written a good bit about Google API keys and their ability to access Gemini. That’s one case. But I’ve seen this pattern with Algolia keys and others.
In these situations, the best path to follow is similar to when we see format collisions between publishable and secret keys: we send a liveness check to determine whether it can access anything sensitive.
Test credentials: needs manual triage
Some SaaS platforms, like Stripe, provide users with a test or sandbox environment. At first, we considered categorically removing these results from generic secret detection results.
However, on closer look, it became clear that some organizations put production data into their test environments. And while a leaked sandbox Stripe key might not be able to send thousands of dollars to a threat actor’s account, it might provide a threat actor with access to customer PII or other important internal data. Since we can’t know whether a test credential provides access to sensitive internal data, we leave these results in the findings and rely on manual triage.
Fixing generic secrets detection one step at a time
Generic secret detection is a filtering problem, and filtering is won by a series of small wins stacked on top of each other. We’re starting by removing what we’re certain about. Our hope is that a few of these small changes will enable effective generic secret detection at scale.
The current release of Betterleaks filters publishable keys from generic results by default. Give it a try!