The standard advice for handling a vulnerability is comfortingly simple: wait for the upstream maintainers to ship a fix, then apply it. It's good advice. It's also completely useless when there is no fix.
That's the situation we build for behind Athena, our vulnerability clearinghouse. Athena works with zero-days that frontier AI models surface, and models are now finding serious, exploitable flaws far faster than maintainers can patch them. For the overwhelming majority of these findings, there is no upstream commit to backport, no maintainer to defer to, and no reference fix to trust. Our starting point is a validated, exploitable vulnerability, an embargo clock, and a question: who is going to fix this?
At coalition scale, the answer must be automation. And that turns out to be a genuinely hard engineering problem, not because getting an AI to write a diff is hard. That part is trivial. It's because everything you'd normally lean on to trust that diff is gone.
The previous post in this series covered where this dangerous work runs. We use a microVM sandbox that lets us treat every job as a controlled detonation. Today’s post talks about what work happens inside that sandbox. We've written before about remediating CVEs in Chainguard Libraries for Python and about the fleet of agents that keeps our own code on standard. This post expands upon what happens when there is no upstream fix to fall back on.
Three ways to fix a vulnerability
It helps to be precise about what "patching" even means, because there are three very different jobs hiding under one word.
The easy one is to bump the dependency. If a fixed version exists, you just adopt it. The harder one is to backport an upstream patch. You take a fix that exists as a commit on main or a maintainer’s PR, and you adopt it to the version you’re running. This is hard work, but it has one enormous advantage. Someone who knows the code decided what "correct" looks like, and you get to stand on that decision.
The third job is authoring the fix yourself, from scratch, for a vulnerability nobody has ever patched. When backporting, we only accept patches that upstream maintainers have already merged into their own repository. We do this because nobody knows a project better than its authors. Here, there's no such patch to accept. The burden of proving the fix is correct falls entirely on us, under embargo, on code we don't own, across every affected version, for thousands of findings. Generating the patch is the least of it. You can learn more about how we handle this here.
Where you run code you don't trust
Before we can generate a single patch, we need somewhere to do it that assumes the code is hostile. Developing an exploit, synthesizing a fix, and building the result all mean running arbitrary build tooling and AI agents against code we have every reason to distrust. And we're doing it with undisclosed vulnerabilities, working exploits, and pre-disclosure patches in the environment.
That environment is the microVM sandbox we covered in depth in the previous post. The short version is that the posture for patch jobs is strict. We deploy one microVM per job. This is the floor that everything below runs on.
Generating a patch is easy. Trusting it is hard.
You can prompt an AI agent to "fix this vulnerability," go get coffee, and come back to a confident, plausible-looking diff. Anyone with any type of model can do that today. The problem is that "plausible" and "correct" are very different things, and when there's no upstream fix to compare against, the only way to tell them apart is to prove it yourself.
The first challenge is generating a patch that actually mitigates the vulnerability without introducing a new one or breaking the thing you're patching. A fix that closes the hole but breaks the API is not a fix. A fix that closes the reported hole while quietly opening a subtler one is worse than no fix and is not something that Chainguard will do.
As with our other remediation work, the value is in the guardrails, not the prompt, and we learned early which guardrails hold. Our first instinct was deterministic gates: conventional code checking each stage's output against fixed rules. For a problem this open-ended, they were too brittle. They tripped over harmless variation and missed what actually mattered. So we lean instead on adversarial review, where every artifact the pipeline produces has to survive being actively challenged before it advances. Rigid rule-checking still plays a part, but a small one. The harder, contextual judgment is what does the real gating.
Underneath the judges is an empirical ground truth that no amount of agent confidence can override. Three questions gate every candidate patch:
First, is it feasible? Can this even be patched in place at this version, or does the fix require changes that the version can't support?
Second, does it regress? We establish a baseline by running the project's existing test suite before touching anything, applying the patch, and running it again. If there are any new failures, we don't ship the patch. (Standing up a green baseline for an old version is its own archaeology, and a problem we've described at length before.)
Third, does it actually fix the vulnerability? Before we attempt a patch at all, we author an exploit test and prove it demonstrates real harm against the unpatched build. Writing the test first, while the vulnerability is the only thing in front of us, is deliberate. If we wrote it after the fix, its design could be quietly shaped by our patch, and it might end up proving only that the patch does what it does. A test that stands on its own is much stronger. We then run that same test against the patched build and require it to fail. The only variable that changes is the patch.
That last check carries more weight here than anywhere else we do this work. When there's no upstream fix, exploit-based proof is the correctness argument. There's no maintainer review to fall back on. The demonstration that the vulnerability is gone and that nothing else broke is the entire case.
Vulnerability Jenga
Now, to make it harder: the same package usually has more than one problem.
The moment you're fixing several vulnerabilities in one package, patches stop being independent. We call it a game of Jenga. Applying one patch may expose a bypass in another. Two fixes may touch the same code and conflict outright. A patch for one issue can inadvertently reintroduce a vulnerability you'd already closed. Pull the wrong block, and the tower you carefully assembled comes down.
This is why we don't patch vulnerabilities one at a time and try to stack the results afterward. We batch before we generate anything. For a given package, we gather the set of findings we intend to fix and patch them together in a single pass. Reasoning about the whole set at once is what lets us catch the fix that would expose a bypass in another, or the two fixes that fight over the same lines, while they're still being written, rather than discovering the collapse later when we try to reconcile independently generated diffs. The set, not the single CVE, is the unit of work.
It also shapes what we're aiming for. Because we're already looking at the package as a whole, the goal is to harden it so it comes back quiet on the next scan, regardless of what version of a frontier AI model you’re using. We want to fix the whole class.
Do the work once, apply it everywhere
Say you've earned a patch you trust for one version. You are not done because not all of your customers are on that version. That vulnerability could be applicable in other versions, too.
A secure patch is expensive to produce, so you want to reuse that effort and apply it reliably across every older supported version. However, a patch is a stubborn artifact. It encodes exact line numbers and surrounding context that drift from one version to the next, so "the same fix" rarely applies cleanly to a release from two years ago. Backporting is where much of the real labor lies, and it's the work you don't want to repeat by hand for every version in the support matrix. Much of the engine exists to carry a trusted fix backward across versions while re-running the full feasibility/regression/exploit gate on each one, because a patch that's correct on the latest release has earned exactly zero trust on an older one until it's proven there too.
The target keeps moving
And even once all of that holds, it doesn't stay held.
Upstream keeps moving. Maintainers eventually disclose their own fixes; new versions land; advisories get amended. An engine that patches zero-days lives on top of all that churn, which means our patches can’t be one-and-done. When upstream ships an official fix for something we'd already patched ourselves, the right move tends to be withdrawing our interim patch and replacing it with the disclosed one. The maintainers' fix is, again, the better trust anchor the moment it exists. So the engine has to track the state of the world, notice when reality has moved, and reconcile, meaning we have to retire superseded patches, adopt upstream ones, and re-validate the package so nothing regresses in the swap. It behaves less like a script that runs once and more like a controller continuously driving each package toward its best available fixed state.
This is where we’ve designed the pipeline to be most autonomous. It’s our solution for our remediation engine to decide what's safe to ship. What keeps that accuracy sharp is our internal checks. When the system encounters something it can't confidently resolve, it stops and asks a human. Drawing that line well, autonomous where the judges have earned it and human where they haven't, is something we tune constantly.
Shipping a fix you can trust
A patch that never reaches a customer's build protects no one, so the last mile matters as much as the fix itself.
We rebuild remediated artifacts from source rather than shipping a loose patch file. This is backed by the same philosophy as the rest of our supply chain work. Provenance and malware prevention start at the build, not the download. Remediated builds carry a distinct version suffix, so they're unambiguous and sort correctly. Our public backports use a cgr.N marker (for Chainguard remediated), and Athena's pre-disclosure remediations use cgp.N (for Chainguard private).
Above all, a security patch that doesn't apply cleanly has to fail the build and never slip through as a warning. The alternative is shipping something that looks remediated, reports as remediated, and remains vulnerable. Security patches have to be fatal. If the patch doesn't apply, the build fails loudly.
Finally, a synthesized fix still needs a verifiable identity. Each remediated build ships signed provenance and an SBOM that records the base version it derives from, the vulnerabilities it addresses, and its relationship to the upstream artifact. A fix with no upstream lineage still has an auditable one. And because a fix nobody's scanner recognizes looks exactly like an unpatched package, we publish the data scanners need to tell 1.2.3 (vulnerable) from 1.2.3-0.cgp.1 (not), the same way we do for our other remediated libraries.
The point
None of the hard parts here were about getting a model to produce a patch. Proving a fix is correct with no upstream fix to check it against is what's hard. So is keeping a tower of interdependent patches standing, carrying trusted fixes across a support matrix without re-earning them by hand, and keeping all of it reconciled against a world that won't hold still. Plus, all of the above must be done inside an environment locked down tightly enough to run untrusted code and automated everywhere it's safe to be. This shit is hard.
Our philosophy is the same one that runs through everything. We would rather ship fewer fixes we can stand behind than more we can't. Sometimes that means we can't patch something yet, and saying so is the honest answer. As models get better at finding vulnerabilities, a lot more of the world is going to need fixes that don't exist yet. Building the machine that can produce those fixes and prove them is hard. That's rather the point. Verifiable trust is and always will be hard.
Check out Athena to see how we can help you deal with the increasing number of AI-discovered vulnerabilities.
Share this article
Related articles
- engineering
This Shit is Hard: How Chainguard is sandboxing Athena
Matt Moore, Co-founder and CTO
- engineering
You can't trust what you can't see: How we keep an eye on a fleet of AI agents
Taylor Bloom, Senior Technical Program Manager
- engineering
This Shit is Hard: Getting software to run on robots (and then getting the robots to work)
Eric Timmons, Chief Engineer, Asylon Robotics
- engineering
This Shit is Hard: How AI keeps our code on standard
Maxime Gréau, Principal Software Engineer
- engineering
Faster than the advisory
Patrick Smyth, Principal Developer Relations Engineer, and Adrian Mouat, Staff Developer Relations Engineer
- engineering
The hardest fork
Dan Lorenc, Co-founder and CEO