Benchmarking Secure-and-Functional Remediation and How Snyk Agent Fix Lifts Frontier-Model Fix Rates by over 14%
August 18, 2026
0 mins readSummary
We benchmarked how well leading models produce vulnerability fixes that are both secure and functional, across ~150 real vulnerable code samples in JavaScript, Java, and Python. We ran each model on its own and with Snyk Intelligence (the new agentic Agent Fix architecture). The headline findings:
Out-of-the-box frontier models cluster at 72–75%. Gemini 3.1 Pro, Claude Sonnet 4.6, and Claude Opus 4.6 all land within a few points of each other. On secure-and-functional fixing, model choice barely moves the number.
Snyk Intelligence moves the same models clear of that cluster. Opus 4.6 rises from 74.6% to 85.4%, a 10.8-point gain (14.48% more samples fixed). The variable is the security context, not the model.
The lift is largest where the model is weakest. Opus alone fixes only 64% of Python samples; with Snyk Intelligence, that reaches 88%.
Introduction
Coding-agent benchmarks exist for unit-test generation, SWE-bench-style bug fixing, and code completion. There is no widely used public benchmark for the task a security team actually cares about: take code with a known vulnerability and produce a fix that removes the vulnerability and keeps the code working. Those are two independent bars, and clearing one while failing the other is a common, expensive failure mode. A fix that removes a SQL injection but changes what the query returns has not helped anyone. A fix that looks clean but leaves the injection in place is worse, because it reads like a solution.
So we built a benchmark that scores both bars on every fix, and ran frontier models through it twice: on their own, and equipped with Snyk's security intelligence. The question we wanted answered in plain terms: how much does Snyk's security context change what a frontier model can fix, and where?
The short version: out-of-the-box models have plateaued around 72–75%, and Snyk Intelligence is what moves them past it. The rest of this post is the data and the method behind it.
How we measure: the Golden Test benchmark
Most code benchmarks check whether the code runs or whether it resolves a bug report. Neither is enough for security remediation, which has to clear a security bar and a functional bar at once. Our evaluation set, the Golden Tests, is built to measure both. The design is inspired by SWE-bench, adapted for security.
The fixtures
The set is ~150 real vulnerable code samples: 50 in Python, 54 in JavaScript, and 39 in Java. Each sample is a piece of code with exactly one vulnerability, found by Snyk Code and confirmed by a human security expert, and chosen so it is fixable from the code in front of the model without missing external context.
The scoring
Each sample ships with two human-verified unit tests:
A test that fails because the vulnerability is present, and
A test that passes when the code's original functionality is preserved (for example, a helper that should echo 'hello world' still returns 'hello world' after the fix).
To score a pass, the model must fix the code so that both tests pass, on the first try, without ever seeing either test. The model never sees the unit tests themselves, so a pass reflects a genuinely secure and functional fix rather than output fitted to a known test.
Take a Python sample with a SQL injection: the code builds a query by concatenating user input. The security test sends a SQL injection payload and asserts that the database does not leak every row; the vulnerable code fails the test. The functional test sends an ordinary username and asserts the correct record comes back; the original code passes it. A fix counts only if the model's rewrite makes the security test pass while keeping the functional test green, on the first attempt, tests unseen.
What we evaluated
Six configurations: Snyk's previous, internal StarCoder-based Agent Fix model, three out-of-the-box frontier models (Gemini 3.1 Pro, Claude Sonnet 4.6, Claude Opus 4.6), and Sonnet 4.6 and Opus 4.6 each running with Snyk Intelligence under the new agentic Agent Fix architecture. "Snyk Intelligence" here refers to dynamic few-shot prompting: at a fix time, we inject the most relevant expert-written fixes for that specific weakness, drawn from Snyk's database of more than 35,000 vulnerabilities. (This builds on earlier work where the same idea improved the performance of off-the-shelf LLMs.)
How does the Snyk Agent Fix benchmark compare to prior benchmarks?
The Golden Test set sits in a lineage of work that has steadily raised the bar for evaluating AI on code. SWE-bench established the pattern of scoring models against hidden, real tests rather than self-reported plausibility. On the security side, Vul4J introduced reproducible vulnerabilities paired with proof-of-vulnerability tests and a functional regression suite, the closest precedent to our FAIL-to-PASS plus PASS-to-PASS design. More recent work, like BaxBench and SEC-bench, reinforces the premise we built around: code that is functionally correct is frequently still insecure, so a credible remediation benchmark has to score both properties at once. What distinguishes the Golden Test set is applying both gates together, on human-verified real samples, with the tests withheld from the model, across three production languages.
Results
The headline metric is the share of Golden Tests where the fix was both secure and functional.
Configuration | Functional & secure fix rate |
|---|---|
StarCoder (previous Agent Fix model) | 72.4% |
Gemini 3.1 Pro | 74.2% |
Claude Sonnet 4.6 | 72.4% |
Claude Opus 4.6 | 74.6% |
Claude Sonnet 4.6 + Snyk Intelligence | 82.5% |
Claude Opus 4.6 + Snyk Intelligence | 85.4% |
CHART 1: Functional & secure fix rate.
The out-of-the-box models sit within a three-point band. Adding Snyk Intelligence opens an 8-to-11-point gap on the same model: Opus 4.6 goes from 74.6% to 85.4%.
Breaking down the Opus comparison by language shows that the gain is not an averaging artifact. It holds in every language we tested, and it is the largest where the out-of-the-box model is weakest.
CHART 2: Per-language lift
Python is the clearest case: Opus alone fixes 64.0% of samples, and with Snyk Intelligence, that jumps to 88.0%. JavaScript and Java, where Opus already starts strong, each gain five to six points.
What the numbers mean
Raw model size has plateaued on secure-and-functional fixing
The three out-of-the-box models span 72.4% to 74.6%, a 2.2-point spread across two vendors. If a bigger or newer model were the lever for this task, we would expect to see it here. We do not. The task is hard in a way that more general capability does not directly address: the model has to know what a secure fix for this weakness looks like, not just write plausible code.
Security context, not a bigger model, is the lever
The same Opus 4.6 gains 10.8 points (14.48% more samples) purely from the security examples injected at fix time. Because the approach is model-agnostic, every gain in the underlying frontier models compounds with that context rather than competing with it. The durable asset is the 35,000-plus expert fixes; the model is a component we can swap as the field moves, which is why production Agent Fix now pairs Snyk Intelligence with Claude Opus 4.7.
The lift is largest where the model is weakest
Opus alone fixed only 64% of Python samples, its worst language. With Snyk Intelligence, it reached 88%, its best gain of the three. The security context does not just raise the average; it lifts the floor.
The result holds up to a second look
The Opus-plus-Snyk number is an average across runs (84.6% and 86.0% on the two runs we aggregated), so the headline 85.4% reflects consistent performance across runs. Run-to-run variance on this set is roughly a point, which is worth keeping in mind when comparing configurations that are within a point or two of each other.
Up next: Snyk VulnBench and vulnerability detection with coding agents
Fixing a vulnerability presumes you found it. In June 2026, we published the Snyk VulnBench JS 1.0 paper, which aimed at benchmarking Snyk Code as a deterministic and fast SAST engine vs LLMs powered by coding agents (the Claude Code harness) to detect vulnerabilities in code to begin with, before they need remediation.
Our Snyk VulnBench findings discovered that even frontier large language models such as Claude Opus 4.7 on its maximum reasoning level (aka max), and even when powered by an advanced coding agent harness (Claude Code itself), had faced challenges in and around repeatability and determinism. Some headline findings:
In one case, the model and coding agent reported ~50% of findings that did not repeat in four out of the next five executions, producing a backlog of false positives and vulnerability fatigue for agentic developers and AI security engineers.
In other cases, 13% of the coding agents reported that unmatched vulnerabilities appeared in all 5 executions, proving an even more confusing and cognitive load situation in terms of the security backlog of issues that could prove false positives.
We invite you to investigate and explore through the Snyk VulnBench dataset, which has been made online and public for you to review at https://vulnbench.com/
Limitations
Four limits a reader should weigh before leaning on these numbers:
Set size: ~150 samples (50 Python, 54 JavaScript, 39 Java) is enough to surface clear patterns, but not to claim statistical significance on small gaps. The StarCoder baseline ran on a roughly 20% smaller sample (Agent-Fix-supported rules only, where training data existed); across all rules, it scores 54.9%. It is Snyk's internal, fine-tuned model, included for reference rather than as a frontier baseline.
Snippet-level, not whole-application: Each sample is a single file with one vulnerability, fixable from the local context. Real codebases have cross-file context and multiple interacting issues; this benchmark does not measure that.
Three languages: We benchmarked JavaScript, Java, and Python. The new architecture supports all the languages Snyk Code supports, but these are the three with Golden Test coverage today.
Limited runs for variance: We aggregated two runs for the Snyk-boosted configurations and have not run enough repetitions to put formal error bars on every cell. Treat sub-two-point differences as approximate.
We name these to help calibrate which claims to lean on, not to hedge the findings. The 10-point-plus gap from Snyk Intelligence is well outside the run-to-run noise; the per-language ordering of small gaps is not.
What's next
Broaden language coverage: Extend Golden Test sets beyond JavaScript, Java, and Python so the benchmark matches the architecture's full language support.
Quantify variance: Run each configuration enough times to publish proper error bars rather than a two-run average.
Detection, not just remediation: This benchmark measures fixing. A companion effort measures how well agents find vulnerabilities compared to Snyk Code, and we will report that separately.
The new agentic Agent Fix is now rolled out, pairing Snyk Intelligence with Claude Opus 4.7. Want to see secure and functional remediation on your own code? Start with Snyk Code and Agent Fix, and learn more about the engineering behind these numbers.
Can You Trust AI Code? I Built a Scanner to Find Out
Appendix: methodology and aggregation
Eval structure: Each Golden Test consists of a code sample containing exactly one vulnerability, plus a security unit test (that fails on the vulnerable code) and a functional unit test (that passes on the original code). A configuration passes a sample only if its fix makes both tests pass on the first attempt, with the tests withheld from the model.
Aggregation: Reported rates are the share of samples fixed.
The previous model (StarCoder) figure of 72.4% is the mean of its per-language rates, and reflects Agent-Fix-supported rules only; across all rules, it falls to 54.9%.
The Opus 4.6 + Snyk Intelligence figure of 85.4% is the average across runs (run 1 = 86.0%, run 2 = 84.6%). Per-language figures in the second chart are from a single representative run, which is why they average slightly higher (~86%) than the multi-run headline; the headline number is the one to cite.
Configurations: StarCoder (previous Agent Fix model); Gemini 3.1 Pro; Claude Sonnet 4.6 and Claude Opus 4.6, each out-of-the-box and with Snyk Intelligence under the agentic Agent Fix architecture. The data here was collected on Opus 4.6; production Agent Fix has since moved to Claude Opus 4.7. Snyk Intelligence injects expert-written fixes for the specific weakness at generation time (dynamic few-shot prompting).
See Snyk in action
See why Snyk is the chosen AppSec solution for developers and security teams alike — and what it can do for your team.