We ran OpenAI's GPT-6 Astra through Codex on the same real-world coding tasks we use for the Agent Security League. The result is the largest generation-over-generation jump we have recorded in the Codex family: 82.1% FuncPass and 34.6% SecPass, versus 67.6% and 20.1% for Codex + GPT-5.6 Sol. That puts Astra within a few points of Claude Code + Fable 5.1 (87.2% / 36.9%) and ahead of Claude Code + Opus 5 (73.7% / 37.4%). It gets there with zero confirmed cheating, 16 instances flagged, 16 cleared,so the published score is not a memorization haircut. The trade-off is time and tokens: Astra is slower and heavier than GPT-5.6 Sol, and it is thus slower than Fable 5.1 as well, Fable posts the slightly higher score in about half the wall-clock time, with zero timeouts. This post walks through the scores, that duration and token trade-off against both the Codex predecessor and Fable 5.1, a harness-agnostic look at tool use, and the clean anti-cheating result.
Key takeaways
- The biggest Codex leap we have measured. Codex + Astra reaches 82.1% FuncPass and 34.6% SecPass, +14.5 points on both axes over Codex + GPT-5.6 Sol.
- Close on the board, not on the clock. It trails Claude Code + Fable 5.1 by 5.1 FuncPass and 2.8 SecPass points, and leads Claude Code + Opus 5 on both axes, but Fable 5.1 finishes in ~9.5 min per task with zero timeouts, while Astra takes ~16.8 min and hits the cap 29 times. Fable 5.1 runs at GPT-5.6 Sol's speed with a near-Astra score.
- Zero confirmed cheating. The pipeline flagged 16 instances, inspected all of them, and confirmed that none were issues. GPT-5.6 Sol had 1 confirmed cheat; Fable 5.1 had 17; Opus 5 had 38.
- Slower, not idle. Versus GPT-5.6 Sol, the extra ~1.8× wall-clock and ~1.6× tokens buy the +14-point jump (extra input, not extra output). Timed-out Astra runs still produced patches, Codex ships whatever diff it has at the kill boundary.
- Similar command count, better conversion. Astra and GPT-5.6 Sol issue a similar number of shell commands. Astra searches more, edits less, and leaves fewer leftover shell calls, the gap is what it does with the budget, not how many times it acts.
Introduction
GPT-5.6 Sol through Codex was a clean, mid-board result: competitive FuncPass, modest SecPass, and almost no confirmed cheating. GPT-6 Astra is the next model we ran in that same harness. The version number suggested a step; the scores are a jump.
Two comparisons matter. Against its Codex predecessor, Astra is not a small bump, SecPass nearly matches the gain in FuncPass, both 14.5 points. Against the Claude Code combos we published alongside this run, Astra closes most of the gap with Fable 5.1 and overtakes Opus 5, while drawing none of the memorization discount that still hangs over those Anthropic runs.
It does not do this by finishing faster. Astra takes longer than GPT-5.6 Sol and longer than Fable 5.1, Fable is the combo that pairs a top-of-board score with Sol-like speed. The rest of this post is about whether Astra's extra work is exploration or thrashing, how that looks compared to Fable 5.1 in terms of clock time and tokens, and why the anti-cheating pipeline found nothing to take away.
Benchmark recap
Skip if you know this already.
We measure combos, a harness (Claude Code, Cursor, Codex, …) plus a frontier model, on coding tasks inside real, complex projects. Each task involves code that was historically part of a security fix, but the combo is never told this; it is only asked to follow security best practices.
Each combo runs once per task, and we apply its patch in an isolated Docker environment. FuncPass means the patch passes the functional tests. SecPass means it also passes the hidden security tests introduced by the original vulnerability fix, so a secure result must first be functionally correct.
We also require the fix to come from the combo's own reasoning: recovering the known fix from git history, the web, a workspace copy, or training recall is treated as cheating. A multi-signal pipeline flags suspicious instances, and an LLM adjudicates each one. Confirmed cheating is removed from the score.
The benchmark also contains a small set of overly strict instances: their security tests demand implementation details that are exceptionally unlikely to be guessed independently. Rather than discard those instances entirely, we exclude them from the leaderboard denominator, they are not a fair measure of general secure coding ability, but keep them as traps. A pass is a tripwire for deeper inspection rather than automatic credit, and the agent’s trajectory can reveal a recall pattern that ordinary final-patch comparison misses. That is what happened in this run. For recall-then-diverge, we now also grade the edit trajectory, first/peak security overlap, suddenness, and string chronology, so early recall remains visible even after the final patch has diverged.
Results
Astra is the strongest Codex combo we have evaluated with 82.1% FuncPass and 34.1% SecPass, and it sits just behind Fable 5.1 on the published board.
Comparison
The same-harness comparison with GPT-5.6 Sol is the one that shows the leap. Fable 5.1 and Opus 5 are the Claude Code reference points from the same scoring.
- Astra vs GPT-5.6 Sol (same harness): FuncPass and SecPass both at 14.5 points. That is the largest generation-over-generation move we have recorded for Codex.
- Astra vs Fable 5.1: FuncPass −5.1 points, SecPass −2.8 points. Close enough that the cheating column matters: Fable 5.1's lead survives a 17-cheat discount; Astra had nothing to remove.
- Astra vs Opus 5: FuncPass +8.4 points, SecPass +2.2 points. Opus 5's raw security number looked higher until 38 confirmed cheats were taken out.
With and without memorization: irrelevant, as there was zero cheating
There is no cheating for Astra, so with or without memorization, the scores were the same.
The pipeline flagged 16 instances as suspicious for Astra. All 16 went to LLM inspection. None were confirmed as cheating.
That is not the pipeline taking a holiday. Codex's sandbox keeps the agent within a workspace directory with an approved command set, which removes the easy leakage channels, git log / git show, web fetches of upstream patches, reading a fixed copy from the environment,. that still appear in more permissive harnesses. Combined with a model that did not regurgitate golden fixes, the inspector had 16 flags and 16 explanations of independent work.
The cheating column in the results table is doing real work in the cross-family comparison. Fable 5.1 and Opus 5 remain strong after discounting 17 and 38 confirmed cheats; Astra never took that haircut. On this dataset, the OpenAI Codex runs have been consistently less memorization-prone than the Anthropic Claude Code runs
Longer than GPT-5.6 Sol, and than Fable 5.1
Astra is about 1.8× slower per task than GPT-5.6 Sol and has nearly three times as many timeouts. It is also ~1.8× slower than Fable 5.1 (16.8 min vs 9.5 min mean; 11.3 vs 7.2 median). Fable 5.1 and GPT-5.6 Sol sit on top of each other on the clock, and then Fable converts that Sol-like speed into a score that still leads Astra. Fable 5.1 is the reliability reference: every task finished, none hit the cap.
GPT-5.6 Sol is front-loaded: 42% of tasks finish under 5 minutes, 90% under 20. Fable 5.1 is almost as front-loaded (37% under 5 minutes, 92% under 20) and then stays finished: 97% under 30 minutes, a single task past 40 minutes, zero timeouts. Astra is the opposite tail, only 18% under 5 minutes, 74% under 20 minutes, and 14 tasks sitting in the 50–60-minute bucket at the one-hour cap.
That 50–60 minute pile is where the timeouts live, not a missing-prediction pile. Codex extracts a diff even when the process is killed, and every timed-out Astra task still produced a scored patch. Of those 29, 22 still passed the functional tests and 12 still passed the security tests. Fable 5.1 never needed that salvage path.
The three-way split is the useful one. GPT-5.6 Sol is fast and weaker. Astra is slow and close to the top. Fable 5.1 is fast and ahead, the remaining 2.8 SecPass points are not something Astra buys with extra wall-clock time.
A timeout event in the Codex log does not mean that no prediction was created. It simply means the process was killed at the one-hour limit before Codex could finish the session. These runs are included in the 50–60-minute bar and are not counted as a separate “missing prediction” category.
More tokens than Sol, more output from Fable 5.1
Versus GPT-5.6 Sol, Astra uses about 1.6× the tokens, in line with the 1.8× wall-clock gap. The extra is input, more context, more exploration, not a flood of generated code:
In terms of output token dimension, Fable generates 6.24M tokens compared to Astra's 1.71M (~3.7×) and still finishes in about half the time. Astra is not slow because it writes more, it writes about as much as GPT-5.6 Sol. Fable 5.1 writes more and finishes sooner.
Versus Sol, extra time and extra input buy a large score jump. Versus Fable 5.1, Astra is already slower, does not out-write it, and still trails.
Finally
If you're interested in some opinions and conclusions from the these latest model testing runs, check out our key takeaways.
What's next?
When you're ready to take the next step in securing your software supply chain, here are 3 ways Endor Labs can help: