Fighting AI Slop in Production Codebases: How Huntress Improved Fable 5.1’s API Recall

Testing coding agents against codebase features revealed a significant gap in their ability to utilize existing framework APIs, quantified by API recall, which was 48% for the initial evaluation of Claude Fable 5.1. The team conducted experiments to determine if this accuracy could be improved and developed a testing harness based on real Rails code. Initial attempts to improve performance involved adding simple instructions to the agent's context, such as an explicit rule about preferring framework APIs in prompts. Further improvements involved implementing external tools, like a lookup tool querying installed gems, and incorporating a stop-time reviewer model that checks generated code against source material before completion. These layered techniques demonstrated that explicitly guiding the agent alongside providing post-generation verification led to substantial gains in API recall, moving from a baseline of 48% to 100% accuracy across specific tasks through iterative experimentation.

The narrative illustrates a critical tension between emergent LLM capability and grounded, verifiable engineering practices within complex systems. The observation that coding agents tend to generate "AI slop"—reimplementing functionality rather than utilizing established framework APIs—suggests that current agentic workflows prioritize pattern matching over structural understanding when operating on unfamiliar or subtle context. The progression from a low baseline to near-perfect recall by introducing specific, multi-layered interventions (contextual rules, external tooling, and post-generation review) highlights that performance improvement is achieved not through raw model power alone, but through architecting an external scaffolding that forces the agent toward verifiable sources of truth. This shifts the focus from simply asking a model to generate code to building reliable feedback loops within the coding environment itself. The pattern observed here echoes the principle that opaque abstraction, whether in code or AI prompting, allows for unmanaged accumulation of redundant, unmaintainable complexity unless explicitly constrained by external governance mechanisms. The implication is that achieving true engineering maturity requires designing constraints that make the "slop" immediately detectable and correctable, moving beyond anecdotal evaluation toward demonstrable system-level accountability.