I read two posts about AI hunting for vulnerabilities and kept asking myself, “Could the same swarm of agents be turned around to build the detections that catch those attacks?” Here is what we built, what broke, and where it goes next.
Every detection engineer does the adversary's job in reverse. The attacker asks, “How do I get in and leave no trace?" The defender asks, “What trace would that leave, and would I catch it?" Same board, opposite sides. If you have followed anything I have written, you know where I land. The best way to defend something is to think like the person trying to break it.
So, when AI started getting good at the attacker's half of that job, I could not stop thinking about the defender's half. This is the story of what we did about it. It is an honest account because the system we run today did not show up clean and finished. It started as a mess of scripts, got automated in pieces, leaned hard on humans, and only slowly turned into the system for Automated Detection Engineering we like to call FORGE. I am not going to walk you through our detection logic or the secret sauce of any specific rule. What I will give you is the approach, the thinking, what worked, what blew up, and where we are taking it.
Key takeaways
- Defense, mirrored from offense. Automated detection engineering takes the multi-agent approach researchers use to hunt vulnerabilities (Cloudflare's Project Glasswing; evilsocket's audit) and points it at the opposite job, building the detections that catch those attacks.
- Many narrow agents, not one big one. "FORGE" runs a staged pipeline — plan, generate, dedupe, adversarial validation, red-team, reachability, report — with a separate adversarial reviewer, instead of one model checking its own work.
- Proven against real data. Candidates are tested against real customer telemetry in Mitiga's Cloud Security Data Lake — and soon against live, safely simulated attacks — not just unit tests.
- Humans stay in the loop. FORGE never auto-deploys; it opens a pull request a human reviews and merges. More than 1,000 validated detections have shipped this way over the past several months.
Where the idea came from
The spark was not mine. I read Cloudflare's writeup on their “Cyber frontier models” / Project Glasswing work, pointing a swarm of narrow AI agents at finding software vulnerabilities at scale. Then I went down the rabbit hole into evilsocket's open-source “audit” harness, a multi-stage pipeline on the Claude Agent SDK that takes the same idea and makes it concrete enough to poke at.
Both are pointed at offense, finding the bug and proving the exploit. And I kept turning it around. Vulnerability hunting and detection engineering are mirror images of each other. One hunts the way in, the other hunts the trace it leaves behind. So, the question almost asked itself. Could you take that same multi-agent harness and flip it from hunting vulnerabilities to building detections?
Why it isn't as easy as it sounds
A vulnerability hunter has a beautifully clean scoreboard. The exploit either pops a shell or it does not. Detection engineering hands you nothing that clean for free. A detector can look perfect and be useless. It might never fire because the events it keys on do not exist in real data. It might fire 10,000 times a day on activity that is completely benign. “Compiles and passes its unit tests” is a long way from “actually catches the attacker and does not bury your analysts.”
And our detectors are not loose scripts. They carry strict metadata, follow a style guide, map to MITRE ATT&CK, and have to clear the same quality rules a senior engineer would enforce in review: dynamic severity, corroborating signals before anything is allowed to scream HIGH, naming and layout conventions, and a dozen more. Ask a model to just write a detector, and you get something that looks great and trips over ten of those rules at once.
Whether a model can write a detector was never the hard part. The hard parts were these. How do you give a swarm of agents a real scoreboard? How do you turn good detection into something an adversarial reviewer can actually check, line by line? And how do you do both without a human standing over every single candidate?
The road to FORGE: messy, then automated, then orchestrated
FORGE was not the beginning. No, our automated detection engineering is the third act of a longer story, and the first two acts are why it works now.
Act one, messy and manual.
In the beginning, it was a pile of disconnected scripts and a lot of human judgment. An engineer picked a gap, researched the technique, hand-wrote the detector, checked it against customer data by hand, and walked it through review. High quality, completely effective, painfully slow. Output was capped at headcount.
Act two, automations and skills.
We started encoding the parts we kept repeating. Coverage-gap analysis, schema lookups, test scaffolding, review checklists, the data-reachability checks—each one became a reusable, documented capability, a skill we could call instead of rebuilding from scratch. Humans were still on every detector, but the loop got shorter and far more consistent. What felt like overhead at the time became the foundation. This phase forced us to write down what “shippable” means as explicit, machine-readable rules. That rulebook is the exact thing our adversarial reviewer reads today.
Act three, the harness.
Once the steps were automated and the bar was written down, we stopped running the steps by hand. Many narrow agents now run them in parallel, with a separate adversarial reviewer in front of the output. That is automated detection engineering.
You cannot orchestrate what you have not first taken apart. The months we spent turning tribal knowledge into explicit skills and explicit rules are the foundation the harness stands on.
How Automated Detection Engineering works
The core bet matches the public research. For a problem with the right shape, many narrow agents plus a separate adversarial reviewer beat one big agent grading its own homework. Detection engineering turned out to have that shape. Each task is naturally narrow, covering one vendor, one log type, one technique, and one event pattern. The rules are explicit. The output format is rigid. And with the right plumbing underneath, you can make validation real.
We referred to it as FORGE because that is how it behaves. Raw material in, a finished and tempered blade out. A run moves through a sequence of stages, each one owning a single job:
- Threat-intel and planning. Survey the relevant ATT&CK techniques and the data schemas, then build a ranked queue of hunt tasks that goes after real coverage gaps first.
- Generate. Many agents, in parallel, each draft one candidate detector for one task.
- Dedupe. Kill candidates that duplicate detectors we already have or work already in flight before we spend a cent validating them.
- Adversarial validation. Two independent reviewer passes (a heavier model, then a faster one) hammer each survivor against the written rules.
- Red team. Another pass tries to break it, enumerating the benign activity that would set it off.
- Reachability. The scoreboard. Run the survivors against real customer data across multiple environments and make them actually fire inside sane noise limits.
- Report. Hand back a single, review-ready pull request containing the detectors, the metadata, and the evidence behind every one.
Different stages run on different models on purpose, heavier where judgment matters, and cheaper where speed does, so cost tracks the difficulty of the work instead of the other way around. One more thing about scope, because people assume the worst. FORGE does not deploy to production, and it does not replace the reviewer. It opens a pull request. A human reviews it once and merges. Humans stay in the loop. Their job changes from author of every detector to approver of a batch.
The data underneath is the hard part
People reduce all of this to “just point agents at the data.” That gets it backwards. The agents are the easy part. The data being ready for them is what makes the whole thing work.
That reachability scoreboard, the thing that turns “looks right” into “provably fires on real attacks and does not drown the SOC,” only works because we spent years building our Cloud Security Data Lake where customer telemetry across a lot of cloud and SaaS platforms is already normalized, enriched, partitioned, and queryable on demand. An agent can effectively ask which environments even carry this kind of event, does my candidate fire there, and how loud is it, and get a real answer in minutes instead of starting a small research project. That is what replaces guesswork with a verdict. It is also the single biggest reason this approach works for us and would be miserable to bolt onto a raw, un-prepped logging stack. The harness is the visible part. The prepared data underneath took years to build, and it is the hard part to copy.
What worked, and what blew up
I would rather tell you about the failures than the wins. They teach more.
Early runs threw away good detectors for dumb reasons. Our first end-to-end runs dropped solid candidates, not because the detections were wrong, but because the reviewer model wrapped its verdict in a friendly paragraph and our strict parser gagged on it. We were failing real work on a formatting technicality. The fix was a more forgiving parser and tighter output contracts. In a multi-agent system, the plumbing between the agents matters as much as the agents.
"Passes review" is not the same as "actually runs." We hit candidates that cleared validation and then failed to even load against real data, burning compute on a run that was dead on arrival. So we added deterministic gates that confirm a candidate truly imports and loads before it is ever allowed near a cluster. Now a structurally broken detector gets caught for free and early, instead of expensively and late.
A reviewer that demands perfection ships nothing. Our first red-team agent rejected almost everything that reached it, because almost every useful detector carries some false-positive risk and some way to slip past it. The question is how much. So we rebuilt it to be quantitative and constructive. It estimates the risk, suggests a concrete tightening, advances with a caveat the human can see, and saves an outright kill for the genuinely hopeless. An adversarial reviewer that wants perfection is just an expensive way to produce zero detections.
Fixing beats discarding. The single biggest jump in quality came from teaching the stages to repair a candidate instead of dropping it. When one gets rejected, a cheaper repair turn rewrites it, and we explicitly tell it which rules it already passed, so a fix for one problem does not break another. Bounded attempts, the cheapestcheapest model that can do the job, fully automatic. That one change moved us from playing whack-a-mole and losing candidates to converging on shippable ones.
How we keep sharpening it
We tune FORGE constantly. Every run records what it cost and why each candidate lived or died, broken down by which gate did the killing, so we can see whether the bottleneck is generation, validation, or reachability, and aim there. Per-vendor lessons pile up in a running log that feeds back into the next run on that vendor, so the system gets sharper on a platform the more it works on it. And because our rulebook is the live source of truth, the pipeline notices when the rules have drifted from the version it was validated against and flags it, instead of shipping against a stale standard.
The cumulative payoff of this whole arc, the hand craft, the automations and skills, and now the harness, is more than 1,000 validated detections shipped over the past several months, at a bar that survives human review. The bottleneck has moved off of writing detectors and onto reviewing them, which is exactly where I wanted it.
What's next: validate a detection by becoming the adversary
The reachability scoreboard answers a strong question. Does this detector fire on the real activity already sitting in customer data? The next question is the one I have been chasing all along. Does it fire on the attack it was built to catch, on demand, even when nobody happens to have run that attack in the window we are looking at?
So soon, the pipeline will close the loop by doing the thing I keep telling everyone to do. Be the adversary. Before a detector ships, FORGE will not just go looking for the attack's footprint in old data. It will safely reproduce the adversary behavior the detector is meant to catch in a controlled environment, and then it confirms the detector fires on that freshly generated activity. A detector for a specific technique will get validated against an actual execution of that technique, not against whatever happened to be logged last month. It is the closest thing detection engineering has to the vulnerability hunter's did-the-exploit-pop-a-shell moment. Do not wait for the attack to show up. Perform it and watch the detection catch it.
That turns the validation gate from this detector matching real-looking data into this detector demonstrably catching the behavior it claims to. That is the standard I want every shipped detection to clear.
The takeaway
It is tempting to read all this as “AI writes our detections now.” It does not, and I would not want it to. What changed is the division of labor. Narrow agents do the parallel grunt work of drafting and checking. An adversarial process enforces a bar we were disciplined enough to write down. A real-data scoreboard, soon backed by live simulation, decides what is good. And humans review and own the result. The durable advantage is the years of prepared data under the agents and the discipline of turning good detection into something a machine can check. The agents are just what finally let us run the whole thing in parallel.
If you build detections for a living, I will leave you with the question that started all of this. When was the last time you proved a detection works by running the attack at it? Not in theory. Not against last month's logs. On demand. That, to me, is what being the adversary looks like when you have switched sides.
FAQ
What is automated detection engineering?
It is the use of a coordinated set of narrow AI agents to draft, validate, and test detection rules, with humans approving the result. It mirrors AI-driven vulnerability hunting in reverse. Instead of finding the way in, the agents build and prove the detections that catch it.
Does FORGE deploy detections automatically?
No, it opens a pull request, and a human reviews it once and merges. It does not deploy to production, and it does not replace the reviewer. Engineers move from authoring every detector to approving a batch.
How are AI-generated detections validated?
Every surviving candidate clears two adversarial review passes and a red-team pass that enumerates the benign activity it might trip on, then a reachability test that runs it against real customer data across multiple environments inside set noise limits. A live-simulation stage that reproduces the target attack and confirms the detector fires is in progress.
Can AI write detection rules on its own?
A model can draft a detector, but drafting was never the hard part. The durable advantage is the written rulebook an adversarial reviewer can check, the real-data scoreboard, and the years of normalized, enriched data underneath. The agents are what let it all run in parallel.