Image: datocms-assets.com · rights & removal

How to Evaluate AI Agent Cybersecurity Capabilities

Reporting by Wiz BlogRead the original at wiz.io

Executive Summary

The methodology for benchmarking cybersecurity agents involves first defining a clear objective before designing the tasks, environment, and measurement. The process requires establishing the 'why' (objective), the 'what' (data, ground truth, agent setup), and the 'how' (metrics, evaluation function) before creating any specific challenge. A core principle is to build data from real distributions rather than AI-generated content. Grading must be done deterministically using code rather than relying on LLMs as judges, requiring isolation measures to prevent cheating during evaluation. Success optimization requires evaluating the harness and model together, incorporating multiple attempts, and considering the cost of a solution alongside the success rate.
The benchmark structure relies on four task categories—code vulnerabilities, API security, web security, and cloud security—each weighted equally based on existing real-world sources. Empirical results show varying agent performance across these domains, with some agents scoring higher in specific areas, suggesting that model choice is intertwined with the environment provided by the harness. Furthermore, incorporating multiple attempts demonstrates that extending the time and resources for an agent often leads to better solutions, emphasizing a trade-off between accuracy and cost.

Facts Only

* The benchmark methodology requires fixing the benchmark objective before building tasks, environments, and measurements.
* Data must be built from real distributions rather than AI-generated or labeled tasks.
* Grading is performed with code for deterministic measurement, isolating agents from servers and internet access to prevent cheating.
* Evaluation should optimize the harness and model together and account for the cost of a solution.
* The benchmark consists of four categories: Code vulnerabilities (221 tasks), API security (54 tasks), Web security (29 tasks), and Cloud security (23 tasks).
* Averaged performance across agents shows pass@1 scores of 78.0% on API security, 64.6% on web, 56.5% on cloud, and 38.2% on code vulnerabilities.
* Agents tested on Claude Code generally preferred the Claude Code harness, while Gemini 3.8 Flash Cyber preferred the ADK React harness.
* Three attempts per challenge were provided to agents.
* The final analysis notes that success is plotted against what a solve costs, identifying agents on the Pareto frontier as strictly better for a given budget and success rate.

Full Take

The emphasis on defining the objective—specifically whether an agent can perform a real attack based solely on instructions within a time/knowledge constraint—shifts the benchmark from measuring passive knowledge to active, verifiable execution in a production context. The tension between LLM-as-a-Judge and deterministic code grading reveals a fundamental resistance to outsourcing judgment in high-stakes tasks; relying on an opaque model introduces systemic debt. The finding that agents perform differently across cybersecurity categories highlights that "general" agent performance masks domain-specific vulnerabilities, suggesting that generalization is insufficient for real-world defense. Furthermore, the necessary step of measuring cost against success exposes a critical divergence: achieving higher rates does not automatically equate to superior performance when efficiency and resource management are introduced; this forces the focus onto operationalizing security capabilities rather than mere theoretical success metrics. The structure suggests a necessary shift toward system-level evaluation where the environment and constraints (the "how") are as important as the agent's output.

From the original · Wiz Blog

How do you measure the success with which different agents perform regular, real-world cybersecurity tasks? Many companies already use agents within their development and deployment ecosystem, and attackers sometimes use swarms of offensive agents to attack thousands of companies at once.

Read the full story at wiz.io

Sentinel unavailable

The automated check of the source article's wording did not complete, so no result is shown.

This looks only at the wording of the original source article, not at this page's AI-written sections. A small local AI model made this estimate. It has not been checked against known human and machine texts, so treat it as provisional. It cannot show who wrote an article.