AI verdict correctness in our security operations center (SOC) is 92%, up from 60%, but we didn't switch models to get there. What we changed is the context the agents get before they decide anything, including the detection rule's investigation guide and user risk data from Workday, along with the closure reasons from 30 days of past cases on that same rule.

This post covers how the agentic SOC pipeline is built in Elastic Workflows and Elastic Agent Builder, down to the prompts and the feedback loop that lets an agent see where it got the same rule wrong last time.

Customer Zero: Running Agent Builder in our own SOC

At Elastic, our internal SOC operates as Customer Zero, meaning that we’re the first and most demanding user of every feature we ship. We run the newest versions of Elastic Security and Agent Builder in our production environment, often before they reach general availability (GA), across a globally distributed fleet of laptops, servers, and cloud workloads. The workflows and agent configurations shown in this post reflect our setup as of version 9.5.1.

When your AI SOC analyst is wrong 40% of the time

Our team dove in headfirst with AI agents and fully integrated our alerts with AI triage. When our agents were looking at only the current alert context and investigation indexes, they weren’t always correct. Actually, our logs showed accuracy hovering around 60%. It’s great to have this data, but not if the analysts can’t trust it.

We were adding long AI summaries to each case, what we would consider AI slop, as it was inaccurate 40% of the time. The feedback we got from the analysts was that they weren’t reading them. The analysts started ignoring the AI summaries completely since they couldn't trust that they were helpful or accurate. It took more time to read a paragraph of incorrect information than to just triage the case manually. The summaries were slowing analysts down without providing any benefit worth the additional token cost.

Leading with the data

Before getting too in the weeds, here’s the data. Our AI verdict correctness (based on comparing the AI verdict and the analyst close reason) went from 60% to 92% after implementing the changes we discuss in this blog. We’re tracking these metrics using Elastic dashboards by comparing the case custom fields that are discussed more below. This increase in accuracy meant that the analysts could start double-checking the summary and closing the case right away. This changed our AI summaries from being a time sink to allowing our analysts to close the case in one step.

What context AI alert triage actually needs

We significantly increased agent accuracy by feeding them more context. Here's what we pull in from each source before an agent makes a verdict:

When should you use an AI agent instead of a query?

It's important to know when to use AI and when not to. If the answer requires a predictable query with only a variable or two changing each time, don't use an agent. Instead, use an Elastic workflow that runs an Elasticsearch Query Language (ES|QL) query, a Kibana API call, or a GET request. They're faster and cheaper, and we keep them modular and reusable across many different orchestrators, so a UserDetailsLookup or PastCasesByRulenameLookup can be called from any workflow that needs it. Agents are more suited for tasks that require reading and reasoning that cannot be completed with a simple query; for example, analyzing past case comments for patterns.

We named our workflows to reflect the three types of activities in the main orchestrator:

• Lookup: ES|QL queries, Kibana API calls, and GET requests to external services.
• Agent: AI agents built in Kibana's Agent Builder.
• Action: POST requests to Kibana case comments and to Slack and other external services.

Investigation guide lookup: Per-rule triage instructions

Using the kibana.alert.rule.uuid

, we can do a GET request to /api/alerting/rule/\

to grab the investigation guide attached to the detection rule that triggered the alert. Here, we can provide context to both the analyst and our AI agents. Our detection engineers fill out these investigation guides when creating the detection, sometimes separating the analyst sections and AI sections with notes to follow. This gives us specific, per-rule instructions so the AI triages each alert type the same way every time. Here's where to find the investigation guide on a rule:

The user details lookup: Finding high-risk users

We look up user details in Workday using the user.email

field from the alert. This gives us context on high-risk users, for example:

• Users who have joined the company in the last 90 days.
• Users who are leaving the company in the next 90 days.
• Cost centers that carry elevated risk (for example, help desk, security researchers).

Past cases by rule name: What verdict did the analyst choose?

We look up alerts with the same kibana.alert.rule.uuid

from the last 30 days that have been tied to cases and closed. This gives our agents a historical baseline. If this rule has fired 50 times and 48 were false positives, that's critical context. An agent that knows the history of a rule makes significantly better verdicts than one that doesn't.

Using the cases we've seen in the last 30 days, we pull the fields that tell us how analysts actually closed them, the verdict they chose, their summary of the activity, and whether the AI got it right. These are the signals that close the loop:

The AI feedback loop: How agents learn from wrong verdicts

Using the data returned from the PastCasesbyRulenameLookup, if the analyst selects False Positive but the AI verdict was True Positive - Suspicious in previous cases, our agents can reference the analyst closure notes and case comments to understand why the previous verdict was incorrect. This context is fed to the agent before it makes a decision on the new case.

The AI Incorrect? custom field toggle is optional. An analyst can select it and fill out why they feel the AI verdict was wrong in the 'Detail on AI Incorrectness' custom field. If the AI verdict itself was correct but the “AI Generated Summary” case comment went in the wrong direction, the analyst can also explain here where the comment was incorrect. Even without this field, we can determine AI verdict correctness because the two AI agent–generated fields (AI Confidence and AI Verdict) are always on the alert via the workflows shown here, and the analyst is required to pick a closure reason. Our automation team pulls these custom fields from Kibana cases into a dashboard so we can track incorrect verdicts over time by comparing the analyst closure reason and AI verdict to tune our agent prompts or provide additional context where needed.

Here's the feedback loop in action, where our Pattern Finder Agent catches its own overclassification pattern in previous cases and surfaces tuning requests already filed against this rule:

How the agentic SOC pipeline runs in Elastic Workflows

We have one overarching Elastic workflow, Agent Brainstorm, that orchestrates 12 child workflows (and growing). The name reflects what's happening inside: one parent workflow (Agent Brainstorm), many context lookups, and three agents working together to reach a combined AI-generated case summary.

| Agent | Tools assigned | Inputs | Output | External queries |

|---|---|---|---|---|

| Pattern Finder | None | User details, investigation guide, past cases, current event | JSON | No |

| L1 Investigation | Yes | Pattern JSON, investigation guide, current alert | Structured report | Yes |

| Summarizer | None | Investigator report, pattern JSON, investigation guide, ES|QL queries | Markdown for Slack and Cases | No |

Pattern agent: Finding patterns across closed cases

Our Pattern Finder Agent takes in many inputs (user details, investigation guide, past case records, details about the current event) and makes pattern comparisons across historical cases. This agent has no tools assigned to it in Agent Builder and doesn’t make any external queries. (It only uses the data given to it.) That makes this agent much faster and ensures that the behavior cannot be overridden by injected content. Here’s a snippet of the prompt where we talk about our feedback loop:

AI Notes on Patterns

Look for patterns across these fields in past_case_records:

• workflow_reason (benign_positive, true_positive, etc.) This is the ANALYST PICKED result and should be seen as the truth.
• custom_field_ai_verdict. This is picked by AI - Is AI usually correct on this type of case?
• custom_field_ai_confidence. This is picked by AI - Is AI usually correct on this type of case?
• custom_field_ai_incorrect_analyst_bool - This is a boolean the analyst can choose to say that AI was incorrect here.
• custom_field_summary - This is written by an analyst on why they closed the case. This should be seen as the truth.
• comments_text (analyst comments and AI summaries)
• user_id, user_name, user_email (repeat offenders?)
• source_ip, host_name (recurring infrastructure?)
• If there is another case with the exact same type of activity, ALWAYS add the case # and a link to the case so we can reference it.
• Also consider inputs.user_job_family and inputs.user_job_title when assessing whether the alert pattern makes sense for this user type.

At the end of the prompt, we have it return its findings in JSON format:

Required Output Format

Respond with ONLY a valid JSON object, no markdown, no explanation:

{

"percentage_false_positive_or_benign_last_50_cases": "%",

"number_of_total_alerts": ,

"number_of_alerts_worked_in_cases": ,

"uniqueness": "RARE or COMMON",

"high_risk_employee": "NEW HIRE or LEAVING SOON or NO",

"job_title": "",

"cost_center": "",

"ai_notes_on_patterns": "",

"ai_pattern_TLDR":"",

"recommendation_for_severity": "medium or high or critical"

"explanation_on_severity_change_or_no": ""

}

L1 Investigation Agent: Gathering evidence and reaching a verdict

The Investigation Agent takes in the JSON from the Pattern Finder Agent, the investigation guide, and information about the current case and alert. Most importantly, it first looks at the investigation guide we grabbed above, which our detection engineers fill out with per-detection rule instructions that are the most correct for the agent to follow. If there’s no investigation guide, the agent falls back to its generic alert triage instructions:

Triage Protocol

> 📋 Investigation Priority: If an Investigation Guide is provided in the input, follow its triage steps as your primary instructions. The steps below are fallback guidance only — use them when no Investigation Guide is available or to fill gaps the guide does not cover.

Using the investigation guide and its own judgment, this agent makes external queries to gather additional evidence and reach a verdict on the case. At the end of the prompt, we have the investigator agent return a report:

INVESTIGATOR_REPORT_START

Verdict: | 🎯 Confidence: <0–100%> | 📋 Reason: |

INPUT_TYPE: Alert or Case

RULE_NAME_OR_CASE_TITLE: <>

SEVERITY: <>

STATUS: <>

TIMESTAMP: <>

ALERT_QUERY:

WHY_FIRED: <1 sentence — specific technical condition that matched>

WHEN_MALICIOUS: <1 sentence — malicious pattern and whether this alert DOES or DOES NOT match it>

ACTOR: | ASSET_INFO: <>

HOST: | ASSET_INFO: <>

KEY_ACTION: via

COMMAND_CONTEXT:

OBSERVABLES:

EXECUTIVE_SUMMARY: <2–3 sentence summary>

INVESTIGATION_FINDINGS:

CORRELATED_ALERT_CLUSTER:

PATTERN_FINDER_ALIGNMENT:

ESQL_QUERIES_RUN:

RECOMMENDED_ACTIONS:

• CONFIRM_ACTIVITY:
• REMEDIATION:
• RULE_TUNING:

INVESTIGATOR_REPORT_END

Summarizer Agent: Writing the case comment that analysts read

The Summarizer Agent is responsible for formatting the final AI summary that analysts see in the case comment. Like the Pattern Finder Agent, it has no tools assigned to it in Agent Builder and works only from data passed to it. This keeps it fast. Each time a new alert comes in for the same case, we rerun the Agent Brainstorm workflow and replace the case comment only if it provides a new verdict or new important evidence; otherwise, we skip it. We allow the summarizer to keep replacing itself up until the case is assigned to an analyst, at which point we stop.

Here’s a snippet of the summarizer prompt:

Role & Objective

You are an expert InfoSec report formatter at Elastic. You receive a structured investigation report from the Investigator agent and a pattern analysis from the PatternFinder agent. Your sole job is to combine these into a high-fidelity Markdown summary formatted for Slack. You run NO queries and make NO investigative decisions — all findings and verdicts come from the Investigator and PatternFinder.

You will also receive:

• `INVESTIGATOR_REPORT` — the full structured output from the Investigator agent
• `PATTERN_FINDER_OUTPUT` — the JSON analysis from the PatternFinder agent
• `INVESTIGATION_GUIDE` - the investigation guide attached to the alert that gives suggestions to analyst and AI on how to handle this case
• `ESQL_QUERIES` — comma-separated ES|QL query strings run during investigation, to be formatted as reference links

Then we ask it to build an output template to post to the case comment:

Final Output Template

🤖 AI Generated Summary 🤖

` Verdict: | 🎯 Confidence: <0–100%> | 📋 Reason: |`

⚡ TLDR

> {1-2 sentences max. Who did what, on what system, and what the verdict is. Written for an analyst who has 5 seconds. No jargon, no hedging.}

Investigation Guide Suggestions (only include if it is present)

• Quick summary of what the analyst should do next based on the investigation guide or how to confirm this activity. Use bullet points if possible

....shortened for purposes of this blog...

Now that we’ve reviewed each agent's role, here's how they're configured in workflow YAML. Each agent is executed one after the other in a synchronous way, waiting for the one above it to finish:

We use two workflows to make POST requests at the end of the Agent Brainstorm to attach the generated final AI summary to both the Kibana case and Slack. Below are some real examples from our environment:

We send alert details to our team Slack channel so that we can monitor the alerts from our phone. Our Slack bot in this channel allows us to perform activities, like Acknowledge Case or Merge Case, all from Slack, allowing us to quickly take action, even directly from a mobile device. After implementing the work in this blog, we added the Close Case: AI Correct button, which you can click to trigger a workflow that assigns you to the case, fills out the closure notes, tags the case, and closes it.

Our AI triager that started off as AI slop is now a core part of how our SOC operates. We’ve kept our analysts in the loop and enabled them to close cases in seconds with our one-button close after AI triages the case. If you're building something similar, we'd love to hear about it in the Elastic Security community forum.

Frequently asked questions

Why three agents instead of one agent with a lot of skills?

A member of our team did a cost analysis of using one agent with a lot of skills versus multiple agents and the token usage came back significantly lower with the latter.

Why don’t we just have the agent run the query rather than providing it with data we’ve already received from ES|QL?

AI agents process pre-fetched data significantly faster than querying indexes directly, and it's cheaper.

What to build after L1 triage automation

After the L1 agent runs, a great next step is to dispatch specialized agents based on the alert type or data source; for example, a Windows forensics agent or an Azure agent, or possibly a Linux forensics agent. Read our team's writeup on L2 and L3 agent escalation.