Aug 6, 2026
Why “AI Pentesting” is the Wrong Term (And Why We Need AI Red Teaming)
Recently, I read a great post by Melvin Tan Zhi Xian sharing his thoughts halfway through the OffSec OSAI+ course. He touched on something crucial that isn’t being talked about enough: how the methodology for attacking AI doesn’t map cleanly to a standard pentest framework. Melvin’s feedback really resonated with me. It sparked a deeper
Recently, I read a great post by Melvin Tan Zhi Xian sharing his thoughts halfway through the OffSec OSAI+ course. He touched on something crucial that isn’t being talked about enough: how the methodology for attacking AI doesn’t map cleanly to a standard pentest framework.
Melvin’s feedback really resonated with me. It sparked a deeper dive into how we, as an industry, are approaching this space, and I wanted to share my personal take (together with my AI) on why we need to shift the conversation.
Right now, the cybersecurity industry is throwing a massive amount of budget at “AI Pentesting.” But there is a growing disconnect: the clients asking for AI security testing don’t always know what they’re asking for, and the vendors selling it are often just running a checklist of prompt injections against an isolated model.
If you have been in offensive security for a while, you know that attackers don’t operate in a vacuum. They don’t just attack an algorithm but they attack the infrastructure hosting the model.
The Reality of the AI Attack Chain: When an Advanced Persistent Threat (APT) targets an enterprise AI deployment, they aren’t just trying to make a chatbot say a bad word. The real attack chain looks like this:
- Breaching the external perimeter.
- Pivoting laterally through the network.
- Compromising the AI agent or service.
- Exfiltrating sensitive training data or manipulating the RAG (Retrieval-Augmented Generation) pipeline.
As Melvin rightly pointed out, this methodology is much closer to Red Teaming—requiring a broader scope, highly creative adversary simulation, and a focus on how an AI deployment serves as a gateway into the broader corporate network.
The “Infrastructure-First” Approach This is exactly the reality of enterprise AI defense, and it’s the philosophy that underpins OffSec’s OSAI+ course.
The biggest misconception about AI security is that you have to be a machine learning engineer to understand it. You don’t. A lot of AI attack vectors still come back to traditional cybersecurity fundamentals: understanding how networks are structured, how services communicate, and how to move through a system.
If you are a seasoned pentester or red teamer, you aren’t starting from zero. You are taking the skills you already have (Active Directory, lateral movement, network protocols) and layering AI-specific exploitation right on top of them.
Going Beyond the Checklist: As the market matures, the conversation needs to shift. A compliance checklist might satisfy an audit, but it won’t stop a breach. To truly secure AI in the enterprise, we have to stop treating AI models as isolated math problems and start treating them as what they really are: just another highly privileged node on the network.
If we want to secure the future of enterprise AI, we need true adversary simulation. We need AI Red Teaming.
A huge thanks to Melvin for sparking this thought process! I’d also love to invite my OffSec colleagues to chime in, please feel free to jump into the comments and correct me if my perspective is off base here!
For everyone else, what are your thoughts? Are you seeing this same shift from isolated AI testing toward full-scope AI Red Teaming in your engagements? Ping me to let me know.