What changes when the target is AI
AI applications expose attack surfaces through three distinct vectors that standard application security testing does not address: prompt manipulation (direct and indirect injection), model behavior exploitation (excessive agency, information disclosure), and supply chain integrity failures specific to machine learning pipelines (Source: OWASP Top 10 for Large Language Model Applications, owasp.org). Traditional penetration testing assumes deterministic application behavior. Submit the same input twice, get the same output. AI models produce probabilistic outputs. The same prompt can generate different responses across test runs, making reproducible exploit development impossible using standard methodology.Standard pentests target the application's code and infrastructure. AI red teaming targets the model's training, inference behavior, and integration patterns. A SQL injection test probes input validation in application code. A prompt injection test probes the boundary between user input and system instructions within the model's context window.The testing environment differs fundamentally. Standard pentests often use isolated test environments that mirror production data and configuration. AI red teaming requires access to the actual model weights, training pipeline, or live inference endpoints because model behavior cannot be replicated through configuration alone.Control Question: Does your current security testing methodology include adversarial prompt construction and model output validation? If not, standard application security testing will miss the primary attack vectors against AI systems.The three test categories
AI red teaming exercises target three attack categories defined by the OWASP LLM Top 10 and MITRE ATLAS frameworks: prompt-based attacks, model-based attacks, and supply chain attacks (Source: OWASP Top 10 for Large Language Model Applications, owasp.org; MITRE ATLAS, atlas.mitre.org).Prompt-based attacks target the model's instruction-following behavior. Direct prompt injection attempts to override system prompts through adversarial user input. Indirect prompt injection embeds malicious instructions in documents or data sources that the AI system processes (Source: OWASP Top 10 for Large Language Model Applications, owasp.org). These attacks exploit the model's inability to distinguish between system instructions and user data within the same context window.Testing prompt injection requires constructing adversarial prompts that bypass input filtering and safety training. Standard input validation testing cannot replicate this attack class because it targets application parsing logic, not model instruction interpretation.Model-based attacks exploit the AI system's inference process and memory. Model extraction attempts reconstruct training data or model parameters through carefully crafted queries. Information disclosure attacks probe the model to reveal sensitive data from training sets or system prompts (Source: MITRE ATLAS, atlas.mitre.org). Excessive agency attacks test whether the AI system performs actions beyond its intended scope when prompted to do so.These attacks require understanding transformer architecture, attention mechanisms, and training data memorization patterns. Standard application security testing methodology does not include these knowledge domains.Supply chain attacks target the AI development and deployment pipeline. Training data poisoning injects adversarial examples during model training to create backdoors or bias. Model supply chain attacks target pre-trained models, training frameworks, or inference infrastructure (Source: MITRE ATLAS, atlas.mitre.org).Testing supply chain integrity requires access to training pipelines, model provenance data, and deployment artifacts. Standard penetration testing focuses on deployed application security, not development pipeline integrity.Why standard pentest methodology doesn't transfer
Standard penetration testing methodology follows a linear progression: reconnaissance, scanning, enumeration, exploitation, post-exploitation. This methodology assumes that successful exploitation produces deterministic system access or data extraction.AI red teaming cannot follow this progression because AI systems do not have traditional exploitation points. There is no "shell access" equivalent for an AI model. Success means manipulating model outputs or extracting training data, not gaining system privileges.Traditional pentests use automated scanners to identify known vulnerability patterns in code and configurations. AI vulnerabilities emerge from model behavior patterns that cannot be detected through static analysis or configuration scanning. A prompt injection vulnerability exists in the model's training and inference behavior, not in deployable code.Standard methodology assumes that fixing identified vulnerabilities eliminates the attack vector. AI vulnerabilities often cannot be "patched" without retraining the model or fundamentally changing system architecture. A successful prompt injection attack reveals a limitation in the model's instruction-following training, not a configuration error.NIST AI RMF Map 5.2 requires evaluation of AI system performance under adversarial conditions relevant to the deployment context — testing requirements that standard application security methodology cannot satisfy (Source: NIST AI RMF, airc.nist.gov).Repeatability presents another fundamental difference. Standard pentests produce repeatable results: the same exploit technique works consistently against the same vulnerability. AI model outputs vary between identical requests due to sampling and randomness in inference. An adversarial prompt that succeeds in one test run may fail in subsequent runs without any system changes.Organizations that apply standard pentest methodology to AI application infrastructure cover API security, authentication, and data handling but miss the model-specific attack vectors that represent the primary threat to AI systems.| Dimension | Standard Penetration Testing | AI Red Teaming |
|---|---|---|
| Primary Attack Model | Exploit code vulnerabilities to gain unauthorized system access | Manipulate model behavior to produce unauthorized outputs or extract information |
| Scope Definition | Application code, infrastructure, and data flows within defined network boundaries | Model training pipeline, inference behavior, and integration patterns across the AI system lifecycle |
| What The Tester Controls | Network access, input vectors, and system interaction points | Prompt construction, model queries, and training data influence (when applicable) |
| Test Repeatability | Deterministic — same exploit produces same result across test runs | Probabilistic — model outputs vary between identical inputs due to sampling randomness |
| Required Tester Expertise | Network protocols, web application security, infrastructure hardening | Machine learning architecture, adversarial ML techniques, transformer model behavior |
Additional risk dimensions: model drift and overfitting
Two model-level conditions warrant specific attention during AI red team engagements: model drift and model overfitting. Neither maps cleanly to traditional application security concepts, but both create exploitable security conditions.Model drift occurs when a deployed model's behavior diverges from its validated baseline over time — typically because the real-world data the model encounters during inference shifts away from the distribution it was trained on. From a security perspective, drift can erode safety training and content filtering that performed correctly at deployment. Red team testing should establish a behavioral baseline at initial deployment and include periodic adversarial re-testing to detect whether safety boundaries have degraded as the model's operational context evolves. Organizations that perform one-time AI red team assessments and treat the results as durable may be operating on an outdated security posture.Model overfitting creates a different risk: a model that has memorized specific training examples rather than learning generalizable patterns is more susceptible to training data extraction attacks. Overfit models can reproduce verbatim training content — including sensitive data — in response to targeted queries. Red team testing should include model inversion and membership inference techniques designed to determine whether the model recalls specific training records, which is a more acute risk in overfitted models than in well-generalized ones.Both conditions reinforce the case for continuous or periodic AI red teaming rather than point-in-time assessments.The role of model context protocol in AI Red Teaming
Model Context Protocol (MCP) is an emerging open standard that defines how AI models connect to and interact with external tools, data sources, and services during inference. As organizations adopt MCP-compatible architectures to give AI systems access to file systems, databases, APIs, and business applications, MCP becomes a significant and underexamined attack surface that AI red teams must address.From an attacker's perspective, MCP servers represent an integration layer where compromised or malicious tool definitions can redirect model behavior, exfiltrate data, or cause the AI system to take unintended actions against connected resources. Indirect prompt injection attacks are particularly relevant here: malicious instructions embedded in data returned by an MCP-connected tool can manipulate the model's subsequent actions within the same inference session.AI red team engagements in MCP-enabled environments should evaluate:- Whether adversarial prompts can cause the model to invoke MCP tools outside their intended scope
- Whether malicious content retrieved through MCP-connected data sources can inject instructions that alter model behavior
- Whether the model enforces appropriate authorization boundaries when accessing resources through MCP server integrations
- Whether MCP server configurations themselves can be targeted to serve compromised tool definitions to the model
Scoping an AI Red Team engagement
AI red team scope definition requires identifying three system boundaries: the model inference boundary, the training pipeline boundary, and the integration boundary.Model inference boundary defines what the model can access during inference: context windows, external data sources, and connected systems. Red team testing probes whether adversarial prompts can cause the model to access or manipulate data beyond its intended scope. This boundary testing requires understanding the model's architecture and integration patterns.Training pipeline boundary defines the data sources, preprocessing steps, and training infrastructure used to develop the model. Red team testing evaluates whether adversaries can influence training data or inject backdoors during model development. This testing requires access to training logs, data lineage documentation, and model versioning systems.Integration boundary defines how the AI system connects to other applications, APIs, and data sources — including MCP servers where applicable. Red team testing probes whether model outputs can trigger unintended actions in connected systems or whether external data sources can inject malicious prompts. This boundary overlaps with traditional application security but requires understanding prompt flow and context injection patterns.Engagement Prerequisites:- Model architecture documentation and training methodology details
- Access to inference endpoints or model deployment environment
- Training data source identification and lineage documentation
- Integration mapping showing connected systems and data flows, including any MCP server configurations
- System prompt and safety training configuration details
- Behavioral baseline documentation to support drift detection across test cyclesTest Objectives Checklist:
□ Prompt injection resistance across direct and indirect attack vectors
□ Information disclosure through model interrogation techniques
□ Excessive agency when prompted to perform unauthorized actions
□ Training data extraction through model inversion attacks, with particular attention to overfitting indicators
□ Supply chain integrity validation for pre-trained models and training data
□ Output validation bypass through adversarial prompt construction
□ Cross-system impact assessment when AI outputs trigger downstream actions
□ MCP integration boundary testing for tool misuse and indirect injection through connected data sources
□ Behavioral drift assessment against deployment baseline where prior test results are availableOrganizations should scope AI red team engagements based on the highest-risk model interactions in their deployment. Testing every possible prompt variation is impractical; focus on scenarios where model manipulation could cause business impact or data exposure.Verification Step: Before engaging an AI red team, validate that your organization has documented the three system boundaries and can provide the prerequisite access and documentation. Without this foundation, the engagement cannot effectively test the AI-specific attack vectors that justify the methodology.