AI application security explained

AI application security addresses vulnerabilities that arise when large language models, machine learning components, or AI agents are integrated into applications. Unlike traditional application security, which focuses on code logic and input validation, AI vulnerabilities often exploit the statistical nature of ML models, extending the security boundary to training datasets, model weights, prompt engineering, and agent behavior patterns. The four attack surface clusters identified are input and context manipulation, supply chain and training data integrity, agent and plugin behavior, and model information disclosure. Traditional tools like SAST, DAST, and WAF are insufficient because they operate at the code level rather than the model behavior level.
The core of these vulnerabilities stems from the difference between deterministic application logic and probabilistic model inference. For identity management, AI agents introduce non-deterministic decision-making that challenges traditional role-based access control. The framework requires shifting security focus to verifying the integrity of data artifacts—such as prompt inputs and model updates—through cryptographic signing, establishing verifiable provenance across the entire AI pipeline.

* Web application firewalls block SQL injection attempts.
* Static analysis tools identify hardcoded secrets in code.
* Dynamic testing validates input sanitization but does not find agent over-permissioning.
* Four attack surface clusters exist: input and context manipulation, supply chain and training data integrity, agent and plugin behavior, and model information disclosure.
* Prompt injection manipulates the LLM via context or instructions.
* Training data poisoning involves injecting malicious examples into datasets to create predictable model behavior.
* Agent behavior can exceed intended boundaries, leading to unauthorized system access.
* Model extraction uses repeated queries to reconstruct proprietary model behavior.
* Input and context manipulation requires input sanitization paired with cryptographic signing of prompt inputs.
* Supply chain artifacts (training data, model weights) should carry verifiable signatures confirming provenance.
* AI agents require least-privilege principles for identity and behavior monitoring.

The narrative outlines a necessary paradigm shift: security must move from verifying deterministic code execution to validating probabilistic model behavior and the integrity of the data/model ecosystem. The fundamental tension lies between traditional, code-centric security tooling and AI's behavior-centric attack surface. Traditional tools fail because they operate on syntactic structure (code) while AI attacks exploit semantic meaning (context and probability).
The implication for human agency is that control over systems requires understanding not just *what* an application does, but *how* the underlying statistical mechanisms make decisions. When agents exhibit non-deterministic behavior, accountability blurs if traditional access controls are insufficient to contain emergent capabilities. The call for cryptographic signing of artifacts introduces a governance layer that addresses the trust gap between deployed assets and their provenance, mirroring established software supply chain security principles.
The pattern suggests an ongoing struggle where technological evolution (AI) outpaces established control frameworks (SAST, DAST). The risk is not merely technical error but the erosion of the assumptions underlying identity and access models—treating non-human agents as deterministic entities. If we cannot instrument or monitor the internal decision process of a probabilistic system, the security assurance shifts from proving correctness to monitoring emergent behavior.
What measures must be taken to institutionalize this shift? How can organizations develop new metrics for agent alignment that complement traditional vulnerability scanning? And how do we ensure that the pursuit of novel AI capabilities does not create an unmanageable gap between authorized permissions and realized actions?