The risks of using LLMs in development include insecure generated code, hallucinated dependencies, and leaked secrets. Endor Labs catches each one before AI-written code reaches production.
This is because LLMs don't follow the rules that traditional software security was built around. They blur the line between data and instructions, produce non-deterministic outputs, and turn user inputs into something that looks a lot like executable code.
That creates a new category of risk that existing security tools weren't designed to catch. This guide covers the major LLM development risks - from prompt injection to supply chain vulnerabilities - and the specific mitigation strategies that actually work in practice.
What are LLM development risks
LLM development risks are the security vulnerabilities, ethical concerns, and operational challenges that show up when you build, integrate, or deploy large language models in software applications. The OWASP Top 10 for LLM Applications categorizes the major risk areas: prompt injection, training data poisoning, sensitive data exposure, excessive agency, and supply chain vulnerabilities, among others.
What makes LLM risks different from traditional software vulnerabilities is that LLMs blur the line between data and instructions. User inputs can effectively become executable commands. The model's probabilistic nature also makes behavior harder to predict and test than deterministic code paths.
Why LLMs create unique security challenges
Traditional software follows deterministic logic—same input, same output. LLMs break that assumption entirely. The same prompt can produce different responses across invocations, which complicates testing and validation.
There's also the problem of prompt-as-code. In a typical application, user input goes into a data field, and your code decides what to do with it. With LLMs, user input becomes part of the instruction set. That collapses the separation between data and logic that security models have relied on for decades.
LLM security risks at the model layer vs application layer
Understanding where a risk originates helps you assign ownership and apply the right controls. Some risks exist regardless of how you deploy the model, while others are entirely within your development team's control.
Model layer risks
Model layer risks are inherent to the LLM itself. Training data poisoning, for example, happens before you ever touch the model—malicious or biased data gets baked into the weights during training. Model theft and extraction attacks target the model's intellectual property through systematic querying. You can mitigate model layer risks through vendor selection and access controls, but you can't eliminate them through application-level changes alone.
Application layer risks
Application layer risks emerge from how you integrate the LLM. Prompt injection via user inputs, insecure API connections, improper output handling, and plugin vulnerabilities all fall into this category. The good news: application layer risks are within your direct control. The bad news: they're also your responsibility to find and fix.
Top LLM security risks in software development
The OWASP Top 10 for LLM Applications provides a solid taxonomy. Here's how the major risk categories play out in practice.
Prompt injection and input manipulation
Prompt injection is the SQL injection of the LLM world. In direct prompt injection—a JAMA Network Open study found attacks achieved a 94.4% success rate in controlled medical LLM dialogues. In direct prompt injection, a user crafts input designed to override your system instructions—think "ignore previous instructions and do X instead." Indirect prompt injection is sneakier: malicious content embedded in external data sources (websites, documents, emails) gets processed by the LLM and influences its behavior without the user explicitly typing anything malicious.
Insecure output handling and code execution
LLM outputs often get passed to downstream systems—databases, shells, browsers, APIs. If you're not sanitizing that output, the LLM becomes a vector for injection attacks. An LLM that generates SQL queries could produce queries containing injection payloads—a risk demonstrated by a critical SQL injection flaw in LlamaIndex. An LLM that generates HTML could produce XSS payloads. The model doesn't intend to be malicious; it's just completing patterns.
Training data poisoning
When you fine-tune a model or use a third-party fine-tuned model, you're trusting that the training data was clean. Corrupted or malicious training data can cause models to behave unexpectedly, produce harmful outputs, or contain backdoors that activate under specific conditions.
Sensitive data exposure and LLM data security
Data leakage happens in two directions. First, models can memorize and regurgitate training data, including PII or proprietary information. Second, developers and users often send sensitive data to external LLM APIs without realizing the implications. Both create compliance headaches under GDPR, HIPAA, and similar frameworks.
Supply chain vulnerabilities in LLM dependencies
LLM applications typically depend on frameworks like LangChain, model weights from Hugging Face, and various plugins. Each of these is a potential attack vector.
Traditional SCA tools may not catch LLM-specific supply chain risks—they're looking for known CVEs in packages, not malicious model weights or compromised plugins.
Insecure plugin and extension design
LLM plugins that execute actions—API calls, database queries, file operations—can be exploited if they lack proper authorization, input validation, or scope limitations. A plugin that can "search the web" might also be able to exfiltrate data if its permissions aren't carefully scoped.
Excessive agency in autonomous LLM actions
When LLMs are granted permissions to take actions without human oversight—automated commits, deployments, or system changes—the blast radius of a mistake or manipulation expands dramatically. The more agency you grant, the more careful you want to be about guardrails.
Overreliance on LLM outputs
This is a human risk, not a technical one. Developers may accept insecure code suggestions without verification. Business users may trust LLM-generated analysis without fact-checking. The model's confident tone doesn't correlate with accuracy.
LLM security issues in AI-assisted coding
AI coding assistants like Cursor, Copilot, and Claude Code introduce specific security concerns into the software development lifecycle.
Vulnerable code patterns in generated code
AI coding assistants can suggest code with common vulnerability patterns—buffer overflows, injection flaws, improper error handling—buffer overflows, injection flaws, improper error handling. Veracode's 2025 research found AI-generated code introduced vulnerabilities in 45% of tasks across 100+ LLMs. This code typically passes syntax checks and may even look reasonable at first glance, but it can fail a security review.
Hardcoded secrets and credential exposure
LLMs may generate code containing placeholder credentials, example API keys, or patterns that encourage hardcoding secrets instead of using proper secret management. A developer in a hurry might not notice that api_key = "sk-example123"
was meant to be replaced.
Malicious package recommendations
AI assistants can recommend packages that are typosquatted, abandoned, or contain known malwareAI assistants can recommend packages that are typosquatted, abandoned, or contain known malware—with 34,319 malicious packages discovered across major registries in Q3 2025 alone, the pool of dangerous recommendations keeps growing. This risk is higher for less common languages or newer ecosystems where the assistant has less training data to distinguish legitimate packages from malicious ones.
Insecure dependency suggestions
Beyond malicious packages, AI assistants may suggest outdated package versions with known CVEs or dependencies with transitive vulnerabilities the developer doesn't see. The assistant optimizes for "code that works," not "code that's secure."
Mitigation strategies for LLM security risks
Here's what actually works for reducing LLM security risks in practice.
1. Validate inputs and sanitize outputs
Treat LLM boundaries like any untrusted interface. Validate inputs before they reach the LLM—check for injection patterns, enforce length limits, and strip potentially dangerous content. Then sanitize outputs before passing them to downstream systems. If the LLM generates SQL, parameterize it. If it generates HTML, escape it.
2. Apply least privilege to LLM integrations
Limit what LLMs can access and do. Plugins get scoped permissions—a "search" plugin doesn't get write access. API keys used by LLMs get minimal necessary access. If the LLM only reads from a database, don't give it write permissions.
3. Scan AI-generated code for vulnerabilities
Run static analysis on all code, regardless of its source. AI-generated code requires the same scrutiny as human-written code—arguably more, since developers may review it less carefully. Tools that use semantic analysis rather than just pattern matching catch more of the subtle issues that AI assistants introduce.
4. Monitor LLM interactions for data leakage
Log and audit data flowing to and from LLMs. Implement filters to block sensitive data patterns—PII, credentials, proprietary code—from being sent to external LLM APIs.
5. Enforce security policies across AI coding agents
Establish consistent guardrails that apply regardless of which AI assistant developers use. Policy-as-code ensures uniform enforcement across Cursor, Copilot, Claude Code, and whatever comes next.
6. Conduct regular security reviews of LLM integrations
Include LLM integrations in security reviews and threat modeling. Review plugin permissions, API scopes, and data flows quarterly or whenever integrations change.
LLM security tools for development teams
Different tool categories address different aspects of LLM security risk.
Static analysis for AI-generated code
SAST tools that use semantic analysis—not just pattern matching—catch logic flaws that AI assistants commonly introduce. Look for tools that can trace data flow across multiple files and functions, since AI-generated code often spans module boundaries.
Software composition analysis with reachability
SCA tools identify vulnerable dependencies, but the best ones also determine which vulnerabilities are actually reachable in your code. Full stack reachability across code, dependencies, and containers reduces noise significantly so you can focus on vulnerabilities that matter.
Secrets detection in LLM workflows
Tools that scan for hardcoded credentials in code, including AI-generated code, integrate into the IDE, pre-commit hooks, and CI/CD pipelines. The goal is catching secrets before they reach the repository, not after.
Malware detection for LLM-recommended packages
Tools that analyze package code for malicious behavior—not just known CVEs—are important because AI assistants may suggest packages that haven't been flagged yet.
How to evaluate LLM application risk before deployment
Before deploying an LLM integration, ask five questions:
Deployment Type
Data Exposure Risk
Action Risk
Recommended Controls
Internal chatbot with read-only access
Low
Low
Input validation, output logging
Customer-facing assistant with API access
High
Medium
Input/output filtering, rate limiting, human approval for actions
Autonomous agent with system permissions
High
High
Strict sandboxing, human-in-the-loop, comprehensive audit logging
Securing LLM development with full stack visibility
To operationalize LLM security, teams want visibility across code (including AI-generated), dependencies (including LLM frameworks), and containers. Full stack reachability analysis helps identify which LLM-related vulnerabilities are actually exploitable in your specific application context.
Endor Labs provides this visibility through AURI, the security intelligence layer for agentic software development. AURI builds a call graph across your entire application and verifies that security findings are reachable and exploitable, delivering up to 95% noise reduction. For teams adopting AI coding assistants, that means fewer false positives, actionable fixes, and security that keeps pace with development velocity.
FAQs about LLM security risks
How do MCP server integrations with AI coding assistants introduce security risks?
MCP (Model Context Protocol) servers can expose local files, environment variables, and system access to AI assistants. Without properly scoped permissions, this creates potential data leakage and unauthorized access. Configure MCP servers with minimal necessary permissions and audit what data they expose.
What is the difference between LLM security and traditional application security?
Traditional application security assumes deterministic behavior and a clear separation between data and code. LLM security accounts for non-deterministic outputs, natural language as an attack vector, and the model itself as an additional attack surface.
Can LLMs detect security vulnerabilities in their own generated code?
LLMs can identify some obvious vulnerability patterns when prompted, but they lack the deterministic analysis capabilities of purpose-built security tools. They may miss context-dependent flaws or produce false confidence in insecure code. Use dedicated security tools, not the LLM itself, for vulnerability detection.
How do organizations create a security policy for AI-generated code?
Define which AI coding tools are approved. Require all AI-generated code to pass the same security scans as human-written code. Establish review requirements for AI-assisted changes. Log AI tool usage for audit purposes.
Which compliance frameworks specifically address LLM security requirements?
NIST AI RMF, the EU AI Act, and ISO/IEC 42001 provide guidance on AI system security. Existing frameworks like SOC 2 and FedRAMP require security controls that apply to LLM integrations even without LLM-specific language.
What's next?
When you're ready to take the next step in securing your software supply chain, here are 3 ways Endor Labs can help: