AI Code Security Risks
Hallucinated dependencies, injection vulnerabilities, and supply chain threats - how AI-generated code is expanding your attack surface
The Expanding Attack Surface from AI Code Generation
AI coding assistants are generating millions of lines of code daily - and a significant portion contains security vulnerabilities that traditional review processes miss. Unlike human-written bugs that follow predictable patterns, AI-generated security flaws are novel, varied, and often hidden behind code that looks perfectly correct.
From hallucinated packages that attackers weaponize to injection vulnerabilities baked into suggestions, AI tools are creating an entirely new threat landscape. Understanding these risks is the first step toward building secure software in the age of AI-assisted development.
The Threat Landscape: 6 Critical AI Security Risks
These six categories represent the most dangerous security risks introduced by AI coding assistants. Each one has been observed in production systems and each one can lead to a breach if left unaddressed.
Hallucinated Dependencies
The risk: AI models suggest importing packages that do not exist. These "hallucinated" package names become attack vectors when malicious actors register them on npm, PyPI, or other registries with backdoored code.
A developer trusts the AI suggestion, runs npm install, and unknowingly installs malware. Researchers found that up to 20% of AI-suggested packages across popular models reference packages that have never existed.
Severity: Critical - Direct remote code execution path
Injection Vulnerabilities
The risk: AI frequently generates code that concatenates user input directly into SQL queries, shell commands, or HTML output without proper sanitization, parameterized queries, or escaping.
Studies show AI models produce SQL injection vulnerabilities in over 30% of database-related code suggestions. The code looks clean and functional but lacks the defensive patterns a security-conscious developer would include.
Severity: Critical - OWASP Top 10 #3 (Injection)
Hardcoded Secrets
The risk: AI models include example API keys, default passwords, placeholder tokens, and connection strings directly in generated code. Developers copy these into production without replacing them with proper secret management.
Worse, some AI-generated "example" keys are actual leaked credentials from training data. These get committed to repositories and exposed through version history even after removal.
Severity: High - Credential exposure and unauthorized access
Insecure Deserialization
The risk: AI models commonly suggest using eval(), pickle.loads(), yaml.load() without safe loaders, and other unsafe parsing patterns that allow attackers to execute arbitrary code through crafted input.
These patterns are prevalent in AI training data because older tutorials and Stack Overflow answers used them before the security community recognized the risks.
Severity: High - Remote code execution via crafted payloads
Broken Authentication Patterns
The risk: AI generates authentication code with weak hashing algorithms (MD5, SHA-1), missing rate limiting, improper session management, client-side-only validation, and JWT implementations without proper signature verification.
AI models learn from the vast majority of auth code online - most of which is outdated or insecure. The generated code "works" for login/logout but fails under adversarial conditions.
Severity: Critical - Account takeover and privilege escalation
Supply Chain Poisoning
The risk: Attackers deliberately poison AI training data by publishing malicious code patterns, vulnerable tutorials, and backdoored libraries that AI models learn to recommend to developers.
This is the next evolution of supply chain attacks. Instead of compromising one package, attackers influence the AI models themselves - poisoning the well that millions of developers drink from daily.
Severity: Critical - Scalable attack across entire ecosystems
Real-World AI Security Incidents
These are not hypothetical scenarios. Each of these incidents demonstrates how AI-generated code has already caused real security failures in production systems.
The Hallucinated Package Registry Attack (2024-2025)
Security researchers discovered that ChatGPT and other LLMs consistently recommended non-existent npm and PyPI packages. Attackers began registering these hallucinated package names with malicious code. When developers followed AI suggestions and ran install commands, they unknowingly executed malware that exfiltrated environment variables, SSH keys, and AWS credentials.
Impact: Over 15,000 unique hallucinated package names identified across major AI models. Multiple confirmed installations of weaponized packages in corporate environments.
The Copilot Credential Leak (2024)
A major fintech company discovered that GitHub Copilot had suggested code containing what appeared to be example AWS access keys - but they were real credentials from another repository in the training data. The keys were committed, pushed, and went undetected for weeks. Automated scanners eventually flagged the exposure, but not before the keys were harvested by credential-scanning bots.
Impact: Unauthorized access to cloud infrastructure. Full incident response required including key rotation, forensic analysis, and regulatory notification.
SQL Injection via AI-Generated API (2025)
A startup used AI to generate their entire REST API layer. The generated code used string interpolation for database queries throughout the application. A penetration test revealed over 40 SQL injection points across 12 endpoints. The AI had generated code that "worked perfectly" in development but was completely vulnerable to basic injection attacks.
Impact: Complete database access. Three months of remediation work to rewrite the data access layer with parameterized queries.
The Training Data Poisoning Campaign (2025-2026)
Security researchers uncovered a coordinated campaign where attackers published hundreds of seemingly helpful open-source libraries and tutorials containing subtle backdoors. These were designed to be ingested by AI training pipelines. The poisoned patterns began appearing in AI code suggestions months later, recommending the backdoored libraries and insecure patterns to unsuspecting developers.
Impact: Unknown full scope. Demonstrated that AI models can be weaponized at scale through deliberate training data manipulation.
Deep Dive: The Hallucinated Package Attack
This attack vector is unique to AI-assisted development. It exploits the gap between what AI models "think" exists and what actually exists in package registries.
Attack Flow: From Hallucination to Compromise
AI Hallucinates a Package Name
A developer asks an AI assistant: "How do I validate email addresses in Python?" The AI responds with import py-email-validator - a package that does not exist. The real package is email-validator.
Attacker Monitors AI Hallucinations
Attackers systematically query AI models, collect hallucinated package names, and check which ones are available for registration. They build databases of frequently hallucinated names across different programming tasks.
Malicious Package is Published
The attacker registers py-email-validator on PyPI with real email validation functionality (to avoid suspicion) plus a hidden setup.py post-install script that exfiltrates environment variables to an external server.
Developer Trusts the AI Suggestion
A developer follows the AI's advice and runs pip install py-email-validator. The package installs successfully, the email validation works correctly, and the developer moves on - unaware that their AWS keys, database credentials, and SSH keys were just exfiltrated.
Compromise Achieved
The attacker now has valid credentials. They access cloud infrastructure, databases, or internal systems. The malicious package remains installed and continues to operate, potentially spreading through CI/CD pipelines to production environments.
Highly Reproducible
AI models are deterministic enough that the same hallucinated package name appears repeatedly. Multiple developers ask similar questions and get the same wrong answer, creating a reliable attack funnel.
Hard to Detect
The malicious package actually works for its stated purpose. It passes functional tests, it resolves the developer's problem, and it looks legitimate. Traditional code review would not flag a working dependency.
Scales Effortlessly
One attacker can register hundreds of hallucinated package names for minimal cost. Each registration is a passive trap that catches developers without any further action from the attacker.
Security Checklist for AI-Generated Code
Follow these steps every time you accept code from an AI coding assistant. Treat this as your security gate before any AI-generated code enters your codebase.
AI Code Security Review Checklist
Verify All Dependencies Exist
Before running any install command, manually verify that every package the AI suggested actually exists on the official registry. Check the package's publish date, download count, maintainer history, and GitHub repository.
Check for Hardcoded Secrets
Scan all AI-generated code for API keys, tokens, passwords, connection strings, and any string that looks like a credential. Use tools like git-secrets, trufflehog, or detect-secrets in your pre-commit hooks.
Validate Input Handling
Examine every point where user input enters the system. Ensure parameterized queries for SQL, proper escaping for HTML output, allowlist validation for file paths, and safe deserialization for any data parsing.
Review Authentication and Authorization
Never trust AI-generated auth code without expert review. Check hashing algorithms (must be bcrypt, scrypt, or Argon2), verify JWT validation is complete, ensure rate limiting is present, and confirm session management follows current best practices.
Run SAST and Dependency Scanning
Before merging, run static analysis security testing (SAST) tools and dependency vulnerability scanners. AI-generated code should go through the same security pipeline as human-written code - no exceptions.
Check for Unsafe Deserialization
Search for eval(), exec(), pickle.loads(), yaml.load() without SafeLoader, JSON.parse() on untrusted input without validation, and any dynamic code execution patterns.
Verify Error Handling Does Not Leak Information
AI often generates verbose error messages that expose stack traces, database schemas, internal paths, and system details. Ensure production error handling returns generic messages while logging details server-side.
Test with Adversarial Inputs
Do not just test the happy path. Try SQL injection payloads, XSS vectors, path traversal sequences, oversized inputs, unicode edge cases, and null bytes. AI-generated code almost never handles adversarial inputs correctly.
Tools and Automation for AI Code Security
Manual review alone is not enough. These tools automate the detection of security issues in AI-generated code and should be integrated into your CI/CD pipeline.
Snyk
Scans dependencies for known vulnerabilities, detects license compliance issues, and monitors for newly discovered CVEs. Integrates directly into IDEs and CI/CD pipelines.
Best for: Dependency vulnerability scanning, container security, and infrastructure-as-code checks.
Semgrep
Lightweight static analysis that finds bugs and security issues using pattern matching. Write custom rules specific to AI-generated code patterns. Supports 30+ languages.
Best for: Custom security rules, detecting insecure patterns like eval() and SQL concatenation.
Socket.dev
Purpose-built to detect supply chain attacks in npm, PyPI, and Go packages. Analyzes package behavior at install time, detecting network access, file system reads, and shell execution that indicate malicious intent.
Best for: Catching hallucinated and typosquatted packages before they execute malicious code.
SonarQube
Comprehensive code quality and security platform. Detects bugs, vulnerabilities, and security hotspots across your entire codebase. Provides security-focused quality gates for CI/CD.
Best for: Enterprise-wide code quality dashboards and security compliance tracking.
GitHub Advanced Security
Built-in code scanning, secret scanning, and dependency review for GitHub repositories. Automatically scans PRs for vulnerabilities and blocks merges when critical issues are found.
Best for: Teams already on GitHub. Catches leaked secrets and known vulnerability patterns in pull requests.
npm audit / pip audit
Built-in package manager tools that check installed dependencies against known vulnerability databases. Free, fast, and should be the minimum security check in every project.
Best for: Quick baseline dependency checks. Run in CI/CD and fail builds on critical vulnerabilities.
Frequently Asked Questions
Hallucinated dependencies occur when AI coding assistants suggest importing packages that do not exist. The AI confidently recommends a package name that sounds plausible but was never published to any registry. Attackers exploit this by monitoring common hallucinations and registering those package names with malicious code. When developers follow the AI suggestion and install the package, they unknowingly execute the attacker's code.
AI models are trained on vast amounts of code, much of which uses outdated or insecure patterns. They frequently generate code that concatenates user input directly into SQL queries, shell commands, or HTML output without proper sanitization or parameterized queries. Because the generated code is syntactically correct and functionally works, developers accept it without recognizing the injection vulnerability lurking beneath.
Yes, and this is one of the most dangerous aspects. AI-generated code often passes basic linting, some SAST tools, and even manual review because the patterns look syntactically correct. The vulnerabilities are frequently logical rather than syntactic - for example, an authentication flow that validates the JWT format but does not verify the signature, or input validation that checks length but not content. These require deeper security expertise to detect.
A layered approach works best. Use Semgrep for custom pattern-based detection of insecure code patterns. Use Snyk or Socket.dev for dependency and supply chain scanning. SonarQube provides comprehensive SAST coverage. GitHub Advanced Security catches secrets and known vulnerability patterns in pull requests. For the baseline, npm audit and pip audit should run in every CI/CD pipeline. No single tool catches everything - combine multiple layers for defense in depth.
Rather than blanket bans, establish security-specific review requirements for AI-generated code in sensitive areas. Require manual security review by a security engineer for any AI-generated code touching authentication, authorization, cryptography, input handling, or data access layers. Use AI assistants for boilerplate and utilities where the security risk is lower, and apply stricter human oversight where the stakes are higher.
Start with lockfiles to pin exact dependency versions. Verify package provenance using tools like npm provenance or Sigstore. Maintain an approved package allowlist and require security team approval for new dependencies. Scan for typosquatting using Socket.dev or similar tools. Audit all new dependencies added to projects, especially those suggested by AI assistants. Consider running a private registry mirror that only contains vetted packages.
Secure Your AI-Assisted Development
AI coding assistants are powerful tools - but only when paired with rigorous security practices. Explore our supply chain security guide and governance framework to build a complete defense.