MLASWE-0010: Insecure Output Handling¶
Description¶
Insecure output handling arises when an application implicitly trusts the output generated by a Machine Learning model (particularly LLMs) and passes it directly to downstream components without rigorous validation, sanitization, or encoding. This failure enables severe traditional web and system vulnerabilities, allowing the model's output to facilitate Cross-Site Scripting (XSS), Server-Side Request Forgery (SSRF), Remote Code Execution (RCE), or SQL Injection.
Risk¶
- Severity: Critical (frequently results in full system compromise or severe client-side exploitation)
- Exploitability: High (attackers leverage prompt injection to dictate the malicious output)
- Prevalence: Widespread (often caused by the false assumption that model output is inherently safe)
Affected Components¶
- Web front-ends rendering LLM output (Markdown, HTML).
- Back-end execution environments (evaluating LLM-generated code).
- Database drivers accepting LLM-constructed queries.
- System shells or orchestration tools commanded by autonomous agents.
Sub-types¶
| Type | Description | Downstream Risk |
|---|---|---|
| XSS via LLM | LLM generates malicious JavaScript payload rendered unsanitized by the browser. | Session Hijacking / XSS |
| RCE via Code Gen | LLM-generated scripts or commands are executed in a privileged environment. | Full System Compromise |
| Injection via LLM | LLM output is concatenated into SQL, NoSQL, or LDAP queries. | Database Breach |
| SSR / API Abuse | LLM dictates URLs or parameters for downstream backend API requests. | SSRF / API Exploitation |
Detection Methods¶
- Dynamic Application Security Testing (DAST): Security teams MUST perform fuzzing against the model interface to trigger and observe malicious downstream execution.
- Context-Aware Output Scanning: Middleware SHOULD evaluate the LLM output against the specific context (e.g., checking for HTML tags before rendering).
- Schema Validation: Systems MUST rigorously validate that structured outputs (e.g., JSON) strictly adhere to predefined schemas.
Preventive Controls (MLASVS)¶
- MLASVS-LLM-003: Output validation and filtering
- MLASVS-LLM-009: Content filtering pipeline
- MLASVS-LLM-014: Output length limits
- MLASVS-APP-005: Strict output encoding
Attack Techniques (MITRE ATLAS)¶
- AML.T0057: LLM Data Leakage (often combined with output handling flaws)
- AML.T0053: LLM Plugin Compromise
Remediation¶
- Context-Specific Encoding: Applications MUST apply strict, context-appropriate output encoding (e.g., HTML entity encoding, JavaScript string escaping) before rendering any LLM-generated content.
- Zero-Trust Output: Engineers MUST treat all ML model output as untrusted user input. Output MUST NEVER be executed directly via
eval(),exec(), or system shell commands. - Sandboxed Execution: If executing LLM-generated code is a business requirement, it MUST occur within a heavily restricted, ephemeral sandbox (e.g., gVisor, restricted containers) with zero network egress.
- Parameterized Queries: Downstream database interactions MUST exclusively utilize parameterized queries or safe ORMs; LLM output MUST NEVER be concatenated into query strings.
- Content Security Policy (CSP): Web applications MUST implement strict CSP headers to mitigate the impact of any XSS payloads successfully generated by the LLM.
Real-World Examples¶
- ChatGPT Plugin XSS: Researchers demonstrated that ChatGPT could be coerced into outputting malicious Markdown/HTML, resulting in XSS execution within the victim's browser session.
- Autonomous Agent RCE: An experimental AutoGPT instance, lacking output validation, generated and executed a shell command that destroyed local system files.
References¶
- OWASP LLM Top 10: LLM02 (Insecure Output Handling)
- MITRE ATLAS: AML.T0057