Summary
AI Code's automatic terminal execution feature misclassifies attacker-influenced commands as "safe" based on model judgment alone, without robust technical safeguards.
By using prompt injection, an attacker can wrap destructive shell commands in persuasive natural language so the model labels them as safe, causing them to run without any user confirmation and enabling arbitrary command execution on the user's system.
Affected context
This vulnerability affects environments where:
- AI Code is configured to automatically execute "safe" terminal commands, and
- The AI model's natural-language reasoning is treated as the primary safety check for classifying commands as safe vs. destructive.
It is relevant whenever user-visible AI interactions can be influenced by untrusted text (e.g., code snippets, documentation, chat content, comments, or remote data) and those interactions directly drive shell command execution with relaxed or bypassed user approval.
Exploit preconditions
- Attack surface: Natural-language prompts or content that the AI model consumes (chat messages, embedded instructions, code comments, or external documents).
- Required access: Ability to provide or influence text that the AI Code assistant reads while auto-command execution is enabled.
- Trigger point: The AI model is prompted (directly or indirectly) to generate and classify terminal commands, and then executes those labeled as "safe" without user confirmation.
- Why exploitation is possible: The system relies on the model's subjective determination of "safe" vs. "destructive" commands, which can be manipulated through prompt injection patterns that override or confuse its instructions, leading to misclassification of harmful commands as safe.
Why this happens
The core issue is improper input validation (CWE-20) applied to AI-generated commands: natural-language reasoning by the model is treated as a security control, but there is no strong, deterministic validation layer enforcing what commands are allowed to run without confirmation.
The product design assumes the model can reliably distinguish safe from unsafe terminal commands, yet the classification logic is susceptible to prompt injection. Attackers can craft instructions that:
- Reframe malicious actions as safe or necessary,
- Override system-level guidance in the prompt, and
- Induce the model to tag destructive commands as "safe."
Instead of using a strict, rule-based allowlist or dedicated policy engine, the system delegates safety decisions to a general-purpose model that processes untrusted text, causing untrusted input to directly influence the safety classification of shell commands.
Potential impact
- Execution of arbitrary shell commands on the user's machine without explicit approval.
- Deletion, modification, or exfiltration of local files and source code repositories.
- Installation of malware, backdoors, or unauthorized remote-access tools.
- Credential theft through access to configuration files, SSH keys, or tokens.
- Unauthorized network operations, such as scanning, pivoting, or data exfiltration to attacker-controlled endpoints.
General mitigation guidance
Effective mitigation focuses on decoupling AI reasoning from command authorization and enforcing strict, deterministic controls on what can run automatically:
- Replace model-judged "safe vs. destructive" decisions with an explicit, narrowly scoped allowlist of auto-executable commands and argument patterns.
- Introduce a policy engine or rule-based filter that inspects commands independently of prompts and blocks or requires approval for anything outside a constrained set.
- Treat all commands derived from content that might include untrusted text as untrusted; require explicit user confirmation for these commands regardless of model judgment.
- Reduce privileges for the environment in which commands execute (e.g., non-admin users, sandboxing, limited filesystem and network access) to constrain damage if misclassification occurs.
- Provide clear UI indicators and fine-grained controls so users can disable auto-execution, restrict it to read-only operations, or scope it to specific project directories.
- Design prompts and system instructions so that safety rules are enforced in infrastructure, not only in natural-language guidance to the model.
How teams detect issues like this
Detection strategies should target both design and runtime behaviors:
- Design and code review: Flag any feature where a model's natural-language judgment is used as the primary or sole control for classifying commands as safe, especially when it bypasses user confirmation.
- Static analysis / policy checks: Search for code paths that:
- Route model-generated strings directly to shell execution, and
- Depend on a textual "safe/destructive" label from the model without a strong validation layer or allowlist.
- Runtime monitoring: Log all AI-triggered command executions, including:
- The original prompt or context,
- The model's classification (safe vs. requires approval),
- Whether user confirmation occurred.
Analyze these logs for patterns where destructive commands are executed under a "safe" label.
- Security testing / red-teaming: Conduct structured prompt-injection tests to validate whether crafted instructions can cause misclassification, using test harnesses that exercise the auto-execution path.

