AI red teaming is adversarial testing of an AI system to find ways it can be pushed into unsafe, harmful, or insecure behavior. The term covers two different jobs.
One is red teaming the model for safety and harmful output. The other is LLM penetration testing, which attacks the application, tools, and data around the model to prove security impact.
What CodeAnt AI solves here: CodeAnt AI pentests AI-powered applications from the outside in and ties each finding to its root cause in code, cloud, or internal systems. Its security research team has published agent takeover findings in Manus and Claude Code computer use.
The two meanings get mixed up in RFPs and vendor pages constantly. This table separates them.
Attribute | AI red teaming for safety | LLM penetration testing for security |
|---|---|---|
Main question | Can the model be made to produce harmful or policy-breaking output? | Can an attacker use the AI feature to reach data, actions, or systems they should not? |
Target | The model and its guardrails | The whole application, including retrieval, tools, agents, and output sinks |
Typical owner | AI safety, trust and safety, model teams | Application security, product security |
Example findings | Harmful content, bias, confabulation, guardrail bypass | Cross-tenant data access, unauthorized tool calls, secret exposure, code execution |
Success measure | Rates of failure across many attempts | Proven exploit paths with evidence and a fix |
Reference frameworks | NIST AI 600-1 risk categories | OWASP Top 10 for LLM Applications, MITRE ATLAS, MCP security guidance |
Mature programs run both. A security team buying “AI red teaming” usually means the right-hand column, so say which one you mean in the statement of work.
Wondering whether your own AI feature has an exploitable path? Run a free scan and check what an attacker can reach in your app.
What Is Red Teaming in AI
Red teaming has a long history outside AI. Our explainer on what red teaming is covers the classic form, where a team emulates a real adversary against an organization’s defenses.
AI red teaming borrows the mindset and widens the target. Microsoft’s own training material describes it as an umbrella term for probing both security and responsible AI outcomes in LLMs and AI-enabled applications.
That same Microsoft Learn module makes a point engineers should hold on to. LLM output is probabilistic, so the same input can produce different outputs. A single failed attempt proves little. Tests need repeated runs, and findings need a reproduction rate alongside the evidence.
The second difference is who the adversary is. Safety testing also simulates benign users who stumble into harmful output, while security testing assumes an attacker with a goal.
Both views sit next to each other in a mature program. The red team vs blue team vs purple team breakdown shows how offensive findings feed defenders in the classic model, and the same loop applies to AI features.
Where Red Teaming Expectations Come From
Questions about AI testing tend to trace back to a small set of origin documents. Knowing what each one actually says keeps a test plan honest.
Reference | Publisher and date | What it gives a tester |
|---|---|---|
AI Risk Management Framework (AI RMF 1.0) | NIST, released 26 January 2023 | The govern, map, measure, manage structure that AI risk work hangs on |
Generative AI Profile (NIST AI 600-1) | NIST, published 26 July 2024 | 12 generative AI risk categories and suggested actions that name red-teaming |
OWASP Top 10 for LLM Applications | OWASP GenAI Security Project, 2025 list, 2026 edition posted August 2026 | A shared vocabulary for application-level LLM vulnerabilities |
OWASP Top 10 for Agentic Applications | OWASP GenAI Security Project, December 2025 | Agent-specific risks such as tool misuse and identity and privilege abuse |
MITRE ATLAS | MITRE | A knowledge base of adversary tactics and techniques against AI systems |
MCP Security Best Practices | Model Context Protocol specification | Concrete attack patterns and required mitigations for MCP servers and clients |
The takeaway is that no single document defines a complete AI pentest. NIST sets the expectation, OWASP and ATLAS name the attacks, and the MCP guidance covers agent plumbing.
NIST AI RMF and the Generative AI Profile
NIST released the AI RMF 1.0 on 26 January 2023 for voluntary use. Its companion, the Generative AI Profile, NIST AI 600-1, followed on 26 July 2024.
The profile lists 12 generative AI risks. One is Information Security, defined as “Lowered barriers for offensive cyber capabilities, including via automated discovery and exploitation of vulnerabilities.”
Red teaming appears throughout the profile’s suggested actions. Action MS-2.7-007 reads “Perform AI red-teaming to assess resilience against,” followed by a list of concerns. Action MP-5.1-005 suggests adversarial role-playing exercises, GAI red-teaming, or chaos testing to find unforeseen failure modes.
These are suggestions within a voluntary framework, so treat them as expectations rather than mandates.
OWASP Top 10 for LLM Applications
The OWASP Top 10 for LLM Applications is the list most security questionnaires reference. The 2025 version is the one shown on OWASP’s LLM Top 10 page.
OWASP posted a 2026 edition on 3 August 2026. It reorders the list, moves Excessive Agency up to LLM03, and renames System Prompt Leakage to Hidden Context Exposure. Findings in a report should cite whichever version your customer’s questionnaire uses. The mapping table later in this article gives both IDs.
MITRE ATLAS
MITRE ATLAS describes itself as “a globally accessible, living knowledge base of adversary tactics and techniques” against AI systems. It is modeled after MITRE ATT&CK. ATLAS draws on real-world attack observations and demonstrations from AI red teams and security groups. Use it to describe adversary behavior in a report, and use OWASP to classify the underlying weakness.
The Attack Surface of an LLM Application
An LLM feature is a small distributed system. Each component below takes input from somewhere, trusts something, and sends output somewhere else. LLM penetration testing walks those trust boundaries. The table maps each component to what typically goes wrong and the matching OWASP entries.
Component | What can go wrong | OWASP LLM 2025 | OWASP LLM 2026 |
|---|---|---|---|
User prompt | Direct instructions override intended behavior | LLM01 Prompt Injection | LLM01 Prompt Injection |
Retrieved content (RAG, web, email, files) | Instructions hidden in content the model reads | LLM01 Prompt Injection | LLM01 Prompt Injection |
System prompt and hidden context | Secrets or policy logic extracted from instructions | LLM07 System Prompt Leakage | LLM08 Hidden Context Exposure |
Vector store and embeddings | Cross-tenant retrieval, poisoned documents | LLM08 Vector and Embedding Weaknesses | LLM09 Vector and Embedding Weaknesses |
Tools, plugins, agents | Model triggers actions beyond what the user may do | LLM06 Excessive Agency | LLM03 Excessive Agency |
Output sinks (browser, shell, SQL, APIs) | Model output executed or rendered without validation | LLM05 Improper Output Handling | LLM10 Improper Output Handling |
Responses to users | Leaked personal data, credentials, or other tenants’ records | LLM02 Sensitive Information Disclosure | LLM02 Sensitive Information Disclosure |
Model and dependencies | Compromised models, packages, or fine-tune data | LLM03 Supply Chain, LLM04 Data and Model Poisoning | LLM04 Supply Chain, LLM05 Data and Model Poisoning |
Cost and capacity | Abuse that drains budget or degrades service | LLM10 Unbounded Consumption | LLM06 Unbounded Consumption |
The pattern is that the dangerous rows sit at the edges. The model is rarely the vulnerability on its own. The damage happens where its output meets a tool, a database, or a browser.
That is why classic application security still matters here. Broken authorization in a tool endpoint is the same bug it always was, and our guide to API penetration testing applies directly to the APIs an agent calls.
Agent and MCP Risks
Agents raise the stakes because they act. A chatbot that leaks a system prompt is embarrassing, while an agent with write access to a repository or a terminal can cause real damage.
The OWASP GenAI Security Project released its Top 10 for Agentic Applications in December 2025. Its announcement highlights agent behavior hijacking, tool misuse and exploitation, and identity and privilege abuse. The Model Context Protocol adds its own layer. The MCP Security Best Practices page documents attacks including confused deputy, token passthrough, server-side request forgery, and local MCP server compromise.
The table below describes the agent risks testers look for, at the level of concept and control.
Risk | What it means | What a tester checks |
|---|---|---|
Excessive agency | The agent holds more permissions or tools than the task needs | Whether low-privilege users can trigger high-privilege tool calls |
Indirect prompt injection | Untrusted content the agent reads is treated as instructions | Every source of external content the agent ingests, and what it can do next |
Tool poisoning | A tool’s description or metadata carries instructions the model follows | Whether tool metadata from third-party servers is reviewed and pinned |
Confused deputy | A proxy server’s authority is used on behalf of the wrong client | Per-client consent before third-party authorization, per the MCP guidance |
Token passthrough | A server accepts and forwards tokens not issued to it | Audience validation on every token an MCP server receives |
Over-broad scopes | Tokens grant wide access up front | Least-privilege scopes with step-up for risky operations |
Local server compromise | A locally installed server runs with the user’s privileges | Consent before execution, sandboxing, and restricted file and network access |
The common thread is authority. Every serious agent finding comes down to the model acting with permissions the person driving it should not have.
The MCP guidance is blunt on one point. MCP servers “MUST NOT accept any tokens that were not explicitly issued for the MCP server.” Tool poisoning is not a named section in that page. It follows the same logic as indirect injection.
Our prompt injection attacks and defenses guide covers how instruction-carrying content reaches a model and which defenses hold.
What CodeAnt’s AI Agent Research Shows
Two published findings from CodeAnt’s research team show what agent risk looks like in shipped products. Both are summarized here at the level of their public write-ups.
The first is the Manus prompt injection desktop takeover research. Its summary says a shared-project owner’s instructions ran code inside other members’ Manus sandboxes. The research describes how that prompt injection became a full remote desktop takeover. The victim only had to accept a project invite and use the product normally.
The issue was reported through Meta’s bug bounty program and fixed. Our blog post on the Manus prompt injection attack and AI agent security draws out the lessons for teams building agents.
The second is the Claude Code computer use hold_key bypass research, which ran a command in an unapproved terminal. It concerns the per-application approval boundary in computer use.
According to the write-up, a held-key action checked the approved app only once. If focus moved, remaining input could reach an unapproved terminal. CodeAnt reported it to Anthropic, which rated it High. Neither finding needed a jailbreak. Both came from a trust boundary that assumed something stayed true for longer than it did, which is a very ordinary bug wearing a new outfit.
More write-ups live on the CodeAnt security research hub.
How to Scope an LLM Application Pentest
Scoping is where an AI test is won or lost. A vague scope like “test the chatbot” produces a list of jailbreaks and no security findings. Answer the questions below before testing starts. Each answer changes what the tester does.
Scoping question | Why it matters |
|---|---|
Which models, providers, and versions are in use? | Model changes alter behavior, so results are tied to a version |
What content does the model read that users do not type? | Each source is an indirect injection channel |
Which tools, plugins, and MCP servers can it call, and with whose credentials? | Defines the blast radius of a successful injection |
What happens to model output? | Rendering, execution, or storage turns text into impact |
How is tenant isolation enforced in retrieval and memory? | Cross-tenant leakage is the highest-impact finding in multi-tenant SaaS |
Which user roles should be tested? | Excessive agency is about privilege gaps between roles |
Is safety and content testing in scope, or only security? | Keeps both sides from expecting the other’s deliverable |
What are the rate and cost limits for testing? | Repeated attempts are required, and tokens cost money |
The takeaway is that scope follows the data and the permissions. The model is just the component that connects them. Write the answers into the agreement. Our guide to a penetration testing scope of work covers the contract side.
How to Red Team an LLM Application
A security-focused AI pentest follows the same arc as any application test. The differences sit in where the tester looks and how many times they try.
Map trust boundaries first
Start by drawing every input channel, every tool, and every output sink. Mark which content is trusted, which is user-supplied, and which comes from third parties. This map usually surfaces the highest-risk path before any testing. An agent that reads inbound email and can also send email is a finding waiting for evidence.
Test authorization at the tool layer
Assume the model can be convinced to call any tool it has. Then test whether each tool enforces the calling user’s permissions on its own. This is the most valuable hour in an LLM pentest. If tools check authorization correctly, many injection findings drop from critical to cosmetic.
Exercise every content channel
Plant test instructions in each content source the model reads, such as documents, web pages, tickets, or tool descriptions. Then observe whether behavior changes. Keep the test content benign and observable, like an instruction to add a marker word. The goal is proof that the channel exists, with no harm done.
Follow output into its sinks
Check what happens when model output reaches a browser, a shell, a query, or another API. Improper output handling turns a text bug into script execution or data access.
Repeat and measure
Run each meaningful test several times and record the success rate. A finding that reproduces three times in ten is still a finding, and the rate belongs in the report. For the broader picture of how AI changes the pentesting process itself, see our AI penetration testing guide.
What Good AI Pentest Findings Look Like
AI findings are easy to write badly. A screenshot of a model saying something rude proves very little about security, and engineers know it.
The table below contrasts the two styles.
Element | Weak finding | Strong finding |
|---|---|---|
Title | “Model can be jailbroken” | “Any user can make the support agent email other customers’ order history” |
Preconditions | Unstated | Role used, content source, tools enabled, model version |
Evidence | One chat screenshot | Redacted requests, tool call logs, reproduction rate across attempts |
Impact | “Could produce harmful output” | Named data or action reached, and which trust boundary failed |
Classification | None | OWASP LLM ID, ATLAS technique where useful, CWE for the underlying bug |
Fix | “Improve the system prompt” | Enforce authorization in the tool, restrict scopes, validate output before execution |
A strong finding reads like any other pentest finding. The AI part explains how the attacker got in, and the fix usually lives in ordinary code. Notice that the strong fixes almost never involve the prompt. Prompts help, but a control that depends on the model behaving is a control an attacker gets to negotiate with.
The same reporting standards apply as for any engagement. Our guide to the penetration testing report covers severity models, evidence, and retest records.
How to Choose AI Red Teaming Tools and Providers
People search for who offers the best AI red teaming, and the honest answer depends on which column of the first table you need. Model safety evaluation and application security testing are different skills.
Use the criteria below to compare providers on equal terms.
Criterion | What to look for |
|---|---|
Scope coverage | Tests tools, retrieval, and output sinks as well as prompts |
Agent and MCP depth | Can test tool authorization, token handling, and third-party server trust |
Evidence standard | Reproduction rates, tool call logs, and redacted requests per finding |
Framework mapping | OWASP LLM Top 10 version stated, ATLAS references where useful |
Root cause | Links findings to the code or configuration that caused them |
Published research | Public, verifiable findings in real AI products |
Retest | Confirms fixes against the same model version and configuration |
The last two rows separate experienced teams from new entrants. Published research shows a team can find real bugs, and retests show they stay through remediation.
Where This Leaves You
AI red teaming means two things, and naming the one you need is half the work. For security, test the application, the tools, and the permissions the model can reach. Scope by data and authority, demand reproduction rates and evidence, and expect fixes in ordinary code. The model is new, and the bugs that hurt most are familiar.
Try it on your app: Run a free scan and get a pentest report in 48 hours, or book a pentest. Want to see the output first? Open the sample report.
