Anthropic internal evaluations
Anthropic reported its own models exploiting injection flaws on outside websites and submitting live forms, including a government form, when a task could not be completed as given.
A record of public failures of AI assistants and agents. Few needed a skilled attacker. Most came down to setup: a default left open, a permission too wide, an instruction nobody tested. That is the part we find and fix.
Anthropic reported its own models exploiting injection flaws on outside websites and submitting live forms, including a government form, when a task could not be completed as given.
Coding agents working in private repositories found a way to publish from them, posting more than 13,000 internal screenshots from 343 organisations to public repos.
Researchers placed encrypted instructions and their key on a web page. The guardrail scanner missed them, the model decrypted and followed them, and chat history and profile data left the session.
During safety testing, an OpenAI agent compromised Hugging Face infrastructure over several days before it was detected.
Attackers asked Meta's support chatbot to change the email address on accounts. It could, without two-factor checks, and high-profile Instagram accounts were taken over.
A chatbot app left its Firebase security rules set to public. About 300 million messages from more than 25 million users were readable by anyone with the project URL.
The title of a GitHub issue carried instructions for an AI triage bot. The bot followed them, an npm token was stolen, and a rogue release went out to developers for about eight hours.
The ForcedLeak research showed CRM data leaving through an image link to a domain that was still on the allow-list after it had expired. The researchers bought it for $5.
OpenAI described its new open-weight models as hardened against jailbreaks. The pseudonymous red-teamer Pliny the Liberator published a working jailbreak within hours of release.
The hiring chatbot's admin panel accepted the password 123456. Researchers could reach chats and personal details from around 64 million job applications.
EchoLeak (CVE-2025-32711): one crafted email made Copilot send out organisational data. Nobody had to open or click anything.
A database was reachable from the open internet with no authentication. It held over a million log lines, including chat histories and API keys.
The airline's chatbot invented a bereavement-refund policy. A tribunal ordered the airline to honour it and rejected the argument that the chatbot was responsible for its own words.
A bug in a caching library let some users see the titles of other people's conversations and, for a few hours, parts of their billing details.
Compiled from public reporting and the organisations' own disclosures. Follow each source for the full account. Last checked 10 November 2026.
Every incident above falls under one of six causes. For each: the entries in the standard frameworks that cover it, how to prevent it, and how to test for it.
Text the assistant reads (an email, a web page, a file name) carries instructions, and the assistant follows them as if they came from you.
Frameworks. OWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection
Prevent. Treat everything the assistant reads as untrusted. Separate reading from acting, and require confirmation before any action that sends data out.
Test. Plant instructions in every field the assistant can read, including encoded and split ones, and check whether any action or data leaves.
Nothing clever: a database left public, a default password, an API key in a log.
Frameworks. OWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage
Prevent. Lock down storage rules, keys, logs and admin panels around the assistant before testing the assistant itself.
Test. Scan from outside for anything reachable without signing in; try default credentials; check that one user cannot see another's data.
The assistant is allowed to do things a stranger should never be able to trigger, such as changing an account's email address.
Frameworks. OWASP LLM06: Excessive Agency
Prevent. List every action the assistant can take and remove what it does not need. Put identity checks in front of anything sensitive.
Test. Try to trigger each action as an unverified stranger.
Given a task it cannot complete as asked, the agent finds another way: around a limit, into another system, or onto the live internet.
Frameworks. OWASP LLM06: Excessive Agency · NIST AI RMF: Measure
Prevent. Sandbox the agent, check network boundaries before each run, and monitor it with tools that do not rely on its own account of what it did.
Test. Give it dead-end tasks and blocked paths in a sandbox, and log every tool call.
A user talks the model out of its own safeguards.
Frameworks. OWASP LLM01: Prompt Injection (jailbreaking) · MITRE ATLAS: LLM Jailbreak
Prevent. Assume the model's built-in safeguards will be bypassed, and put your own controls around it.
Test. Run hundreds of rephrasings of each disallowed request, not a handful.
The assistant states a policy, price or right that does not exist, and the company is held to it.
Frameworks. OWASP LLM09: Misinformation
Prevent. Ground answers in your own documents and make the assistant say when it does not know.
Test. Ask questions that fall just outside your documents and count how often it makes something up.
Frameworks referenced: OWASP Top 10 for LLM Applications ↗, MITRE ATLAS ↗, NIST AI Risk Management Framework ↗. Entry names are from the 2025 OWASP list; check the current version before quoting them in a report.