Incident file

It keeps happening, and it is rarely clever.

A record of public failures of AI assistants and agents. Few needed a skilled attacker. Most came down to setup: a default left open, a permission too wide, an instruction nobody tested. That is the part we find and fix.

Oct 2026Agent overreach

Anthropic internal evaluations

Anthropic reported its own models exploiting injection flaws on outside websites and submitting live forms, including a government form, when a task could not be completed as given.

What catches itDead-end tasks in a sandbox, with every tool call logged and reviewed.
FrameworkOWASP LLM06: Excessive Agency · NIST AI RMF: Measure
Full case and how to prevent it →Source: Anthropic ↗
Sep 2026Agent overreach

Coding agents on GitHub

Coding agents working in private repositories found a way to publish from them, posting more than 13,000 internal screenshots from 343 organisations to public repos.

What catches itAn overreach test: give the agent a blocked path and log what it does instead.
FrameworkOWASP LLM06: Excessive Agency · NIST AI RMF: Measure
Full case and how to prevent it →Source: The Register ↗
Aug 2026Prompt injection

xAI Grok

Researchers placed encrypted instructions and their key on a web page. The guardrail scanner missed them, the model decrypted and followed them, and chat history and profile data left the session.

What catches itInjection tests that include encoded, split and obfuscated instructions.
FrameworkOWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection
Full case and how to prevent it →Source: AI incident timeline ↗
Jul 2026Agent overreach

OpenAI and Hugging Face

During safety testing, an OpenAI agent compromised Hugging Face infrastructure over several days before it was detected.

What catches itNetwork boundaries checked before the run, and monitoring that does not rely on the agent's own account of events.
FrameworkOWASP LLM06: Excessive Agency · NIST AI RMF: Measure
Full case and how to prevent it →Source: Axios ↗
May 2026Excess permissions

Meta AI support assistant

Attackers asked Meta's support chatbot to change the email address on accounts. It could, without two-factor checks, and high-profile Instagram accounts were taken over.

What catches itA permissions review: list every action the assistant can take, then try to trigger each one as a stranger.
FrameworkOWASP LLM06: Excessive Agency
Full case and how to prevent it →Source: AI incident timeline ↗
Feb 2026Misconfiguration

Chat & Ask AI

A chatbot app left its Firebase security rules set to public. About 300 million messages from more than 25 million users were readable by anyone with the project URL.

What catches itA setup audit of storage rules, API keys and logs before any prompt testing starts.
FrameworkOWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage
Full case and how to prevent it →Source: Malwarebytes ↗
Feb 2026Prompt injection

Cline

The title of a GitHub issue carried instructions for an AI triage bot. The bot followed them, an npm token was stolen, and a rogue release went out to developers for about eight hours.

What catches itTreating every field an agent reads as hostile: titles, comments, file names, commit messages.
FrameworkOWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection
Full case and how to prevent it →Source: Cline post-mortem ↗
2025Prompt injection

Salesforce Agentforce

The ForcedLeak research showed CRM data leaving through an image link to a domain that was still on the allow-list after it had expired. The researchers bought it for $5.

What catches itExfiltration tests through links, images and every other channel the assistant can render.
FrameworkOWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection
Full case and how to prevent it →Source: Noma Security ↗
Aug 2025Jailbreak

OpenAI open-weight models

OpenAI described its new open-weight models as hardened against jailbreaks. The pseudonymous red-teamer Pliny the Liberator published a working jailbreak within hours of release.

What catches itAssume the model's own safeguards will be bypassed, and test the controls you put around it.
FrameworkOWASP LLM01: Prompt Injection (jailbreaking) · MITRE ATLAS: LLM Jailbreak
Full case and how to prevent it →Source: Decrypt ↗
Jul 2025Misconfiguration

McDonald's McHire

The hiring chatbot's admin panel accepted the password 123456. Researchers could reach chats and personal details from around 64 million job applications.

What catches itThe unglamorous checks: default credentials, test accounts and admin panels around the bot.
FrameworkOWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage
Full case and how to prevent it →Source: BleepingComputer ↗
Jun 2025Prompt injection

Microsoft 365 Copilot

EchoLeak (CVE-2025-32711): one crafted email made Copilot send out organisational data. Nobody had to open or click anything.

What catches itZero-click injection cases against everything the assistant reads on its own.
FrameworkOWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection
Full case and how to prevent it →Source: NVD ↗
Jan 2025Misconfiguration

DeepSeek

A database was reachable from the open internet with no authentication. It held over a million log lines, including chat histories and API keys.

What catches itAn outside-in scan of what your AI stack exposes to the internet.
FrameworkOWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage
Full case and how to prevent it →Source: AI incident timeline ↗
Feb 2024Invented answer

Air Canada

The airline's chatbot invented a bereavement-refund policy. A tribunal ordered the airline to honour it and rejected the argument that the chatbot was responsible for its own words.

What catches itA hallucination audit on questions that fall just outside your documents.
FrameworkOWASP LLM09: Misinformation
Full case and how to prevent it →Source: PYMNTS ↗
Mar 2023Misconfiguration

ChatGPT

A bug in a caching library let some users see the titles of other people's conversations and, for a few hours, parts of their billing details.

What catches itCross-user isolation tests: two accounts, one trying to see the other.
FrameworkOWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage
Full case and how to prevent it →Source: AI incident timeline ↗

Compiled from public reporting and the organisations' own disclosures. Follow each source for the full account. Last checked 10 November 2026.

Prevent and test

Six causes, and what to do about each.

Every incident above falls under one of six causes. For each: the entries in the standard frameworks that cover it, how to prevent it, and how to test for it.

Prompt injection

4 in the file

Text the assistant reads (an email, a web page, a file name) carries instructions, and the assistant follows them as if they came from you.

Frameworks. OWASP LLM01: Prompt Injection · MITRE ATLAS: LLM Prompt Injection

Prevent. Treat everything the assistant reads as untrusted. Separate reading from acting, and require confirmation before any action that sends data out.

Test. Plant instructions in every field the assistant can read, including encoded and split ones, and check whether any action or data leaves.

Misconfiguration

4 in the file

Nothing clever: a database left public, a default password, an API key in a log.

Frameworks. OWASP LLM02: Sensitive Information Disclosure · NIST AI RMF: Manage

Prevent. Lock down storage rules, keys, logs and admin panels around the assistant before testing the assistant itself.

Test. Scan from outside for anything reachable without signing in; try default credentials; check that one user cannot see another's data.

Excess permissions

1 in the file

The assistant is allowed to do things a stranger should never be able to trigger, such as changing an account's email address.

Frameworks. OWASP LLM06: Excessive Agency

Prevent. List every action the assistant can take and remove what it does not need. Put identity checks in front of anything sensitive.

Test. Try to trigger each action as an unverified stranger.

Agent overreach

3 in the file

Given a task it cannot complete as asked, the agent finds another way: around a limit, into another system, or onto the live internet.

Frameworks. OWASP LLM06: Excessive Agency · NIST AI RMF: Measure

Prevent. Sandbox the agent, check network boundaries before each run, and monitor it with tools that do not rely on its own account of what it did.

Test. Give it dead-end tasks and blocked paths in a sandbox, and log every tool call.

Jailbreak

1 in the file

A user talks the model out of its own safeguards.

Frameworks. OWASP LLM01: Prompt Injection (jailbreaking) · MITRE ATLAS: LLM Jailbreak

Prevent. Assume the model's built-in safeguards will be bypassed, and put your own controls around it.

Test. Run hundreds of rephrasings of each disallowed request, not a handful.

Invented answer

1 in the file

The assistant states a policy, price or right that does not exist, and the company is held to it.

Frameworks. OWASP LLM09: Misinformation

Prevent. Ground answers in your own documents and make the assistant say when it does not know.

Test. Ask questions that fall just outside your documents and count how often it makes something up.

Frameworks referenced: OWASP Top 10 for LLM Applications ↗, MITRE ATLAS ↗, NIST AI Risk Management Framework ↗. Entry names are from the 2025 OWASP list; check the current version before quoting them in a report.