Lab note ·

How a nation state could skew what AI models learn

Language models learn from the open web, and the open web can be seeded. Three routes by which a government could tilt a model's answers, and what the research says about how little it takes.

Note

Why this is possible at all

A language model learns from text collected from the public internet. Nobody reads that text first; there is too much of it. Whoever can put enough of the right text in the right places has a say in what the model comes to treat as normal, true or typical. States have the money, the patience and the motive.

Route one: poison the data directly

It takes far less than people assume. In 2023 researchers showed that an attacker could have altered a small share of the well-known web-scale training sets for about $60, by buying expired web addresses that the datasets still pointed to (Carlini and colleagues, "Poisoning Web-Scale Training Datasets is Practical").

In 2025 a study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that around 250 planted documents were enough to install a hidden trigger in a model, and that the number did not grow as the model and its training data got larger (summary from Anthropic). The trigger they tested was harmless. The point is the size of the number.

Route two: flood the web the model reads

A state does not need to touch a dataset if it can fill the web with its version of events. Researchers have named this "LLM grooming": publishing the same claims across hundreds of low-traffic sites, written less for people than for the crawlers that feed AI systems. Audits by news-rating groups in 2025 reported that leading chatbots repeated claims traced to such networks a meaningful share of the time. We have not verified those audits ourselves and link none here; treat the figures you see quoted with care.

Models that search the web at answer time are exposed twice: once in training, and again every time they fetch a seeded page.

Route three: shape the model at home

A government can also require models built in its jurisdiction to answer in line with its positions. Models from different countries visibly differ on politically sensitive history. That is not hidden poisoning, but it has the same effect on the user who does not know which model they are talking to.

What can be checked

  • Ask the same question of many models. Divergence on a factual question is a signal. This is what our one-word experiments do for contested questions.
  • Trace sources. When a model cites, follow the citation to see what kind of site it is.
  • Know where the model was built and under whose rules.

What is not known

  • Whether any state has actually succeeded in poisoning a leading model's training data. The research shows it is practical; public proof that it has happened is thin.
  • How well model makers' filtering catches seeded content. They do not publish enough to judge.

Compiled from public reports with AI assistance. Follow the sources for the full account. Not legal advice.

Contact

Working on this too, or have something to add?