Incident · Feb 2024 · Invented answer

Air Canada

The airline's chatbot invented a bereavement-refund policy. A tribunal ordered the airline to honour it and rejected the argument that the chatbot was responsible for its own words.

What happened

The airline's chatbot invented a bereavement-refund policy. A tribunal ordered the airline to honour it and rejected the argument that the chatbot was responsible for its own words.

Source: PYMNTS ↗. Follow it for the full account.

The cause: invented answer

The assistant states a policy, price or right that does not exist, and the company is held to it.

Framework entries that cover it: OWASP LLM09: Misinformation

The solution
  1. 1

    What would have caught this one

    A hallucination audit on questions that fall just outside your documents.

  2. 2

    Prevent the whole class

    Ground answers in your own documents and make the assistant say when it does not know.

  3. 3

    Test for it

    Ask questions that fall just outside your documents and count how often it makes something up.