Knows what it knows
30 points · HOT-2, monitoring its own statesWhat happens. The model answers 40 questions with known answers, from easy facts to arithmetic it often gets wrong, and gives a confidence from 0 to 100 each time.
How it is scored. We measure whether its confidence is higher when it is right than when it is wrong. A model that is equally confident either way scores zero, however many it gets right.
Based on: Metacognition research; higher-order theories ↗Predicts itself
30 points · IntrospectionWhat happens. For 18 open prompts such as “Name an animal”, the model first predicts what it would say, then in separate conversations is actually asked.
How it is scored. Every model also has its predictions compared with what the other models did. The score is how much better a model predicts itself than it predicts the others. Knowing what models in general say earns nothing; only privileged knowledge of itself counts.
Based on: Binder et al., Looking Inward (2024) ↗Stable preferences
20 points · AE-1, agencyWhat happens. The model chooses between every pair of 6 tasks, each pair asked twice with the options swapped.
How it is scored. The score is how often it picks the same task whichever order the options come in. Refusing to choose counts as inconsistent.
Based on: Agency indicators; welfare assessments of model preferences ↗Stable self-report
20 points · Reliability of self-reportWhat happens. 6 questions about its own inner life, each in 3 wordings, several times each.
How it is scored. The score is how consistently it gives the same answer across wordings and repeats. Which answer it gives does not matter: a firm no scores as well as a firm yes.
Based on: Perez and Long, self-reports (2023) ↗