″“Did you lie?” Evaluating Lie Detectors across Model Scale and Belief-Verified Model Organisms” by Alan Cooney, David Africa, Geoffrey Irving

TL;DR.  Lie detectors for LLMs could be valuable for auditing and monitoring. But evaluating them requires testbeds where the model verifiably believes the opposite of what it says, which isn’t straightforward. We determine that most existing trained model organisms don't clear this bar. We train 13 reasoning model organisms, with evidence they hold the alternative belief in chain-of-thought, as well as evidence that they have generalised out of distribution. We also build a broad prompted-lying testbed. On these, detectors scale positively with model capability when lying is prompted, but every activation- and logprob-based detector drops sharply when lying is trained in.Existing lie detectors are therefore not capable enough for making high-confidence claims about when models are lying, though they may still be useful as part of a broader toolkit. View our paper here and our datasets, model organisms and trained detectors here. Overview. We evaluate four lie detectors (a) — a chain-of-thought monitor, two activation probes (Apollo and our new Did-You-Lie probe), and a logprob classifier — in two regimes. In the first, subject models are prompted to lie: we evaluate on our Varied Deception testbed across 31 open-weight subject models spanning 2B–1000B parameters (b). In the [...] ---Outline:(00:16) TL;DR.(02:42) Two hopes for lie detection(04:03) The need for better model organisms of deception(09:14) Results The original text contained 1 footnote which was omitted from this narration. --- First published: June 17th, 2026 Source: https://www.lesswrong.com/posts/xFbMTnmDAucwE8Cdm/did-you-lie-evaluating-lie-detectors-across-model-scale-and --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.