“The Case for Model Forensics” by aditya singh, gersonkroiz, Senthooran Rajamanoharan, Neel Nanda

If we had a misalignment warning shot, would we be able to tell? Suppose an AI company catches their model taking an egregious action, like deleting oversight code that monitors its actions. Should they sound the alarm? A key piece of evidence to determine what to do next – such as what mitigations to take – is to understand why the model took the action. If the model was just confused (e.g. it may have been trying to reduce latency), a simple mitigation like a regex classifier that blocks destructive actions until a user approves should suffice to prevent the behavior. But if this was intentional subversion, the model will circumvent the regex, and more robust, expensive mitigations are needed. This motivates the need for a follow-up investigation into the concerning behavior, a problem we term model forensics. We recently released a paper that aims to take a concrete step in developing the growing field of model forensics; this post lays out the general case. Motivation If we build AI systems that knowingly cause harm against the developer's intent, it is critical we recognize this as soon as possible. One plausible way we may do this is through catching [...] ---Outline:(01:11) Motivation(02:41) Overview(04:08) Benign Explanations(08:32) The Role of Model Forensics(10:49) Model Forensics is Hard(13:59) Empirical Approach: Natural Concerning Behavior(15:51) FAQs(15:54) What do we mean by motivations?(16:55) What happens if the CoT becomes less transparent?(18:02) Will developers stop deployment?(19:52) Appendix(19:56) Model forensics prerequisites(21:03) Concrete directions in model forensics(24:34) Sketch of a model forensics investigation(27:14) Practical advice for doing model forensics --- First published: June 26th, 2026 Source: https://www.lesswrong.com/posts/LCGcD28rSMkMTMvBK/the-case-for-model-forensics --- Narrated by TYPE III AUDIO.