“9 kinds of hard-to-verify tasks” by Cleo Nardo

Introduction Some people talk about "hard-to-verify tasks" and "easy-to-verify tasks" like these are both natural kinds. But I think splitting tasks into "easy-to-verify" and "hard-to-verify" is like splitting birds into ravens and non-ravens. Easy-to-verify tasks are easy for the same reason — there's a known short program that takes a task specification and a candidate solution, and outputs a score, without using substantial resources or causing undesirable side effects.By contrast, "hard-to-verify tasks" is a negative category — it just means no such program exists. But there are many kinds, corresponding to different reasons no such program exists. Listing kinds of hard-to-verify tasks I might update the list if I think of more, or if I see additional suggestions in the comments. Verification requires expensive AI inference. A verifier exists and works fine, but each run costs enough compute that you can't afford the number of labels you'd want. Given two proposed SAE experiments, say which will be more informative. Running both to find out costs $100–$1000 per comparison.Given two research agendas (e.g. pragmatic vs ambitious mech interp), say which produces more alignment progress. Same structure, but each comparison costs millions.Verification requires expensive human time. The verifier [...] ---Outline:(00:10) Introduction(00:55) Listing kinds of hard-to-verify tasks(04:38) Implications --- First published: April 20th, 2026 Source: https://www.lesswrong.com/posts/NEscrkxr9SxHpGayB/9-kinds-of-hard-to-verify-tasks --- Narrated by TYPE III AUDIO.