“o1 is a bad idea” by abramdemski

This post comes a bit late with respect to the news cycle, but I argued in a recent interview that o1 is an unfortunate twist on LLM technologies, making them particularly unsafe compared to what we might otherwise have expected:The basic argument is that the technology behind o1 doubles down on a reinforcement learning paradigm, which puts us closer to the world where we have to get the value specification exactly right in order to avert catastrophic outcomes. RLHF is just barely RL. - Andrej KarpathyAdditionally, this technology takes us further from interpretability. If you ask GPT4 to produce a chain-of-thought (with prompts such as "reason step-by-step to arrive at an answer"), you know that in some sense, the natural-language reasoning you see in the output is how it arrived at the answer.[1] This is not true of systems like o1. The o1 training rewards [...] The original text contained 1 footnote which was omitted from this narration. --- First published: November 11th, 2024 Source: https://www.lesswrong.com/posts/BEFbC8sLkur7DGCYB/o1-is-a-bad-idea --- Narrated by TYPE III AUDIO.