“Four LLM loss functions → four flavors of LLM misalignment” by Steven Byrnes

It seems to me that, for every loss function that we use to train LLMs, we get a very distinct flavor of LLM misalignment. Here's the summary table, and then we’ll go through the rows separately. Training stage Loss function Flavor of misalignment Famous examples Pretraining & SFT Imitative learning (next-token prediction) “Seven deadly sins” misalignment Bing-Sydney, “Emergent misalignment” RLHF & DPO Human approval “Glazing” misalignment GPT-4o RLVR Automatic verifier “Literal genie” misalignment HuggingFace hacking RLAIF Approval from another LLM “Trickster” misalignment “Current AIs seem pretty misaligned to me” Warning: I’m not an LLM power-user myself, but rather relying on reports I’ve read. Also, I don’t consider LLM alignment to be my primary area of expertise. I’m open to feedback! 1. Imitative learning → “seven deadly sins” misalignment Training stage Loss function Misaligned behavior Pretraining, SFT Imitative learning (next-token prediction) Any and all of the vices of humanity In imitative learning, the LLM tries to predict what the next token of text will be. Then those predictions magically turn into its outputs. See my earlier discussion: “LLM pretraining magically transmutes observations into behavior, in a way that is profoundly disanalogous to how brains work”. This leads to LLM behavior [...] ---Outline:(00:55) 1. Imitative learning → "seven deadly sins" misalignment(04:24) 2. Human approval → "glazing" misalignment(06:35) 3. Automatic verifiers → "literal genie" misalignment(08:05) 4. LLM judges → "trickster" misalignment(12:06) Afterword The original text contained 1 footnote which was omitted from this narration. --- First published: August 10th, 2026 Source: https://www.lesswrong.com/posts/GRmvZsHXH4vaijPMv/four-llm-loss-functions-four-flavors-of-llm-misalignment --- Narrated by TYPE III AUDIO.