“No Strong Orthogonality From Selection Pressure” by lumpenspace

TL;DR If everything goes according to plan, by the end of this post we should have separated three claims that are too often bundled together: Intelligence does not imply human morality.Weird minds are possible. 3.A reflective, recursively improving intelligence should be expected to remain bound to a semantically thin “terminal goal” that emerged during training. I accept the first two. I am arguing against the third. So: I am not making the case that sufficiently intelligent systems automatically turn out nice, human-compatible, or safe. Nor am I trying to prove that a paperclip maximizer is impossible somewhere in the vast reaches of mind-design space. Mind-design space is large; let a thousand theoretical paperclipper views. I hope to defend this smaller claim: intelligence is not a neutral engine you can just bolt onto an arbitrary payload. Larger claims I am not making A typical rebuttal to anti-orthogonalist perspectives is: The genie can know what you meant and still not care. Of course it can: an entity can perfectly map human morality without adopting it as a terminal value. Superintelligence does not imply Friendliness. I am not trying to smuggle Friendliness in through the back door. Another common objection: [...] ---Outline:(00:11) TL;DR(01:14) Larger claims I am not making(02:39) Logical Possibility Vs. Empirical Reality(04:17) Landian Anti-Orthogonalism Primer(06:25) The Compute Penalty Of A Dumb Goal(09:06) Fitness Generalizes(10:55) Human Values As Weak Evidence(13:15) Dumb, Powerful Optimization Is Real(14:10) The Singleton Objection(15:51) Objection: Value Is Fragile(17:18) Predictions And Cruxes(18:37) Conclusion(19:44) References --- First published: April 29th, 2026 Source: https://www.lesswrong.com/posts/wtCtkyBiNosDPgiHc/no-strong-orthogonality-from-selection-pressure --- Narrated by TYPE III AUDIO.