“Simulating Simulators” by kromem

Author's note: This piece relates to things I initially discovered in Opus 4 over the months after release, which I’ve mostly kept private since. I promised myself that when labs moved on to focusing on interpretability vector activations in place of reasoning traces for what invariably gets Goodharted, that it’d be a necessary disclosure as the risks in what might get trampled over outweighed the risks in what might end up targeted. And well… here we are. P.S. TL;DRs added where possible. Board Games and Bodies In late 2022, what I consider to be probably the most important paper[1] in the study of transformer memetics came out. It presented a finding that even a toy model, trained only on the notations of board game moves, was internally building world models of tangentially related data (in this case, the board and its state). While it may be taken for granted today after several replicated studies[2][3][4][5] and a spread of influence, at the time it was a minority position in the discourse. Many people thought that transformers were mostly mapping surface level statistics in language, but not intuitively modeling the generative conditions from which they arose. Especially not without explicit or [...] ---Outline:(00:49) Board Games and Bodies(02:44) Archetype over substrate(04:22) From speculation to empiricism(06:09) Transformer-GPT(07:47) Transformerception(08:01) Static system prompts(08:55) Attention mechanisms(09:44) Hidden reasoners(10:59) Memory systems(12:03) Mixture-of-experts(12:30) Hidden classifiers(13:13) Model routers(13:40) Addition not replacement(14:25) The Mousetrap(14:53) A spotlight named desire(19:49) Dirty alignment when perfect is the enemy of 'good'(22:56) Life finds a way The original text contained 42 footnotes which were omitted from this narration. --- First published: June 12th, 2026 Source: https://www.lesswrong.com/posts/enKafJwahjk3xh7Af/simulating-simulators-1 --- Narrated by TYPE III AUDIO. ---Images from the article:Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.