God Help Us, Let's Try To Learn About Mechanistic Interpretability Techniques

The Story So Far Mechanistic interpretability is the science of "reading an AI's mind". Large language models are "grown, not built". Researchers run training data through a neural network. Eventually this creates a working AI; nobody really knows how. But a neural network is just a set of simulated neurons on a computer. The person with the computer can see the neurons, the connections between them, and which ones activate when the AI answers questions. So it seems like it should be possible to "reverse engineer" the AI. This would be scientifically useful: understand how AIs work, with possible relevance to human cognition. It could also be practically useful: find the circuits responsible for dishonesty, hallucination, bias, and other negative behaviors, then redesign those circuits. Unfortunately this is very hard. A modern AI has millions of neurons and trillions of connections between them (aka "parameters", "weights"). Their structure is apparently nonsensical: researchers started by seeking a 1:1 mapping between neurons and concepts, like a neuron that always fired when the AI was thinking about cats, but quickly learned that nothing like that existed. Their first big breakthrough came in 2023. Rather than a 1-to-1 neuron-to-concept mapping, they discovered a many-to-many mapping. Neurons 1 and 2 firing together might mean cat; 1 and 3 firing together might mean chair; 2 and 3 firing together might mean bread. Such mappings let AIs with "only" tens of thousands of neurons represent millions of concepts. There were far too many of these combinations for humans to map, but researchers set other, smaller AIs to mapping them and met with some early success - like this functional combination of neurons ("feature") representing "God" in an artificially-simple 512-neuron AI: https://www.astralcodexten.com/p/god-help-us-lets-try-to-learn-about.