Why Transformers Fail at Counting

This episode explores a mechanistic interpretability paper arguing that transformers often fail at counting not because they lack an internal notion of quantity, but because the pathway that converts that latent count into digit tokens is poorly aligned. It explains key ideas like linear probes, the logit lens, attention, LoRA, constrained next-token evaluation, and autoregressive generation to show how the authors separate “the model knows” from “the model can say.” The discussion highlights striking evidence that intermediate hidden states can encode counts almost perfectly while the corresponding digit readout directions remain nearly orthogonal, creating a readout bottleneck. Listeners would find it interesting because it reframes a familiar model weakness into a precise geometric and causal diagnosis, with implications for how to fix generation failures in modern model families like Pythia, Qwen3, and Mistral. Sources: 1. Why Transformers Fail at Counting https://arxiv.org/pdf/2605.03258 2. Teaching Arithmetic to Small Transformers — Andrew McLeish, David Irving, Simon Sokota, Max Black, Berlin Chen, et al., 2024 https://scholar.google.com/scholar?q=Teaching+Arithmetic+to+Small+Transformers 3. Language Models Use Trigonometry to Do Addition — Stephen McLeish, et al., 2024 https://scholar.google.com/scholar?q=Language+Models+Use+Trigonometry+to+Do+Addition 4. Faithfulness of Linear Probes in Transformers — Various probe-critique literature; a representative reference should be cited explicitly by the author, 2019-2024 https://scholar.google.com/scholar?q=Faithfulness+of+Linear+Probes+in+Transformers 5. ROME: Locating and Editing Factual Associations in GPT — Kevin Meng, David Bau, Alex Andonian, Yonatan Belinkov, 2022 https://scholar.google.com/scholar?q=ROME%3A+Locating+and+Editing+Factual+Associations+in+GPT 6. The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets — Wes Gurnee, et al., 2023 https://scholar.google.com/scholar?q=The+Geometry+of+Truth%3A+Emergent+Linear+Structure+in+Large+Language+Model+Representations+of+True%2FFalse+Datasets 7. A Mathematical Framework for Transformer Circuits — Nelson Elhage, et al., 2021 https://scholar.google.com/scholar?q=A+Mathematical+Framework+for+Transformer+Circuits 8. Finding Transformer Circuits with Edge-Level Attribution Patching — Neel Nanda, et al., 2023 https://scholar.google.com/scholar?q=Finding+Transformer+Circuits+with+Edge-Level+Attribution+Patching 9. Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs — Aaditya K. Singh, DJ Strouse, 2024 https://scholar.google.com/scholar?q=Tokenization+counts%3A+the+impact+of+tokenization+on+arithmetic+in+frontier+LLMs 10. Efficient numeracy in language models through single-token number embeddings — Linus Kreitner, Paul Hager, Jonathan Mengedoht, Georgios Kaissis, Daniel Rueckert, Martin J. Menten, 2025 https://scholar.google.com/scholar?q=Efficient+numeracy+in+language+models+through+single-token+number+embeddings 11. Arithmetic-Based Pretraining Improving Numeracy of Pretrained Language Models — Dominic Petrak, Nafise Sadat Moosavi, Iryna Gurevych, 2023 https://scholar.google.com/scholar?q=Arithmetic-Based+Pretraining+Improving+Numeracy+of+Pretrained+Language+Models 12. Rethinking Weight Tying: Pseudo-Inverse Tying for Stable LM Training and Updates — Jian Gu, Aldeida Aleti, Chunyang Chen, Hongyu Zhang, 2026 https://scholar.google.com/scholar?q=Rethinking+Weight+Tying%3A+Pseudo-Inverse+Tying+for+Stable+LM+Training+and+Updates 13. Latent Causal Probing: A Formal Perspective on Probing with Causal Models of Data — Charles Jin, 2024 https://scholar.google.com/scholar?q=Latent+Causal+Probing%3A+A+Formal+Perspective+on+Probing+with+Causal+Models+of+Data 14. AI Post Transformers: Linear Classifier Probes for Intermediate Layers — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-16-linear-classifier-probes-for-intermediat-927ae3.mp3 15. AI Post Transformers: Neural Chameleons and Evading Activation Monitors — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-14-neural-chameleons-and-evading-activation-bc470e.mp3 16. AI Post Transformers: How Induction Heads Emerge in Transformers — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-05-03-how-induction-heads-emerge-in-transforme-a7bfcb.mp3 17. AI Post Transformers: Language Models are Injective and Hence Invertible — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-03-21-language-models-are-injective-an-7545e0.mp3 18. AI Post Transformers: Latent Space as a New Computational Paradigm — Hal Turing & Dr. Ada Shannon, 2026 https://podcast.do-not-panic.com/episodes/2026-04-05-latent-space-as-a-new-computational-para-810f39.mp3 Interactive Visualization: Why Transformers Fail at Counting