Emergent Symbolic Mechanisms for Reasoning in Large Language Models

This paper investigates the emergent reasoning capabilities of large language models (LLMs). Through a detailed study of the open-source LLM Llama3-70B, the authors uncover evidence for an emergent three-stage symbolic architecture that supports abstract rule induction. This architecture involves symbol abstraction, symbolic induction, and retrieval mechanisms implemented by specific attention heads within the model. The findings suggest that LLMs may achieve abstract reasoning not merely through statistical approximation, but by developing internal mechanisms akin to symbol processing, potentially bridging the gap between neural and symbolic AI approaches.