Tenstorrent AI Inference Architecture: Deep Dive into Tensix Dataflow

The provided research report analyzes Tenstorrent’s AI inference architecture, a design that prioritizes a software-managed interconnect over traditional deep cache hierarchies. Led by Jim Keller, the company utilizes a MIMD architecture composed of hundreds of independent Tensix tiles, each featuring five RISC-V "baby" cores that orchestrate fixed-function math engines. Unlike GPUs that rely on expensive HBM, Tenstorrent chips use distributed on-chip SRAM and more affordable GDDR6 memory to achieve superior cost-per-token efficiency for large-scale models. The technology is built on an Ethernet-native fabric, allowing seamless scale-out across multiple chips without requiring dedicated switch silicon. While the architecture excels in compute-bound prefill tasks and long-context regimes, it faces significant bottlenecks in single-user decode latency due to lower memory bandwidth compared to high-end hardware. Furthermore, independent reviews suggest that current software limitations often leave roughly half of the silicon’s physical cores idle, representing a primary execution risk.