The Hardware Friction Map

TL;DR * The Hardware Friction Map asserts that the survival of neural architectures is determined by economics, as hardware imposes a “compute tax” based on how much an idea deviates from subsidized GPU primitives like dense matrix multiplications. * Architectures are classified into four zones (Green, Yellow, Orange, Red) based on the increasing engineering and compute cost required to clear Gate 1: hardware and infrastructure viability, ranging from techniques that require only retraining to those that are architecturally misaligned with current GPU stacks. * Examples of friction include Yellow Zone techniques like FlashAttention, which require custom kernels and take about two years for universal adoption, and Orange Zone techniques like Mixture-of-Experts (MoE), which necessitate a distributed system overhaul requiring 36+ months. * High friction creates an economic moat for large labs that can afford the 36-month engineering burn to rewrite cluster schedulers and kernels, but betting on high-friction architectures is often death for startups due to the required runway. Thanks for listening! Subscribe for free to receive new posts and support my work.https://lambpetros.substack.com/ This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit lambpetros.substack.com