Why AI Companies Keep Wasting Billions On Idle GPUs

AI companies are splashing billions on more GPUs before they use the ones they already have. Many sit idle, others use a fraction of their potential. For the hyperscalers, cloud providers and AI companies, buying more GPUs does not solve much if the GPUs already in the rack spend too much time waiting for the rest of the system.The next battle in the AI infrastructure wars is efficiency: GPU utilization, NICs, memory movement, networking, CPUs and software that keeps expensive hardware doing useful work.Moshe Tanach runs NeuReality AI, a company building infrastructure for AI inference. In this conversation, he explains why the next wave of AI infrastructure depends on GPU utilization, NICs, memory movement, networking and software efficiency, not just buying more chips.This is a technical conversation about AI inference, infrastructure efficiency and the hardware layer beneath ChatGPT, Claude, Gemini and the AI tools now being built into business software.Please enjoy the show.Thinking on Paper is a technology podcast about AI, Space, quantum computing, science, and the systems shaping your life. 🏠 ⁠Buy us a beer on Substack⁠🫵 C⁠hoose your own technology adventure ⁠📺  ⁠Watch our beautiful faces on YouTube ⁠🎧 R⁠emember Steve Jobs on APPLE⁠📺 ⁠Get clips and exclusive videos on Instagram ⁠--Chapters(00:00) The GPU challenge(01:27) Training Vs Inference(05:45) Memory & Keeping The GPU Busy(07:31) Deep Seek, Blackwell & Ruben(09:03) How Ripe Is CPU For Reinvention?(11:03) AI Agents Run On CPUs(12:51) Do We Need All The Computation?(14:35) GPU Utilization Rates(16:50) Tokens In A Context Window(20:59) Data, Knowledge & Wisdom(23:56) Why Are Your GPUs Busy? (A Truck Analogy)(29:07) Hyperscalers(31:59) The Decode Phase(36:47) What More Efficient GPUs Means For The User