How 24,000 companies keep their AI from Breaking in Production | Rohit Agarwal, Portkey

Over 1 Trillion AI tokens pass through Portkey every single day.Every AI product eventually runs into the same problem. The prototype works, but once it goes live the system has to manage multiple models, rising token costs, unpredictable latency, and infrastructure that was never built for AI workloads.That is the problem Rohit Agarwal is solving with Portkey, an AI gateway that sits between applications and the models, whether that’s GPT-4, Claude, or Gemini.With 24,000 companies routing their AI through Portkey, Rohit sits on ground-level data on how AI is actually being used in production. Which models enterprises are betting on. Where costs are quietly climbing. How usage patterns shift as companies move from pilots to real products.When AI spend surpasses cloud spend, and Rohit believes it will, the infrastructure running underneath it becomes one of the most important bets in tech. This episode explores what it takes to run AI systems at that scale.00:00 – Trailer01:05 – 500 billion AI tokens every day04:05 – First to call an "AI gateway"07:26 – Where did the Gateway insight come from?12:08 – How Portkey is winning this space13:05 – Picking the right gambles over wrong ones14:16 – What are LLM endpoints?15:21 – AI will 100% surpass cloud spend19:00 – Hype is coming from people still in Q&A mode19:33 – AI employees over humans in customer support?23:00 – For AI startups, traffic > revenue24:43 – The bubble is in valuations, not utility26:05 – How Rohit built his personal automations28:38 – Costliest model is most used now33:21 – What's going right and wrong for AI companies37:49 – Hiring a VP of sales after $15M is possible today39:57 – What edge does Claude have over other models?43:35 – Founders need a "why me vs. why Anthropic" story52:56 – What if Anthropic or AWS builds a gateway?55:41 – Predictions for the next 12 months59:40 – How big is the opportunity in Agents?01:00:48 – Startups now have to prove it's not a weekend project01:01:54 – Is Build v/s Buy no longer a Debate?01:03:50 – What would Rohit build if starting up today?01:05:58 – How Portkey is different from an API gateway01:08:26 – MCP / tool calling enables agentic workflows01:12:00 – Portkey's Community-driven early GTM01:13:34 – Startups have only 2 reasons for Open core-------------India’s talent has built the world’s tech—now it’s time to lead it.This mission goes beyond startups. It’s about shifting the center of gravity in global tech to include the brilliance rising from India.What is Neon Fund?We invest in seed and early-stage founders from India and the diaspora building world-class Enterprise AI companies. We bring capital, conviction, and a community that’s done it before.Subscribe for real founder stories, investor perspectives, economist breakdowns, and a behind-the-scenes look at how we’re doing it all at Neon.-------------Check us out on:Website: https://neon.fund/Instagram: https://www.instagram.com/theneonshoww/LinkedIn: https://www.linkedin.com/company/beneon/Twitter: https://x.com/TheNeonShowwConnect with Siddhartha on:LinkedIn: https://www.linkedin.com/in/siddharthaahluwalia/Twitter: https://x.com/siddharthaa7-------------This video is for informational purposes only. The views expressed are those of the individuals quoted and do not constitute professional advice.Send us Fan Mail