AI Tokens Problem - Solvable!

What this episode covers

Didi and Lital open with casual sports talk (NBA, NHL, World Cup traffic) before returning to a cliffhanger about soaring AI token costs. They cite Marc Benioff saying Salesforce will pay Anthropic $300M in tokens for coding, and discuss Microsoft pausing Anthropic use after exhausting its AI budget. They argue costs are driven by energy/physics and by developers running many parallel agents, making outcome measurement crucial so token spend aligns with valuable work rather than performative usage. They describe organizational resistance ("Luddites") and poor prompting/problem decomposition ("one-liners"), stressing the need for core computer science thinking. They predict new solutions: on-prem/local models, company-built data centers, and software “broker” layers to route tasks to the right models and enforce policy. They describe building such a layer for security SOC triage to reduce alerts to ~2% needing humans, and discuss tradeoffs between general models for innovation and cheaper specialized models for repetitive tasks. They close noting rapid startup exits in this space.   Topics 00:24 Sports Rants 02:43 World Cup Traffic 03:43 AI Token Shock 07:59 Measuring AI Productivity 08:45 Luddites And One Liners 13:17 Family Travel Chaos 15:02 Token Optimization Layers 19:14 Generalists Vs Specialists 23:45 Efficiency War Stories 26:57 Future AI Cost Tools 28:55 Startup Exits Teaser 29:43 Wrap Up And Outro

PodParley-generated summary based on available episode metadata and transcript content.