AI Observability: What Could Possibly Go Wrong?

Join us as John Mark Troyer and Rakesh Gupta break down what AI observability actually means once agents leave the demo and hit production - and why the old playbook for monitoring doesn't cut it anymore. John Mark and Rakesh walk through why errors and latency are just the starting point for agents, how quality became a much harder thing to measure once bots went from answering questions to taking autonomous action, and why token-based costs are creating a confusing new economics problem for engineering teams. You'll learn the difference between online and offline evals, why a new engineering role has emerged just to build testing harnesses for agents, how trace data works differently when every prompt is its own trace, and what teams are doing to catch prompt injection and other AI-specific failure modes before they become expensive mistakes. Timestamps 0:00 Welcome & Introduction 3:20 Full Disclosure - Observe, Snowflake, and How This Conversation Started 7:07 From Developer Concerns to Boss's Boss's Boss - Spending Out of Control 8:29 What Actually Gets Measured - Errors, Latency, Quality, and Cost 10:30 The Casino Chip Problem - Confusing Token Pricing Models 13:47 Defining Quality When the Task Itself Is Nebulous 18:41 The New Role - Engineers Who Just Build Testing Harnesses 22:00 Non-Determinism and Why Testing Agents Is Expensive 32:10 Trace Data, Tool Calls, and What Observability Tools Actually See 55:08 Prompt Injection, Zero-Width Characters, and Real World Failures How to find John Mark: https://www.linkedin.com/in/johnmarktroyer/ How to find Rakesh: https://www.linkedin.com/in/rg0/ Links from the show: