The Blackbox: Your AI Product Is Quietly Failing (ft. Julien Roux, CRO at Nebuly)

Most AI teams are flying blind - they can show you how many people used their AI, but not whether it actually worked.In this conversation, host Johannes Deubener sits down with Julien Roux, CRO and co-founder of Nebuly (the “Google Analytics for AI”) to unpack what relevant AI Analytics look like.They explore why treating AI like traditional software (“build it, ship it, forget it”) is a fundamental mistake, and why AI needs to be managed as a living product: launched, observed, and continuously improved based on how people actually use it. Julien explains how most companies still rely on shallow metrics - logins, number of prompts, session duration - and why those can be dangerously misleading. In conversational AI, more usage can mean more frustration, not more value.Chapters00:05 Meet Julien Roux02:23 From Mechanical Engineering to Finance05:26 What “AI-Powered" Toothbrushes Mean08:11 AI "Launch-and-Learn" Explained10:50 From Investor to AI Founder: Why Julien Bet on AI Analytics13:03 Executives’ Fantasy vs. Employee Reality in AI16:57 Rocky Waters: Careers, Juniors, and the Future of Work21:18 Pareto in Practice: Focus on the 10–15% of AI Use Cases That Drive 80% of Value24:00 Why Traditional Analytics Fail for AI Assistants29:21 “Google Analytics for AI”: What Nebuly Actually Measures32:11 Explicit vs. Implicit Feedback in AI Conversations36:18 100x More Actionable Feedback: The Iveco Case Study38:50 Fixing AI Quality: Knowledge Bases, RAG, and the 80% Non-Technical Problems41:16 Mining Business Signals from Customer-Facing AI (Churn, Upsell, Demand)47:16 From Black Box to ROI: Turning Time Saved into Dollars51:43 What CFOs Will Ask AI Teams in 202656:00 The Future of AI Analytics, Skills, and Infrastructure--------------------------------------------------------------Don't be shy and come say hi:Connect with Johannes: https://www.linkedin.com/in/johannesdeubener/Follow Julien: https://www.linkedin.com/in/julienroux92/Check out Nebuly: https://www.nebuly.com/--------------------------------------------------------------More about AI To Go:✨ Watch more Episodes on YouTube: https://www.youtube.com/channel/UCiL3_cC9nmXALQiaqlYs2Kw✨ Listen on Apple Podcasts: https://podcasts.apple.com/us/podcast/ai-to-go-podcast/id1866997458✨ Listen on Spotify: https://open.spotify.com/episode/5X9XgIv1qtCYtExziNMyXK✨ Receive all Updates on LinkedIn: https://www.linkedin.com/company/ai-to-go-podcast/