How Stream Built an AI Support Agent with One Part-Time Engineer | Nick Rogers

Stream's AI support agent resolves 80% of their customer tickets. It was built by one engineer, working part-time, inside a 60-person product and engineering team.In hundreds of conversations with CX leaders evaluating self-build, this is the exception. Most attempts stall between an impressive hackathon demo and a system reliable enough for production. Nick Rogers, Chief Product and Technology Officer at stream (formerly Wagestream), explains why his team is one of the few that crossed that gap.Nick walks through the test he'd give any leader weighing build versus buy (his answer: if you need to ask, partner), the "wait for the frontier" tempo of trying, failing, and coming back when a new model release shifts what's possible, and the results of stream's recent model bake-off, including the model that surprised the team for being worse than expected.Follow The Squawk podcast for hot takes and big ideas in CX. Perfect for a short commute and guaranteed to leave you with something new to think about, or implement, for your CX operation.Timestamps:00:00 - Cold open00:41 - Intro to Nick & stream01:55 - Nick's career path04:00 - Why stream's self-build is the exception05:07 - The chronological story: GPT-3.5 to Gemini 2.0 Flash07:50 - The culture that lets stream try and fail09:15 - Eval methodology: golden datasets and A/B testing in production11:30 - 70% to 80% deflection, a 33% reduction in tickets11:55 - The model bake-off and the Claude Opus surprise13:20 - Latency versus quality tradeoffs14:46 - "How many of your 60 engineers work on this?" - effectively one16:18 - Why talent matters (Joel on Software's "high notes")17:39 - The self-build vs buy test: "if you need to ask, just buy"19:30 - The "wait for the frontier" strategy20:58 - Risk: from wild hallucination to agentic actions24:41 - Knowledge management as the new CX job26:42 - Lightning round28:51 - Recommended reading: Hackers and Painters by Paul Graham29:48 - Wrap-upWhere to find Nick:LinkedIn: https://www.linkedin.com/in/rogersnm/