Skip to content
From Probabilistic to Trustworthy: Building Orion, an Agentic Analytics Platform episode artwork

Oct 11, 2025 1 hr 12 min

From Probabilistic to Trustworthy: Building Orion, an Agentic Analytics Platform

from AI Engineering Podcast Hosted by Tobias Macey

Summary In this episode of the AI Engineering Podcast Lucas Thelosen and Drew Gillson talk about Orion, their agentic analytics platform that delivers proactive, push-based insights to business users through asynchronous thinking with rich organizational context. Lucas and Drew share their approach to building trustworthy analysis by grounding in semantic layers, fact tables, and quality-assurance loops, as well as their focus on accuracy through parallel test-time compute and evolving from probabilistic steps to deterministic tools. They discuss the importance of context engineering, multi-agent orchestration, and security boundaries for enterprise deployments, and share lessons learned on consistency, tool design, user change management, and the emerging role of "AI manager" as a career path. The conversation highlights the future of AI knowledge workers collaborating across organizations and tools while simplifying UIs and raising the bar on actionable, trustworthy analytics. Announcements Hello and welcome to the AI Engineering Podcast, your guide to the fast-moving world of building scalable and maintainable AI systems When ML teams try to run complex workflows through traditional orchestration tools, they hit walls. Cash App discovered this with their fraud detection models - they needed flexible compute, isolated environments, and seamless data exchange between workflows, but their existing tools couldn't deliver. That's why Cash App rely on Prefect. Now their ML workflows run on whatever infrastructure each model needs across Google Cloud, AWS, and Databricks. Custom packages stay isolated. Model outputs flow seamlessly between workflows. Companies like Whoop and 1Password also trust Prefect for their critical workflows. But Prefect didn't stop there. They just launched FastMCP - production-ready infrastructure for AI tools. You get Prefect's orchestration plus instant OAuth, serverless scaling, and blazing-fast Python execution. Deploy your AI tools once, connect to Claude, Cursor, or any MCP client. No more building auth flows or managing servers. Prefect orchestrates your ML pipeline. FastMCP handles your AI tool infrastructure. See what Prefect and Fast MCP can do for your AI workflows at aiengineeringpodcast.com/prefect today. Your host is Tobias Macey and today I'm interviewing Lucas Thelosen and Drew Gillson about their experiences building an agentic analytics platform and the challenges of ensuring accuracy to build trust Interview Introduction How did you get involved in machine learning? Can you describe what Orion is and the story behind it? Business analytics is a field that requires a high degree of accuracy and detail because of the potential for substantial impact on the business (positive and negative). These are areas that generative AI has struggled with achieving consistently. What was your process for building confidence in your ability to achieve that threshold before committing to the path you are on now? There are numerous ways that generative AI can be incorporated into the process of designing, building, and delivering analytical insights. How would you characterize the different strategies that data teams and vendors have approached that problem? What do you see as the organizational benefits of moving to a push-based model for analytics? Can you describe the system architecture of Orion? Agentic design patterns are still in the early days of being developed and proven out. Can you give a breakdown of the approach that you are using? How do you think about the responsibility boundaries, communication paths, temporal patterns, etc. across the different agents? Tool use is a key component of agentic architectures. What is your process for identifying, developing, validating, and securing the tools that you provide to your agents? What are the boundaries and extension points that you see when building agentic systems? What are the opportunities for using e.g. A2A for protocol for managing agentic hand-offs? What is your process for managing the experimentation loop for changes to your models, data, prompts, etc. as you iterate on your product? What are some of the ways that you are using the agents that power your system to identify and act on opportunities for self-improvement? What are the most interesting, innovative, or unexpected ways that you have seen Orion used? What are the most interesting, unexpected, or challenging lessons that you have learned while working on Orion? When is an agentic approach the wrong choice? What do you have planned for the future of Orion? Contact Info Lucas LinkedIn Drew LinkedIn Parting Question From your perspective, what are the biggest gaps in tooling, technology, or training for AI systems today? Closing Announcements Thank you for listening! Don't forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used. Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes. If you've learned something or tried out a project from the show then tell us about it! Email [email protected] with your story. To help other people find the show please leave a review on iTunes and tell your friends and co-workers. Links Gravity Orion Data Engineering Podcast Episode Site Reliability Engineering Anthropic Claude Sonnet 4.5 A2A (Agent2Agent) Protocol Simon Willison AI Lethal Trifecta Behavioral Science Grounded Theory LLM as a Judge RLHF == Reinforcement Learning from Human Feedback The intro and outro music is from Hitman's Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

From Probabilistic to Trustworthy: Building Orion, an Agentic Analytics Platform

0:00 1:12:19

Chapters

  1. 0:05 Welcome and guest intro: agentic analytics and trust
  2. 1:28 Backgrounds: Lukas and Drews paths into AI and data
  3. 3:42 Early ML stories: mortgages, optimization, and computer vision play
  4. 9:00 Introducing Orion: asynchronous AI analyst and accuracy mandate
  5. 13:25 From text-to-SQL to grounded agents: strategies for reliable insights
  6. 19:03 From dashboards to actions: trends, root causes, and recommendations
  7. 23:41 Push vs pull analytics: behavior change, bias, and literacy
  8. 27:04 Agentic architecture: context engineering and compounding errors
  9. 33:43 Readiness and data foundations: semantics, freshness, and expiry
  10. 39:04 Boundaries and security: agent ecosystems, governance, and risk
  11. 45:26 Observability and evolution: taxonomy, autoraters, and testing
  12. 50:33 Self-improvement loops: deterministic tools and context hygiene
  13. 54:43 Tooling design: fewer complex tools, registries, and future actions
  14. 58:00 Customer use cases: qualitative analysis at scale surprises
  15. 1:00:15 Lessons learned: consistency, scope, and capability boundaries
  16. 1:04:16 When not to use agents: latency, repeatability, and novelty
  17. 1:06:33 Beyond UIs: AI knowledge workers and direct data interaction
  18. 1:07:50 Big gaps: change management and becoming an AI manager
  19. 1:11:13 Closing thoughts and sign-off

No transcript yet.

Requests help us decide what to transcribe next.

Frequently asked questions

How long is this episode of AI Engineering Podcast?

This episode is 1 hour and 12 minutes long.

When was this AI Engineering Podcast episode published?

This episode was published on October 11, 2025.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this AI Engineering Podcast episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.