The Data Splash

Lineage isn't just for compliance anymore- it's how AI agents understand your data. Harel Shein (Datadog, OpenLineage TSC) explains why.

Atlassian's Head of Data Engineering & AI Enablement, Prakash Reddy, joins host Ido Bronstein, Upriver's Co-founder and CEO, for an honest conversation about using AI to automate data engineering work at scale.Six months in, Atlassian is shipping real results, 5-day tickets in under 3 days, an on-call agent that triages production failures, and a clear roadmap for AI-ready data. But the most useful part of this episode is what had to be true before any of it worked: a multi-year migration to declarative YAML pipelines, environment isolation, and a clean medallion architecture.In this episode: • Why AI doesn't work without foundational data architecture • The 3 pillars Atlassian picked for AI ROI (and what they skipped) • How to measure productivity gains in total cost of ownership, not velocity • Why hallucinations (3-4 out of 10) are a workflow problem, not a model problem • The "coalition of the willing" approach, bottom-up experiments + top-down consolidation • A prediction on role convergence: knowledge engineer, context engineer, agent orchestrator • Why the moat stops being SQL, and what replaces itWhether you're a data engineer trying to make sense of where the role is headed, a data leader planning an AI rollout, or just curious how a company at Atlassian's scale is approaching this, this episode is built for you.⏱ CHAPTERS 00:00 Intro & 30-Second Splash 02:00 How Atlassian's data org is structured 05:00 Why automate data engineering with AI? 08:00 The foundation that made AI possible 10:00 The 3 pillars: incremental dev, on-call, AI-ready data 13:00 Real productivity numbers (and how to measure them honestly) 17:00 Hallucinations, guardrails, and what actually breaks 21:00 Org design: bottom-up + top-down 25:00 The future of data roles — convergence is coming 30:00 Closing thoughts🎙 ABOUT DATA SPLASH Data Splash is a podcast for data engineers, data leaders, and anyone trying to make sense of AI and data right now. Brought to you by Upriver.🔔 Subscribe for new episodes weekly.🔗 LINKS • Upriver: [https://www.upriverdata.com/] • Connect with Prakash Reddy: [https://www.linkedin.com/in/prakashreddy1357/] • Connect with Ido Bronstein: [https://www.linkedin.com/in/ido-bronstein/]#DataEngineering #AI #Atlassian #DataPlatform #LLMs #AIAgents