OpenLineage, AI Agents, and the Future of Data Engineering - with Harel Shein, Datadog

Everyone says they need data lineage. Few can say why. Harel Shein, Senior Engineering Manager at Datadog and an OpenLineage steering committee member, makes the case that AI turns lineage from a nice-to-have graph into infrastructure. We cover what OpenLineage actually is, why agents querying your warehouse need it to understand what data means, the maintainer tax of vibe-coded PRs, and Harel's wishlist: push the standard into every engine like Spark, and extend it to ML. One spec to rule them all.⏱ CHAPTERS 0:00 Intro — What we're getting into 1:05 Rapid-fire questions 2:18 What OpenLineage actually is 3:22 Why Apache Airflow built it in (and the Linux Foundation) 5:10 How AI is reshaping open source maintenance 6:10 Vibe-coded PRs, maintainer burden, and the AGENTS.md fix 7:37 Why coding agents understand open source better than closed source 8:35 Real-world use cases: ops, data quality, compliance, cost 11:38 How AI changes the lineage use case — amplification 13:59 Lineage as the foundation for agents 16:05 MCP, self-serve data, and the trust problem 19:37 Lineage for unstructured data — the hard problem 22:03 If you had an army of engineers, what would you build? 24:39 Where data engineering is heading 26:29 Takeaways and one spec to rule them all🎙 ABOUT DATA SPLASH: Data Splash is a podcast for data engineers, data leaders, and anyone trying to make sense of AI and data right now. Brought to you by Upriver.🔔 Subscribe for new episodes weekly.🔗 LINKS • Upriver: https://www.upriverdata.com/ • Connect with Harel Shein: https://www.linkedin.com/in/harelshein/ • Connect with Ido Bronstein: https://www.linkedin.com/in/ido-bronstein/#DataEngineering #AI #Datadog #LLMs #AIAgents