EPISODE · Oct 4, 2023 · 1H 2M
158: The Orchestration Layer as the Data Platform Control Plane With Nick Schrock of Dagster Labs
from The Data Stack Show · host Rudderstack
Highlights from this week’s conversation include:Nick’s background and journey in data (2:28)Founding Dagster Labs (7:50)The evolution of data engineering (12:32)Fragmentation in data infrastructure (15:04)The role of orchestration in data platforms (19:53)The importance of operational tools for data pipelines (25:01)Lessons learned from working with GraphQL (26:19)The role of the orchestrator in data engineering (34:51)The boundaries between data infrastructure and product engineering (37:33)Different orchestrators in the data infrastructure landscape(42:03)The role of MLOps in data engineering (46:04)Data Quality and Orchestration (51:04)Future of Data Teams and Orchestration (54:27)Final thoughts and takeaways from (58:01)The Data Stack Show is a weekly podcast powered by RudderStack, the CDP for developers. Each week we’ll talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data.RudderStack helps businesses make the most out of their customer data while ensuring data privacy and security. To learn more about RudderStack visit rudderstack.com. Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
Embed this episode
What this episode covers
This week on The Data Stack Show, Eric and Kostas chat with Nick Schrock, Founder of Dagster Labs. During the episode, Nick discusses his background at Facebook and his involvement in successful open-source technologies like React and GraphQL. Nick explains the mismatch between the complexity and available tools in the data infrastructure space, leading him to start Dagster Labs. The group also talks about the challenges and fragmentation in the data engineering industry, the need for better abstraction layers, the role of orchestrators, comparing it to GraphQL's role in product engineering, the importance of data orchestration in the future of data infrastructure and engineering, and more.
NOW PLAYING
158: The Orchestration Layer as the Data Platform Control Plane With Nick Schrock of Dagster Labs
No transcript for this episode yet
Similar Episodes
Similar Podcasts
No similar podcasts found.