Airflow Sucks for MLOps // Stephen Bailey // MLOps Podcast #141

MLOps Coffee Sessions #141 with Stephen Bailey, Airflow Sucks for MLOps, co-hosted by Joe Reis.// AbstractStephen discusses his experience working with data platforms, particularly the challenges of training and sharing knowledge among different stakeholders. This talk highlights the importance of having clear priorities and a sense of practicality and mentions the use of modular job design and data classification to make it easier for end users to understand which data to use.   Stephen also mentions the importance of being able to move quickly and not getting bogged down in the quest for perfection. We recommend Stephen's blog post "Airflow's Problem" for further reading.// BioStephen has worked as a data scientist, analyst, manager, and engineer, and loves all the domains equally. He currently works at Whatnot, a collectibles marketplace that focuses on live shopping, and has previously worked in privacy tech at Immuta. He has his Ph.D. from Vanderbilt University in educational cognitive neuroscience, but it has yet to help him understand why his three children are so crazy.// MLOps Jobs board jobs.mlops.community // MLOps Swag/Merchhttps://mlops-community.myshopify.com/// Related LinksAirflow's Problem blog post: https://stkbailey.substack.com/p/airflows-problemAirflow's Problem and the reception it got on Hacker News: https://news.ycombinator.com/item?id=32317558--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletters, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Joe on LinkedIn: https://www.linkedin.com/in/josephreis/Connect with Stephen on LinkedIn: https://www.linkedin.com/in/stkbailey/Timestamps: [00:00] Stephen's preferred coffee [00:19] Introduction to co-host Joe Reis [01:40] Takeaways [06:29] Subscribe to our newsletters! [06:55] Shout out to our sponsor, Wallaroo! [08:05] Whatnot [10:47] Stephen's side hustle [14:35] Stephen's work breakdown at Whatnot [18:03] Fundamental tensions in the data world [21:27] Initial questions to answer that you were on the right path [24:06] Recommender systems [28:15] Coordinating with ML teams [29:43] Daxter [31:38] Too advanced, more challenging [34:37] Orchestration layer [36:14] Decision criteria [39:23] Human design aspect of Daxter [40:53] Orchestration layer centralization and sharing knowledge with stakeholders [46:18] Airflow's Problem and the reception it got on Hacker News [51:00] Wrap up