A Missing Link in the ML Infrastructure Stack // Josh Tobin // MLOps Meetup #57

MLOps community meetup #57! Last Wednesday, we talked to Josh Tobin, Founder, Stealth-Stage Startup.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠// Abstract:Machine learning is quickly becoming a product engineering discipline. Although several new categories of infrastructure and tools have emerged to help teams turn their models into production systems, doing so is still extremely challenging for most companies. In this talk, we survey the tooling landscape and point out several parts of the machine learning lifecycle that are still underserved. We propose a new category of tool that could help alleviate these challenges and connect the fragmented production ML tooling ecosystem. We conclude by discussing similarities and differences between our proposed system and those of a few top companies.// Bio:Josh Tobin is the founder and CEO of a stealth machine learning startup. Previously, Josh worked as a deep learning & robotics researcher at OpenAI and as a management consultant at McKinsey. He is also the creator of Full Stack Deep Learning (fullstackdeeplearning.com), the first course focused on the emerging engineering discipline of production machine learning. Josh did his PhD in Computer Science at UC Berkeley, advised by Pieter Abbeel.// Related Linkshttps://josh-tobin.comcourse.fullstackdeeplearning.com----------- Connect With Us ✌️-------------   Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Josh on LinkedIn: https://www.linkedin.com/in/josh-tobin-4b3b10a9/Timestamps:[00:00] Introduction to Josh Tobin[01:18] Background of Josh in tech[08:27] We're you guys behind the Rubik's Cube?[09:26] Rubik's Cube Project[09:51] "Research is meant to show you what's possible to solve."[11:07] "That's one of the things that's started to change, and I think the MLOps world is maybe a part of that. What I'm excited about this is that people are focusing on the impact of their models."[13:18] Insights on Testing[17:11] Evaluation Store[18:33] "Production Machine Learning is data-driven products that have predictions in the loop."[23:40] Analyzing and moving forward[24:02] "My medium-term mindset on how machine learning is created is that there's still gonna be humans involved, but humans will be more efficient with tools."[25:50] Is there a market for this?[27:40] "The long tale of machine learning use cases is becoming part of every product and service, more or less, the companies create, but it's the same way the software part of the products and services the companies create these days. It's going to create an enormous amount of value."[30:09] Talents[32:52] Organizational by-ends and knowledge[35:16] Tools used for Evaluation Store39:59] Difference from Monitoring Tool[42:10] Who is the right person to interact with in the Evaluation Store?[50:05] Technical challenges of Apple and Tesla[53:30] "As Machine Learning use cases are getting more and more complicated, higher and higher dimensional data, bigger and bigger models, larger training sets, many companies would need in order to continually improve their systems over time."