The Godfather Of MLOps // D. Sculley // MLOps Coffee Sessions #32

Coffee Sessions #32 with D. Sculley of Google, The Godfather Of MLOps.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠//BioD is currently a director in Google Brain, leading research teams working on robust, responsible, reliable, and efficient ML and AI. In his time at Google, D worked on nearly every aspect of machine learning and has led both product and research teams, including those on some of the most challenging business problems.// Links to D. Sculley's PapersML Test Score: https://research.google/pubs/pub46555/Machine Learning: The high-interest credit card of technical debthttps://research.google/pubs/pub43146/Google Scholar:https://scholar.google.com/citations?user=l_O64B8AAAAJ&hl=en--------------- ✌️Connect With Us ✌️ -------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: https://www.linkedin.com/in/aponteanalytics/Connect with Vishnu on LinkedIn: https://www.linkedin.com/in/vrachakonda/Connect with D. Sculley on LinkedIn: https://www.linkedin.com/in/d-sculley-90467310/Timestamps:[00:00] Introduction to D. Sculley[00:40] The Biggest Papers were written by D for Machine Learning[02:08] What's changed since you wrote those papers?[02:56] "No 1, there is an MLOps community."[04:38] Old best practices[05:12] "The fact that there are jobs titled MLOps, this is different than it was 5 or 6 years ago."[06:30] Machine Learning Systems then and now[07:08] "There wasn't the level of general infrastructure that was looking to offer the large-scale integrated solutions."   [07:57] ML Test Score[11:09] "The Test Score was really written for situations where you don't care about one prediction. You care about millions or billions of predictions per day."[12:27] "In the end, it's not about the score. It's about the process of asking the questions, making sure that each of the important questions that you're asking yourself, you have a good answer to."  [13:04] What else is needed in the Test Score?[14:36] Stratified testing  [17:05] Counterfactual testing[18:34] Boundaries[19:15] Dark ages[20:27] How do you try in Triage?[21:10] "Reliability is important. There are no small mistakes. If there are errors, they're going to get spotted and publicized. They're going to impact users' lives. The bar is really high, and it's worth the effort to ensure strong reliability."[23:11] How do you build that interest stress test?[24:39] "I believe that stress test is going to look like a useful way to encode expert knowledge about domain areas."[25:37] How do I bring robustness?[27:22] Underspecification Paper[30:58] "It's important to be evaluating models on this auto domain stress test and make sure that we understand the implications of what we're thinking about while we are in deployment land."[32:27] Principal challenges in productionizing Machine Learning[34:57] "As we expose our models to more specifics, this means there are more potential places our models might be exhibiting unexpected or undesirable behaviour."[42:37] Splintering of ML Engineering[46:00] Communities shaping the MLOps sphere[46:42] "It's much better to have one large community than three smaller communities because of those edufacts."[47:47] Concept of technical debt in machine learning.[49:28] "The good idea is to tend to make their way into the community if they are in a form that people can digest and share."