Operationalizing Machine Learning at a Large Financial Institution // Daniel Stahl // MLOps Meetup #56

MLOps community meetup #56! Last Wednesday, we talked to  Daniel Stahl, Head of Data and Analytics Platforms, Regions Bank.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter// Abstract:The Data Science practice has evolved significantly at Regions, with a corresponding need to scale and operationalize machine learning models. Additionally, highly regulated industries such as finance require a heightened focus on reproducibility, documentation, and model controls.  In this session with Daniel Stahl, we will discuss how the Regions team designed and scaled their data science platform using DevOps and MLOps practices.  This has allowed Regions to meet the increased demand for machine learning while embedding controls throughout the model lifecycle.  In the 2 years since the data science platform has been onboarded, 100% of data products have been successfully operationalized.// Bio:Daniel Stahl leads the ML platform team at Regions Bank and is responsible for tooling, data engineering, and process development to make operationalizing models easy, safe, and compliant for Data Scientists.  Daniel has spent his career in financial services and has developed novel methods for computing tail risk in both credit risk and operational risk, resulting in peer-reviewed publications in the Journal of Credit Risk and the Journal of Operational Risk. Daniel has a Master's in Mathematical Finance from the University of North Carolina at Charlotte.     Daniel lives in Birmingham, Alabama, with his wife and two daughters.----------- Connect With Us ✌️-------------   Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Dan on LinkedIn: https://www.linkedin.com/in/daniel-stahl-6685a52a/Timestamps: [00:00] Introduction to Ben Wilson [00:11] Ben's background in tech [01:17] "How do you do what I have always done pretty well, which is being as lazy as possible in order to automate things that I hate doing. So I learned about Regression Problems." [03:40] Human aspect of Machine Learning in MLOps[05:51] MLOps is an organizational problem[09:27] Fragile Models[12:36] Fraud Cases[15:21] Data Monitoring[18:37] Importance of knowing what to monitor for[22:00] Monitoring for outliers[24:16] Staying out of Alert Hell[29:40] Ground Truth[31:25] Model vs Data Drift on Ground Truth Unavailability [34:25] Benefit to monitor system or business-level metrics [38:20] Experiment in the beginning, not at the end [40:30] Adaptive windowing [42:22] Bridge the gap [46:42] What scarred you really bad?