Calibration for ML at Etsy - apply() special // Erica Greene and Seoyoon Park // MLOps Coffee Sessions #78

MLOps Coffee Sessions #78 with Erica Greene and Seoyoon Park, Calibration for ML at Etsy - apply() special.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠// Abstract This is a special conversation about Machine Learning calibration at Etsy. Demetrios sat down with Erica Greene and Seoyoon Park to hear about how they implemented Calibration into the Etsy Machine Learning workflow.The conversation is a pre-chat with these two before their presentation at the apply() conference on February 10th. Register here: applyconf.com// Bio Erica Geen Erica is an engineering manager with a background in machine learning. She's passionate about developing programs and policies that support women and other underrepresented groups in technology.Seoyoon Park Backend software engineer and aspiring software architect interested in producing scalable, performant, and fault-tolerant applications by keeping up to date with best practices and industry standards. Seoyoon strives to better himself and his peers by advocating for frequent knowledge transfers and promoting a culture of continuous learning. Constantly looking for opportunities to grow as a developer and become a leader of the industry.--------------- ✌️Connect With Us ✌️ ------------- Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/registerCatch all episodes, blogs, newsletter, and more: https://mlops.community/Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with Erica on LinkedIn: https://www.linkedin.com/in/ericagreene/Connect with Seoyoon on LinkedIn: https://www.linkedin.com/in/seoyoonpark/Timestamps:[00:00] Quick notes[00:47] Introduction to Erica Greene and Seoyoon Park[02:37] Looking for an editor[04:23] Seoyoon Park's background[06:01] Erica Greene's background[08:07] Seoyoon's transition to ML[10:25] Erica's take as team manager[11:58] Additional points from Seoyoon[13:17] Early wins in ML[15:41] Seoyoon's start in ML[17:54] Three core things for Erica[19:07] What is calibration?[22:39] Calibration uses[24:46] Shift in different models[26:11] On Calibration of Modern Neural Networks[26:52] Importance of Calibration to models[28:31] Implementation of Calibrating[31:24] Calibration metrics[35:07] Calibration testing[36:50] The bug encounter[39:09] Debugging the fault[43:16] Erica's war story[46:01] Seoyoon's war story[48:31] Wrap up