Metaflow: Netflix Machine Learning Platform with Savin Goyal

Netflix runs all of its infrastructure on Amazon Web Services. This includes business logic, data infrastructure, and machine learning. By tightly coupling itself to AWS, Netflix has been able to move faster and have strong defaults about engineering decisions. And today, AWS has such an expanse of services that it can be used as a platform to build custom tools. Metaflow is an open source machine learning platform built on top of AWS that allows engineers at Netflix to build directed acyclic graphs for training models. These DAGs get deployed to AWS as Step Functions, a serverless orchestration platform. Savin Goyal is a machine learning engineer with Netflix, and he joins the show to talk about the machine learning challenges within Netflix, and his experience working on Metaflow. We also talk about DAG systems such as AWS Step Functions and Airflow.