MLOps Coffee Sessions #12: Journey of Flyte at Lyft and Through Open-source // Ketan Umare

Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠Why was Flyte built at Lyft?What sorts of requirements does an ML infrastructure team have at Lyft?What problems does it solve/use cases?Where does it fit in the ML and Data ecosystem?What is the vision?Who should consider using it?Learnings as the engineering team tried to bootstrap an open-source community.Ketan Umare is a senior staff software engineer at Lyft, responsible for the technical direction of the Machine Learning Platform, and is a founder of the Flyte project. Before Flyte, he worked on ETA, routing, and mapping infrastructure at Lyft. He is also the founder of Flink Kubernetes operator and a contributor to Spark on Kubernetes. Prior to Lyft, he was a founding member of Oracle Baremetal Cloud and led teams building Elastic Block Storage. Prior to that, he started and led multiple teams in Mapping and Transportation optimization infrastructure at Amazon. He received his Master's in Computer Science from Georgia Tech, specializing in High-performance computing, and his Bachelor's in Engineering in Computer Science from VJTI Mumbai.Besides work, he enjoys spending time with his daughter and wife. He loves the Pacific Northwest outdoors and will try anything new.LyftPricing, Locations, Estimated Time of Arrivals (ETA), Mapping, Self-Driving (L5), etc.What sort of scale, storage, and network bandwidth are we looking at?Tens of thousands of workflows, hundreds of thousands of executions, millions of tasks, and tens of millions of containers!Flyte: more than 900k workflows executed a month and more than 30+ million container executions per monthTypical flow of information?What are the user stories you’re typically dealing with at Lyft?How do you set it up?On-prem, cloud, etc.Helm installable?Why Golang?What problems does it solve?Complex data dependencies? WhyOrchestrated compute on demandReuse and sharingKey featuresMulti-tenant, hosted, serverlessParametrized, data lineage, and cachingAdditionally, if the run invokes a task that has already been computed before, regardless of who executed it, Flyte will smartly use the cached output, saving you both time and money.Versioning, sharingModular, loosely coupledSeems like you guys recognize that the best task for the job might be hosted elsewhere, so it was important to integrate other solutions into Flyte.Flyte extensionsBackend plugins - is it true you can create and manage k8s resources like CRDs for things like Spark, Sagemaker, BigQuery?Drop a Starhttps://flyte.orgFlyte community----------- Connect With Us ✌️-------------Join our Slack community: https://go.mlops.community/slackFollow us on Twitter: @mlopscommunitySign up for the next meetup: https://go.mlops.community/registerConnect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/Connect with David on LinkedIn: ⁠https://www.linkedin.com/in/aponteanalytics/⁠Connect with Ketan on LinkedIn: https://www.linkedin.com/in/ketanumare/