Hands-on serving models using KFserving // Theofilos Papapanagiotou // Data Science Architect at Prosus // MLOps Meetup #40

MLOps community meetup #40! Last Wednesday, we talked to Theofilos Papapanagiotou, Data Science Architect at Prosus, about Hands-on Serving Models Using KFserving.Join the Community: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTJoinIn⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠Get the newsletter: ⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠⁠https://go.mlops.community/YTNewsletter⁠⁠⁠// Abstract:We looked at some popular model formats like the SavedModel of Tensorflow, the Model Archiver of PyTorch, pickle&ONNX, to understand how the weights of the NN are saved there, the graph, and the signature concepts.We discussed the relevant resources of the deployment stack of Istio (the Ingress gateway, the sidecar, and the virtual service) and Knative (the service and revisions), as well as Kubeflow and KFServing. Then we got into the design details of KFServing, its custom resources, the controller and webhooks, the logging, and configuration.We spent a large part in the monitoring stack, the metrics of the servable (memory footprint, latency, number of requests), as well as the model metrics like the graph, init/restore latencies, the optimizations, and the runtime metrics, which end up in Prometheus. We looked at the inference payload and prediction logging to observe drifts and trigger the retraining of the pipeline.Finally, a few words about the awesome community and the roadmap of the project on multi-model serving and inference routing graph.// Bio:Theo is a recovering Unix Engineer with 20 years of work experience in Telcos, on internet services, video delivery, and cybersecurity. He is also a university student for life; BSc in CS 1999, MSc in Data Coms 2008, and MSc in AI 2017.Nowadays, he calls himself an ML Engineer, as he expresses his passion for System Engineering and Machine Learning.His analytical thinking is driven by curiosity and a hacker spirit. He has skills that span a variety of different areas: Statistics, Programming, Databases, Distributed Systems, and Visualization.----------- Connect With Us ✌️-------------  Join our Slack community:  https://go.mlops.community/slackFollow us on Twitter:  @mlopscommunitySign up for the next meetup:  https://go.mlops.community/registerConnect with Demetrios on LinkedIn:  https://www.linkedin.com/in/dpbrinkm/Connect with Theofilos on LinkedIn:  https://linkedin.com/in/theofpa