Generative AI on Kubernetes

In this episode of the Kubernetes Bytes podcast, Ryan and Bhavin sit down with Janakiram MSV - an advisor, analyst and architect to talk about how users can run Generative AI models on Kubernetes. The discussion revolves around Jani's home lab and his experimentation with different LLM models and how to get them running on NVIDIA GPUs. Jani has spent the past year becoming a subject matter expert in GenAI, and this discussion highlights all the different challenges he faced and what lessons he learnt from them.   Check out our website at https://kubernetesbytes.com/  Episode Sponsor: Elotl   https://elotl.co/luna https://www.elotl.co/luna-free-trial  Timestamps:  02:02 Cloud Native News  15:31 Interview with Jani  01:11:00 Key takeaways  Cloud Native News:  https://www.techerati.com/press-release/octopus-deploy-acquires-codefresh-to-boost-kubernetes-and-cloud-native-delivery/ https://www.civo.com/blog/kubefirst-joins-civo   https://cast.ai/kubernetes-cost-benchmark  https://www.techradar.com/pro/vmware-customers-are-jumping-ship-as-broadcom-sales-continue-heres-where-theyre-moving-to  https://cloudonair.withgoogle.com/events/techbyte-making-ai-ml-scalable-cost-effective-gke https://dok.community/dok-events/dok-day-kubecon-paris/  https://training.linuxfoundation.org/certification/certified-argo-project-associate-capa   Show Links:  https://www.youtube.com/janakirammsv  https://www.linkedin.com/in/janakiramm/ - NVIDIA Container Toolkit - https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/index.html NVIDIA Device Plugin - https://github.com/NVIDIA/k8s-device-plugin NVIDIA Feature Discovery - https://github.com/NVIDIA/gpu-feature-discovery Hugging Face Text Gen Inference - https://huggingface.co/docs/text-generation-inference/index Hugging Face Text Embeddings Inference - https://huggingface.co/docs/text-embeddings-inference/index ChromaDB - https://www.trychroma.com/