How Kubernetes StatefulSet PVC Resizing Causes Node Disk Failures
from DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations Hosted by Fexingo
Kubernetes 1.24 introduced the ability to expand PersistentVolumeClaims for StatefulSets, but the feature has a dangerous edge case. When a StatefulSet’s storage template is updated, the controller can trigger concurrent resize operations across multiple replicas, overwhelming the underlying storage backend and causing node disk failures. In this episode, Lucas and Luna walk through a real incident at a mid-sized SaaS company where a simple PVC size bump from 50GB to 100GB caused a cascading failure across three nodes. They explain the mechanics of the resize API, the race condition in generic ephemeral volumes, and the mitigation strategies—including pod-level PDBs and phased rollouts. DevOps engineers currently running StatefulSets with expanded PVCs will walk away with a concrete checklist to avoid this silent cluster-killer. #Kubernetes #StatefulSet #PVCResize #NodeDiskFailure #PersistentVolumeClaim #DevOps #CloudNative #InfrastructureFailure #RaceCondition #ETCD #Kubelet #StorageBackend #Persistence #CascadingFailure #Technology #FexingoBusiness #BusinessPodcast #ProductionIncident Keep every episode free: buymeacoffee.com/fexingo
Embed this episode
No transcript yet.