How Kubernetes MutatingAdmissionWebhook Timeouts Cause Cluster-Wide Latency
from DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations Hosted by Fexingo
Lucas and Luna dive into a surprisingly common Kubernetes failure: MutatingAdmissionWebhooks that silently time out under load, bringing cluster-wide latency to a crawl. They walk through a real incident at a mid-size fintech company where a single webhook — a simple config-injection sidecar — started timing out at around 60 requests per second, causing every API call to pile up. They explain why Kubernetes doesn't enforce a hard timeout by default, how the API server queues webhook responses, and why your observability stack likely won't catch this unless you're monitoring API server request duration at the verb level. They also cover the fix: setting a proper failurePolicy and timeoutSeconds, plus a hint about using webhook-side caching. If you run any admission webhook in production, this one hits close to home. #Kubernetes #MutatingAdmissionWebhook #ClusterLatency #SiteReliabilityEngineering #K8sIncidents #AdmissionController #APIServerPerformance #CloudNativeInfrastructure #DevOps #Technology #FexingoBusiness #BusinessPodcast #Fintech #KubernetesTroubleshooting #WebhookTimeout #K8sFailureModes #ProductionOutage #KubernetesBestPractices Keep every episode free: buymeacoffee.com/fexingo
Embed this episode
No transcript yet.