How Kubernetes Node Journal Rotation Hides Crash Loops
from DevOps Daily with Fexingo: CI/CD, Kubernetes, and Modern Software Operations Hosted by Fexingo
In this episode of DevOps Daily, Lucas and Luna dig into a silent killer in Kubernetes clusters: systemd journal rotation. When the kubelet's logs are rotated out before a crash is captured, debugging becomes guesswork. They walk through the mechanics of journald's size and time-based rotation, the systemd-journald daemon's default rate limits, and how a busy cluster can lose critical kubelet logs in seconds. They share a real-world scenario where a node crash loop went unnoticed for hours because the logs were gone. They also discuss practical fixes: setting SystemMaxUse and RuntimeMaxUse, adjusting RateLimitIntervalSec, forwarding logs to a central aggregator, and using node problem detector to catch issues early. By the end, you'll know exactly how to ensure your kubelet logs survive the next incident. #Kubernetes #DevOps #Systemd #Journald #LogRotation #Kubelet #CrashLoop #Troubleshooting #SiteReliability #CloudNative #Observability #Logging #NodeProblemDetector #IncidentResponse #TechPodcast #Business #FexingoBusiness #Podcast Keep every episode free: buymeacoffee.com/fexingo
Embed this episode
No transcript yet.