The Data Engineer's guide to optimizing Kubernetes, with Niels Claeys

Niels Claeys shares how his team at Dataminded built Conveyor, a data platform processing up to 1.5 million core hours monthly. He explains the specific optimizations they discovered through production experience, from scheduler changes that immediately reduce costs by 10-15% to achieving 97% spot instance usage without reliability issues.You will learn:Why the default Kubernetes scheduler wastes money on batch workloads and how switching from "least allocated" to "most allocated" scheduling enables faster scale-down and better resource utilizationHow to achieve 97% spot instance adoption through strategic instance type diversification, region selection, and Spark-specific techniquesNode pool design principles that balance Kubernetes overhead with workload efficiencyPlatform-specific gotchas like AWS cross-AZ data transfer costs that can spike bills unexpectedlySponsorThis episode is brought to you by Testkube—where teams run millions of performance tests in real Kubernetes infrastructure. From air-gapped environments to massive scale deployments, orchestrate every testing tool in one platform. Check it out at testkube.ioMore infoFind all the links and info for this episode here: https://ku.bz/hGRfkzDJWInterested in sponsoring an episode? Learn more.