Rewriting the Hydra Queue Runner in Rust (nixcon2025)

The Hydra Queue Runner is a critical component responsible for orchestrating and executing build tasks within the Hydra infrastructure. Its reliability and efficiency are paramount for our Nix community, serving as a key component of the Nixpkgs infrastructure that builds the world's largest package set. A significant challenge with the current implementation lies in the communication between the Queue Runner and the builders. This system, built on SSH, directly read from and wrote to the running Nix daemon, which imposed a hard limit on connected builders. This talk will begin by explaining the current Hydra CI infrastructure and how Hydra works as a whole, looking at the evaluator, Queue Runner, PostgreSQL, hydra-notify, and hydra-web components and how they interact with one another. We will then detail the comprehensive redesign and re-implementation of the Hydra Queue Runner, transitioning from its existing architecture to a robust, high-performance solution built in Rust, specifically to overcome these limitations. We will present an overhaul of the remote communication protocol, migrating from the SSH implementation to gRPC, leveraging its benefits for performance, type safety, and interoperability. This change also benefits our ability to introduce generic messages unrelated to the Nix protocol, which enables the monitoring of the system utilization of all builders, making scheduling decisions more agile. Furthermore, we introduced comprehensive tracing capabilities, making the new Queue Runner significantly more debuggable and maintainable. We will then dive into the build pipeline, distinguishing between steps and runnables, examining the changes made to the build queues (now handling each platform with a separate queue) and what is needed to resolve a derivation. Lastly, we will present compelling benchmarks demonstrating how these changes have significantly accelerated builds and enabled a substantial increase in the number of concurrently connected machines. about this event: https://talks.nixcon.org/nixcon-2025/talk/SNFQ7J/