When Async Attacks!

What happens when the Oxide API is slow? A podcast episode! More specifically, one about how the team employed all manner of debugging techniques to track it down to one obscure and configurable async runtime feature! Bryan and Adam were joined by members of the team to talk about that journey and the tools we used (and made!) along the way.In addition to Bryan Cantrill and Adam Leventhal, we were joined by Oxide colleagues, Dave Pacheco, Eliza Weisman, and Augustus Mayo.Previous episodes mentioned:Oxide and the Chamber of MysteriesThe Saga of SagasDTrace at 20Cultural IdiosyncrasiesMr. Nagle’s Wild RideA Debugging OdysseyRTO or GTFOSome of the topics we hit on, in the order that we hit them:Falling in Love with RustTokio Runtime Builder – disable_lifo_slotmagic‑trace (GitHub)Magic Trace podcast episode from Jane Streetdiesel‑dtrace (GitHub)omicron issue commentqorbstatemaptokio‑dtracetokio issue #7411Visualizing Systems with StatemapsPostgreSQL WAL INIT ZEROStatemaps: Visualizing System Behavior (YouTube)The statemaps that we referred to:Nexus by thread, discussed starting at 55:29. (This statemap has some states coalesced; the full version is also available.)Nexus by Tokio task, tagged by thread, discussed starting at 1:15:33The D scripts that we referred to:nexus-statemap.d used to generate the initial statemapnexus-profile.d to understand what was consuming CPUtokio-statemap-tagged.d to generate the Tokio task statemapIf we got something wrong or missed something, please file a PR! Our next show will likely be on Monday at 5p Pacific Time on our Discord server; stay tuned to our Mastodon feeds for details, or subscribe to this calendar. We'd love to have you join us, as we always love to hear from new speakers!