Skip to content
Voices of Oxide (Interview) episode artwork

Sep 26, 2025 1 hr 16 min

Voices of Oxide (Interview)

from The Changelog: Software Development, Open Source

Voices of Oxide on the pod! Cliff Biffle (engineer), Dave Pacheco (engineer), and Ben Leonard (designer) are on the show today. Jerod and I were invited to Oxide's annual internal conference called OxCon to meet the people and to hear the stories of what makes Oxide a truly special place to work right now. Cliff Biffle is working on all Hubris and firmware. Cliff says "There's a lot that happens before the 'main CPU' can even power on." Dave Pacheco is leading the efforts on Oxide's "Update" system. And Ben Leonard in charge of all things brand and design at Oxide.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Voices of Oxide (Interview)

0:00 1:16:14

Chapters

  1. 0:00 This week on The Changelog
  2. 1:22 Sponsor: Depot
  3. 3:40 Cliff Biffle on Hubris
  4. 5:36 Oxide's writing culture
  5. 7:17 Cliff's blogging
  6. 8:28 Caring for product
  7. 9:58 Into the firmware
  8. 12:02 Cliff on Go
  9. 12:34 Tried to not write Hubris
  10. 14:15 Kernaling Hubris
  11. 15:29 One rack. Many Hubris'.
  12. 16:11 Oxide chips
  13. 17:16 Users of Hubris
  14. 18:29 Being IRL together
  15. 19:59 Being on the inside
  16. 20:31 The Office meets Silicon Valley
  17. 22:17 People churn
  18. 23:52 Uniform compensation
  19. 25:53 We're all owners in some way
  20. 27:33 Sponsor: CodeRabbit
  21. 28:40 Dave Pacheco on Update
  22. 30:09 It's Mupdate
  23. 31:47 What's an "Update"?
  24. 34:31 Air gap users
  25. 35:18 Update on Hubris
  26. 37:31 Update without rebooting
  27. 40:55 Failure to Update
  28. 45:35 What if you didn't have to reboot?
  29. 47:50 New novel testing
  30. 50:42 Dave and Bryan is history
  31. 53:07 Solving these problems with Rust
  32. 57:44 Dave on OxCon
  33. 59:33 Ben Leonard on Oxide's design
  34. 1:00:34 Subtle tweaks
  35. 1:01:14 Product is the design
  36. 1:03:09 Designing Oxide's rack
  37. 1:06:32 Designing the unseen hardware
  38. 1:08:24 Oxide's values to design values
  39. 1:09:54 Properly excited
  40. 1:11:59 It's the little things
  41. 1:12:56 Is growth exciting?
  42. 1:14:02 Super awesome and beautiful
  43. 1:14:35 Closing thoughts and stuff
of matches

Transcript Auto-generated, may contain errors

Well, friends, it is your favorite podcast, The Changelog. Yes, Jared and I have a special episode for you. Jared and I and team went to Emeryville, California at the invitation of Oxide. We went to Oxide's HQ. They have an annual conference every year. It's an internal conference called Oxcon, and they invited us out to celebrate and to peel back the layers to have a good look at the inside of Oxide. So they recently raised a Series B round of $100 million, and they also just signed a purchase order.

a massive purchase order that is truly helping them cross the chasm. Now, we also have three awesome conversations for you. First up is Cliff Biffle. He's in charge of all things Hubris. Hubris is Oxide's operating system. And Cliff is also in charge of pretty much everything that happens before the CPU and powers on. Next up is Dave Pacheco. He's in charge of all things update. And update is the system for which they update the system. It's really important. And last up is Ben Leonard. Ben is in charge of all things design and brain for Oxide. And if you like how Oxide looks, I do. Well, that's Ben. A massive thank you to our friends and our partners over at Fly. Check them out at fly.io. All right, let's do this.

what's up friends i'm here with kyle galbraith co-founder and ceo of depot depot is the only build platform looking to make or build as fast as possible but kyle this is an issue because github actions is the number one ci provider out there but not everyone's a fan explain that I think when you're thinking about GitHub Actions, it's really quite jarring how you can have such a wildly popular CI provider, and yet it's lacking some of the basic functionality or tools that you need to actually be able to debug your builds or deployments. And so back in June, we essentially took a stab at that problem in particular with Depot's GitHub Action Runners. What we've observed over time is effectively GitHub Actions, when it comes to actually debugging a build, is pretty much useless. The job logs in GitHub Actions UI is pretty much where your dreams go to die. They're collapsed by default. They have no resource metrics. When jobs fail, you're essentially left plain detective.

clicking each little dropdown on each step in your job to figure out like, okay, where did this actually go wrong? And so what we set out to do with our own GitHub Actions with observability is essentially we built a real observability solution around GitHub Actions. Okay, so how does it work? All of the logs by default for a job that runs on a Depot GitHub Action Runner, they're uncollapsed. You can search them. You can detect if there's been out of memory errors. You can see all of the resource contention that was happening on the runner. So you can see your CPU metrics, your memory metrics, not just at the top level runner level, but all the way down to the individual processes running on the machine. And so for us, this is our take on the first step forward of actually building a real observability solution.

around GitHub actions so that developers have real debugging tools to figure out what's going on in their builds. Okay, friends, you can learn more at depot.dev, get a free trial, test it out, instantly make your builds faster. So cool. Again, depot.dev. I'm responsible for parts of the really low level firmware on the computer. So everything from machine turning on up through fans and power management and all of that basically all the stuff that happens before the thing our customers think of as the computer turns on. Right. That's all well me and my colleagues now. Originally it was me. Is that like BIOS stuff or is it before BIOS? What's before BIOS? What indeed. Okay. So these big AMD and Intel processors and even the big ARM nowadays that you see in something like an iPad. There's actually a lot of work that has to go on to allow them to turn on. They need a bunch of different like voltage supplies stable. They need a bunch of

clock signals set up and devices set up and you got to get their flash ready for them before they wake up so they can get their code out of it. And so there's almost always one or more small processors in the machine that are responsible for doing all of that dirty work that nobody ever thinks about because it happens in the moment between like when you hit the power button and when the screen wakes up. Yeah. So that's our job. Pretty fast, like 20 seconds? 15 seconds? Ours is milliseconds. Okay, milliseconds. To me as a user, it's like 15 seconds. They mentioned PCBs, you're building your own boards. Yeah. So you're going first principles on a lot of stuff. Yeah, absolutely. And you're a first principles kind of guy? Yeah. You like that? I think I actually designed the first board we made. No way. Back before we hired actually qualified electrical engineers. Okay.

It was an MVP or was it? Yeah, it was just for testing out some of our circuits we proposed for the bigger expensive work because it's nice to make cheap things that if they're wrong, you can throw them away. Right, right. And so Offsite has this writing culture. I assume that you were kind of attracted to that or kind of helped formulate it. No, it was pretty well established when I got here. Okay. But like the fact that we make people write a bunch of stuff during the application process, the materials and like that whole packet of like, why do you want to work here? What's an interesting problem? Because it really saves time in the interviews. But like for a lot of people, that's daunting. For me, it was like, oh, you want me to write a bunch of stuff. I like writing. I can do writing. Nice. Yeah. Was that a process you liked, this?

Requests for discussion, right? RFDs? Had you done it before? Is this something that was... The previous companies I'd worked for had a process around design docs that kind of was similar, but not exactly. We are way more writing focused here. And in particular, most of the docs are living. So as we learn what we're doing, we go back and fix the docs that we wrote when we were done to try to better reflect. So that way they can also serve as documentation for the next person to come through. So it's not perfect, but I honestly couldn't really point to what I would change. It's working pretty well as long as you've got people that are invested in the process and that are comfortable expressing themselves in writing. Would you call it document-driven?

Do you think you start with docs or an idea, not so much a spec, but fleshed out first before you do it? I actually feel like one of the ways people sometimes have a hard time starting here is if they treat it too much like that. So it's not like you need to write a thousand paragraphs in English before you can write a line of code. A lot of things here start as a prototype. But then if you want to build consensus, if you want to get other people involved, if you want to try to get feedback, that's when you need to write everything down and share it. So when you were writing on your blog, this was about Rust, right? Lately, yeah. But back then when they found it? Since like 2015. Okay. So how'd you find Rust and why'd you like it? So I was working in firmware at Google doing like...

high altitude balloon tracking and communication stuff, and we were using C, and I've been using C since I was a kid, because my dad was . But it's really hard to produce correct software that doesn't contain bugs, particularly on a team with different experience levels, working in C, and I was bringing people in at intern level, up through experienced developers, and trying to get everybody working together and productive as a team, and just like the problems with a lot of work in process, you can manage it, like car companies do this all the time, but it takes them a tremendous amount of overhead. And I just wanted some way out of that. So there were a couple of different alternative languages I was watching at the time and Rust was the one that matured at about the right time and got enough things right to be worth spending time on. This is 2019 you said? This would have been 2015 originally. Way back. So you said recently, but that was a decade ago. Yeah. I'm old. Just want to point that out. All right. So you're writing Rust blogs. You're writing Rust. Here comes Oxide. You start working on Oxide. You're writing more Rust. I think Oxide was an interesting opportunity because I don't super care about our product.

I care that it's good. I want to make a good product. I'm never going to buy one of these. You're probably not ever going to buy one. I can aspire to buy one. We can set that aside. It's like a fancy McLaren sports car. I'm probably never going to buy one. This product isn't for me. I had to find other ways to really get motivated around it. The main things are this is a team that really wants to try to do things right from the ground up, which I can get behind. That sounds like a hell of a challenge. The team is amazing. My co-workers are amazing. You should talk to more of them. We will.

So you have to look for all the other ways to do this. And sort of the how do you build an engineering org from a team that fits around a small restaurant table to this size and be able to bring in people that don't have relevant experience, be able to bring in people that maybe had a career change but are enthusiastic. And building a framework in the software but also in the processes and the documentation to support expanding the team like that, that's the thing I got really passionate about. In the firmware world, is it written in Rust or is it written in C? Our stuff's all Rust. All Rust. So there's no... Yeah, from the time you apply power. There's like one thing in C, right? Wasn't it like some operating system? Yeah, there's still a C. Yeah, the host Alumos Helios is our sort of version of the Alumos operating system, which is descended from SunOS Solaris. That's mostly C. It's also older, pretty well-tested C. It's not new, potentially buggy C, so we think it's a lower risk. Does it change much? It doesn't change that much, although we've obviously had to extend it a bunch, but we've been doing a lot of the extending and changing in Rust. But other than that, like, all the stuff on top of that, all the stuff below that is all Rust. That's a good thing. Yeah, we think it's good.

I was talking to somebody that said if they had to not write Rust here. Yeah. Like, let's say Go, for example. All right. They were like, nah, nah, I can't do that. How do you feel about that? I don't know about Go specifically, but there's other languages. Not that it goes bad, but like compared to Rust for some of the things you solve. Not for what I'm doing. It goes not in the picture. No, it's not. I mean, there really aren't a lot of options on the systems I work on, which are like the 50 cent microcontroller that's, you know, inside of your credit card. Like, there's just not a lot of resources. I never thought about that. Computer in my pocket. What makes Rust uniquely... There's probably a bunch of computers in your pocket right now, actually. At least four. What makes Rust uniquely positioned for firmware? So Rust... Or your lower level things. The thing that Rust got...

took from C and the C family is like the C family gives you really fine-grained control over what the computer's resources are being used for at any given time. So you have tight control over how much memory is being used. You have tight control over whether memory is used at all or if you try to solve a problem through some other way. Or even size. Or size. Size of code, size of flash required. Strings numbers. Replicates that control pretty well. Languages that go are less focused on that and don't come out of the box with as much help in that area.

Do you mess with Go at all? I have a little bit, yeah. How do you feel when you do that? It's okay. I mostly just kind of feel like a foreigner. Like, it's not really my native territory. I could get more comfortable with it if I needed to. What are you doing with Go when you do play with it? I do some, like, periodically I'll do, like, they have, like, the advent of code exercises or, like, programming exercises that people put out annually, and I try doing them in other languages just to kind of keep my brain stretchy. Yeah. So I've done stuff like that, but I've never used it in anger. Never for anything real. Tell us about hubris.

Yeah, so I tried really hard not to write Hubris. When I got here, there's this other operating system called Talk. It's also in Rust. Okay. Targets the same sort of very low-level, deeply embedded. How do you spell that talk? T-O-C-K, like you'd expect. Like TikTok. Yeah, but not like TikTok. Not the platform. Right. The way clocks work. Yeah, the original TikTok. Yeah. They were trying to use TikTok.

Sorry. It fried my brain. So they were trying to use Talk when I got here. And I had some previous experience with Talk. I know the people that wrote it. So I got in line and tried really hard to make it work for our application. But we just kept hitting areas where their design intent and the things we needed didn't really overlap. Talk is mostly at the time being written with educational use cases in mind. So they wanted kids to be able, kids, university students who are adults, I'm old, to be able to dynamically reload programs on it as they're working and nice use case stuff like that. But we really don't want that for security reasons. We want any code that runs on this better be what we shipped when we shipped it.

We put a bunch of work in trying to work around that. And then finally in like May of 2020, I think I wrote an RFD that was like, guys, I think we're going to have to do our own thing. And like, here's a rough sketch of how it might look like. And there were enough people here that had been involved in operating systems works before that they all kind of, we pressed our heads together and they said, okay, this might work. Like take a week and see how much we can prototype. And we got a thing working. And then it seemed compelling enough that now it's, there's...

what, 64 to 70 computers running it inside every rack, all the little service processors you don't think about? They're all running that. They're all in hubris. So how big or small is hubris in terms of like line count or whatever? So it's the core kernel is like a thousand lines of code. But there's a bunch of other stuff you want to make it useful. But it runs right now on everything from like sub-50 cent microcontrollers that you wouldn't even spot on a printed circuit board because they're just a tiny fleck of silicon up through the big service processor that we use to run the oxide rack, which is like a, it's basically a computer you would have been really excited to own in 1999 or 2000.

but now it's $3 inside of another chip. And does that job. Are there multiple instances of Hubris on a given full rack? Yeah, so every slide has at least two. There's the service processor that is responsible for basically care and feeding of the big AMD chip. Then there's a root of trust that handles security and crypto. That's a separate copy. And then there's, so there's those two on every compute slide. There's two in every switch. There's two in every power supply. And that's everything in the rack. But then a bunch of our manufacturing tools are also running Hubris. So all the little boards we plug into a thing to program it or, you know, to interpose in an interface for testing and be like, you know, I need to remove this fan and simulate the fan controller. All of our tools for that are all.

Why so many copies? Is that hard to manage? It's like... Yes and no. Multiple updates, different versions potentially even? That's true. It has pluses and minuses. So like the SP versus root of trust split, which is the main source of the many, many copies. There has to be at least one of these on each board because the board's got to be able to power itself on because it might be the first one powering out. That's a specialized case. That makes sense to be a copy bit. So one makes sense. Why two?

And the honest reason why, too, is we can buy one chip with the features we need to do the service processor. We can buy one chip with the crypto security features we need to do the root of trust. We can't get them both in one chip right now. And we can't afford to make iron chips. So when you can, we might have one. We might not. There's advantages to having a famous... Are you alluding to making chips in the future? Hmm? Are you alluding to making chips in the future? I mean, we'll probably have to. Yeah. Would it be a collab with, like, AMD or an existing... AMD returns our phone calls now. Yeah. It's very exciting. Is that new? Yeah, it's pretty beautiful. But hey, Oxide, we'll take that phone call. That's cool. Yeah, it's nice.

Probably not. So we have some FPGAs on the newest generation server board, which are basically, yeah, so it's a field programmable gate array as the full nerd expansion, but the purpose of the chip is it's basically a bunch of, it's a Lego set for integrated circuits. You've got a bunch of generic logic circuits that you can then program to act like another chip, and it's slower and more expensive than the other chip would be, but if you can't afford the million plus dollars to get started making your own chip, this is like a way to think it, essentially. We have one of those on the next generation server kind of playing around with some things we would do if we made our own chip. And Hubris is open source. Does anybody else use Hubris? I've heard from about five other companies that are using it in production. Really? Yep. And I can't remember which one of them. Let me say that publicly. Also, I can't remember them anyway right now, but I can check my notes. Sure. Volvo is really interested, but we don't have the certifications that they would need as a car company. But I've been talking to somebody about what that would take.

They can contribute though, right? It's open source. They could. The thing that would need to contribute, unfortunately, is a bunch of money for consultants to go through the certification process, which they're not excited about. So that's fine. That's fine. Well, they probably have, maybe they have more money than you all now? They just got some money. I don't know how well you think so. I bet they do. I have no idea. It's all though. Yeah. Car companies, they burn money. Don't they burn money? Yeah. There's like three other startups I've heard from that are using it in products. That's pretty cool. I'd like to get more people using it, but there's some work we need to do to make it more friendly to people that aren't outside.

Right. Because right now, if there's a trade-off we have to make, and one thing would make us ship faster, and the other thing would make it more general for other customers, we almost always have to pick the we ship faster option. Right. One of the fun things about being here in the building with everybody is every once in a while, a fan just goes crazy. This is not a person who enjoys it. A literal fan just starts to...

What's going on when that happens? That's a great question. So one of two things is going on. Okay. The good thing is somebody just ran something on that machine that's like boosted all the CPUs. It's like when your laptop starts trying to take off. Sure. So something made the CPUs go really fast, everything ramped up, machine got hot, fan turns on, cools the machine back down. That's sort of the working as intended. What's probably happening here is that something's crashing. So we have a chip on the board that's a hardware watchdog that if it doesn't receive regular instructions on what to do with the fans, it assumes the worst and it ramps the fans up to make sure we don't overheat. To avoid, okay. This means that if you're doing a firmware update...

on the service processor, the hubris-based service processor that's responsible for sending those messages, and it's gone for more than watchdog, however, the setting is seconds, then the chip takes out, ramps the fans up, and then it wakes up and finishes doing its update, and then the fans go back down. So this is why, I don't know if you saw earlier, but whenever those fans go up, I'm like... You're like the watchdog for the watchdog. I can yell that in chat. Because you're one of the people that works out of the office generally. Yeah, and like, I might have wrote the code that's messing up. So you better get out of your seat and go do something. So frequently I'm like, dummy, dummy, this one's dummy.

So as being somebody who's regularly in the office, but most folks aren't, what does Oxcon do for you in terms of your camaraderie with your colleagues or the excitement level? How do you feel about it? I get to find out how tall everybody is. You can't tell that on the computer. Like Aaron. Aaron's like damn near seven feet tall, but he looks normal on the computer. So that's been interesting. Other than that, gosh, I don't think the company would work without this, honestly. I mean, how would you even... It doesn't even feel like you're at a real company. It's like you're watching a TV show of a company.

Good point. Like The Office? Like The Office or Silicon Valley. Oh, gosh. Are you a fan? I worked there, so I couldn't really make it through the show. Like, my boss was personally married on the show. I worked there. He loved it. Did he? Yeah, so Astro Teller at Google X was the inspiration for the Hooli X guy in the show, and, like, he thought it was hilarious. He had a showing of the episode. With the monkey guy, the guy. Yeah, so he didn't have a monkey, the real guy. Oh, man. That was good. But, yeah, so I was watching the show a little bit, and I'm like, why are you making me watch this? Like, this is my day job. Like, I'm going to watch Game of Thrones or something. Yeah, I can understand that.

So you've never gotten past season one or even one episode? No, I made it like three episodes in. How about now? You feel better about me since you're free of that world? I'm good. You're in a different world now? Sadness. He lived it. Yeah, I'm all right. I was listening to Brian tell one of his stories. I was like, oh my gosh, that was literally in the show. Like something he described from the stage today regarding money, funding. And I was like, that was literally copied. I mean, it's real life, but it's there. And so I can understand that. For me, it's entertainment because I haven't worked for Google and I haven't done your life. For you, I can imagine how it's PTSD. Yeah, I also don't watch a lot of TV, so I'm kind of picky about what I spend my time on.

Hopefully we don't do anything to get a show made about us. Or you do, and it's good. That could happen, that could happen. I just started watching that. That's a hell of a show. That's worth your time. One thing I've been thinking about is churn. Is there any churn here at all? We've had people leave. Yeah. Without being, you know, TMI, what are some of the reasons? Have they been negative or has it just been just mutual separation? I actually really like basically everybody who's left. So, like, for some people, this work environment doesn't work. Honestly, it barely works for me. Like, the all remote thing, I actually took this job because I turned down two other offers that were fully remote because Oxide at the time wasn't. Yeah. This was in February 2020, so you can guess what happened next. Yeah. So.

Oh, well. Literally next. It's a good thing my workers are great, but the whole remote thing doesn't work for some people, and sometimes you can't ever really get in the swing of things. We've had folks where this turns out to set off other past work trauma. We've all got work PTSD from some shitty former boss, and if things are happening here that's too much like that, you might get freaked out and decide, do you need to leave? Which I totally respect. We've had folks like Arian. Arian joined before me and was involved in bringing me over here, and he just left last year because he's like, I've been here for five years. I feel like I've done all the startup-y stuff I can do here. I'm going to go do a new startup. Just done. Yeah.

So mostly good reasons, not like, this place sucks, I'm out of here. I think there's a lot of upside. You could probably find people that think this place sucks. Yeah? Yeah. I'm not totally sure who, but I'm pretty sure you could. One person? Probably one or two. Two people. Yeah. I don't know. Out of 80, that's pretty good odds. It's not too bad. How do you feel about the uniform compensation stuff? I think it's amazing. It's worked this long. It was part of the reason I joined. Because I came in all knives out, expecting to negotiate. And Steve's like, so what we're doing is we give people stock according to this formula, and we pay everybody the same amount of dollars. I'm like, well, that saved me a lot of stress. Like, sold. Yeah. Probably the opposite of your time at Google. Yeah. 100%. Oh, my God.

My only real concern there is that one of the things I really like doing is bringing people in who either don't have a lot of experience in the industry or are just out of school or just out of some other job. Right, mentoring them up. Right. And are we comfortable bringing in people that are basically interns and paying them pretty good Bay Area salaries? Maybe we are. Yeah. But I do want us to be able to bring those people in because that's how we get the next generation of us. That's a good point. But if we can do that while keeping the compensation uniform or at least fair, that would make me really happy. Yeah.

What does it do for your personal ability to show up to not worry about compensation? A lot. As much as you had to before. Oh, my God. So, like, not having to worry that some of my coworkers are getting screwed over by having not, like, hardball negotiated in their interview. So I became a manager at Google of a team that I had previously been on. And at that point, at the next promotion cycle, I was able to see everybody's salaries for the first time. And that was how I found out that there was, like, $100,000 a year difference in salaries among people at the same level on my team. Wow. That's a lot of money. Yeah, and it was mostly us guys that had the higher numbers. Yeah. It was kind of crap. So, like, I don't have to worry about that here, which is great. And I feel like people are a little more comfortable talking about...

Both job conditions and also kind of like financial stress. Like some people have been really open about like, you know, my husband lost his job and like we've got the new kid and so it's kind of rough right now. And I feel like people are a little more comfortable sharing that because we all know what we each other make. So it's not like you're going to reveal that like, oh, wow, you're being really overpaid and now all your colleagues are mad at you, right? So it's got its perks. This may not be accurate, but one thing I thought about was the fact that it seems like you all are owners of the company. Like everyone in there that works here owns. Small owners. Right. You all have equity. Some may have more, some may have less. And the compensation is the same across the board. But what changes is you have a different job.

One person has a CEO job. Now that person may have more equity, but that's because they also started the company. It's because they were here early. The compensation is a little different. The day-to-day, the check, the reason you show up, it seems like an even playing field. And y'all are sort of owners in a way, as much as you can be owners equitably. I really don't like hierarchy. So when I'm managing people, I view manager is just a job. It doesn't mean that I'm above you or in control of you or better than you. It's that I'm going to do the manager things. You can do the engineering things. And we're both happy. And I'll develop skills here. You develop skills there. But I kind of feel like we've got the same thing going up through Steve. It's like Steve is the CEO. He does the CEO work that we don't want to do.

Who wants to do that job, right? That job you described on stage, I was like, wow, that's a hard job. Like our sales team, they do the sales that I don't want to do. They're good at it, clearly. They actually make different amounts of money. They're the one corner case because salespeople, we have an incentive thing where the more they sell, the more money they make, which they were excited about and we don't have a problem with. So, yeah, I kind of, I do think it helps with the sort of sense of like we're all in this together. Yeah. Which is good. 100%. Yeah. Well, we'll let you get back to it. All right. Thanks for chatting with us. It's been awesome. Yeah, my pleasure. Appreciate it.

Well, friends, the news is out. Our friends over at CodeRabbit, CodeRabbit.ai, they've raised a massive Series B, and they've launched their CLI reviews tool. It is now out there. I've been playing with it. It's cool. The bottleneck is not code. The bottleneck is CodeReview. With so much code happening, so many people coding now, so much code being generated, and so many things competing for developers' time and attention to maximize, CodeReview still remains a bottleneck. But not anymore. CodeRabbit, CLI code reviews, code reviews in your pull requests.

Code reviews in your VS code and more. Teams now have a true answer to what it means to code review at scale. Code review at the speed of AI and CodeRabbit is right there for you. You'll learn more at CodeRabbit.ai. We'll link up their latest blog announcing their Series B and their announcement of their CLI review tool. Again, CodeRabbit.ai. Okay, friends. Next up is Dave Pacheco talking about Oxide's update. Here we go. What is the update on the update?

Well, let's see. We're working on shipping the first version of what we call a self-service update for the Oxide rack. So today, the process for updating the Oxide system, including the control plane and everything, involves a support process where our support engineers are getting on the system through a debug interface. And that works pretty well for a lot of things, and it's very simple. But it doesn't work for a lot of customers to have an Oxide person involved in the whole upgrade operation. And so we want that to be something that they can do through the API, just like they can do all the rest of the infrastructure stuff. But that is kind of a big deal because...

That means that the control plane is driving the update, which means the whole control plane is online during the update. And so, you know, people use different metaphors for this. It's replacing all the parts on the car while you're driving down the freeway, replacing all the parts in the plane while it's in the air or whatever. But it's just a lot of work to make that work. How long have you been working on that? I've been working on it for about two years. How long has Oxide been working on that? We needed an update process for our MVP. And so that started before that and finished before that because we launched before that. That's the process that we use today in our support-based process. This is like an upgrade and rewrite. Kind of, yeah. So the idea behind the first version of Update was what we call the minimum upgradable product, which was Mupdate. And so the idea here is that, you know, it's an MVP. We're shipping it really as fast as we can, but there's a lead time between when you deliver software to the factory and when you actually get it at the customer site. And that's a period in which we can continue working on the MVP. And so what we needed was for there to be enough in that.

first thing that we could update it to whatever software we wanted once we got to the customer site. And that became the minimum upgradable product, which is an update. And that's the procedure that we have today. So the priority there was about having a robust support procedure for recovering the software on any one of our compute sleds. And that's something that we knew we needed even separately from update. And so that's why it came first, was this idea that we need the ability to recover a sled, it's one of our compute sleds, no matter what state it was in, and we can use that to do our initial updates. And that's what we've been doing for the last two years. So in that sense, we've been working on it for a while, but in terms of being able to have the control plane do a more operator-friendly update, that's been about two years. And actually even that,

there are a bunch of building blocks involved in that, like what I've come to call dynamic reconfiguration of the system. So having the ability for any component to come and go while the system is running is kind of a prerequisite for that, but also allowed us to deliver other important things that customers would expect to be there, like the ability to remove a sled, replace a sled, add a new sled to the system. So the first year of the update project was really building this foundation that we use for these other support procedures as well. Gotcha. What exactly is an update? Describe, is it big, is it small? Yeah, no, that's a really good question because you think of the control plane as like, it's the control plane, it's just like one big piece of software or something like that. But actually, every single one of our, so we have,

In a rack, we've got 32 sleds, we've got two switches, and we've got a couple PowerShell controllers. Every one of those has a service processor, a root of trust, and the root of trust has its own software and the bootloader software. Then on all the 32 sleds, we also have a host OS, which comes in two parts for, you know, historic reasons around bootloaders and stuff. Then we have all the control plane software on top of that, including storage software, which is one per disk. What all this means is that when you update the software in an oxide system, you're updating literally hundreds of components. And, you know, we're kind of doing it one at a time. And you're also going through all these intermediate states where you're running some of the old software and some of the new software.

So you're asking, what is an upgrade? We're replacing all the software running on everything in the system. And it's a lot of different things. And that list I just gave doesn't even include a lot of stuff that for us gets bundled. Like CPU has microcode. NICs have their own firmware. For the update system, that's simplified because it all gets part of the host OS. But there's a lot of software in the system. And it's updating all of those things. But the whole idea of what we're doing is that operators don't have to think about any of that stuff. So our release process puts together a giant zip file. It's like two or three gigs of data. You download that from us. You can look at it if you want. You can validate it, whatever. But then you upload it to the API and you hit go. And then the system goes insurance for probably a couple hours up front.

and then you come back and then the whole thing's updated. So the idea is that the operator's only thinking about this policy. They're not thinking about all these other things that are involved. That's nice as a zip file. I mean, several gigs though. It's a lot of software, yeah. I mean, even the bandwidth cost on that, do you measure that? Does it matter? As you grow your customer base? I mean, obviously not. I mean, because you're getting paid lots of money, but you got to worry about those things, right? Speed to get it, accessibility. It's a good question. You know, it's not something we've been focusing on for the most part so far. It's the kind of thing where the customer's currently going to be responsible for getting that from whatever our download site is, whether that's GitHub to the Rack. Once. Yeah.

So they might get that on their laptop, downloading it from GitHub or something like that. And then they'll upload it to the rack. Yeah, that part's GitHub. It's a release. All right, that makes sense. And then uploading it to the rack is over their network. And we've kind of been assuming it wouldn't be an issue. That's a fair question, whether they would consider it an issue. I'm talking about size. The file size, I guess most people can download a couple gigs pretty easily without it being a major problem. But some people get it.

Yeah, that's really important too. And that's what this model works pretty well for air gaps because you're downloading it to you, the customer are downloading it to your laptop and then uploading it. No connection to the rack. Definitely doesn't care. It doesn't even know if it's connected to the internet. That's fine. It doesn't care. You could imagine a nicer experience where the rack was connected to the internet and could see, Oh Oxide just published a new thing and I'm going to download it. Maybe I only download certain parts that I need or I download it one part at a time. Maybe that helps with some of the bandwidth stuff, but actually it's not something most of our customers are interested right now. Most of them are actually more interested in the

I'm really not connected to the internet and I don't want you to, and I'm doing this because I care about my security and my privacy and my data. I definitely don't want the rack talking to the internet. So that's why we've done it the way we've done it. We talked to Cliff earlier about Hubris. So when you look at Hubris, that's the operating system. You mentioned a couple of chips, every different device on there. When the update comes, is that on top of Hubris, the API of Hubris? Or is that, how are they compared to each other, this update and Hubris? Yeah, so Hubris is a sort of, it's an operating system. We use it in a couple of different components in the service processor and the root of trust on all these systems, on the sleds, the switches and the PowerShell controllers. So it's one of the things that we update.

It is also true that in order to update everything, we talk to the service, or in order to update much of the system, we end up talking to the service processor, which is talking to that Hubris thing. So we end up using the current version of Hubris through the service processor to be able to update the service processor itself, the root of trust, the root of trust bootloader, and the host OS as well. All the control plane stuff is on top of that and doesn't go through Hubris. Is this a novel problem that you all invented, given your architecture? That's a good question. Yeah, a lot of the details are specific to our architecture and pretty novel. It's the sort of thing that I expect cloud providers today have their own bespoke software for, and in fact, large deployments of on-prem stuff.

will have their own bespoke software to do a lot of this stuff. But a lot of it is also stuff that people kind of just don't update. Like, how often do you update your BIOS? How often does a company running on-prem software update their BIOS? Probably not all the time. But our model involves delivering a lot of value through stuff like that, and we do need to be able to update that stuff. I remember back at one of my past jobs, we did have to go update the BIOS on 64 systems or something like that. And you've been in the BIOS thing, right? You're clicking through the thing, and how do you do that on 64 systems? And at the time, this wasn't a productionized thing, but at the time, I didn't know that iTerm or whatever has this mode, which is like...

send all my keystrokes to all the other panes. You just taught me that right now. Someone just opened up 64 panes and was just like, enter, enter, tab, over. Oh my gosh, that would be that person. It's a real problem, right? When you have this software, it's such a low layer that's not really designed to be interacted with by automation. Everyone that's had to do this has had to come up with their own way to do it. Well, one reason why you don't update your bios very often is because you've got to reboot your machine. And I know that ultimately your guys' goal is no reboot update, right? That's not what you're working on now, though. That's going to be the next phase, and it's going to be a much smaller part of the problem for us. At least, you know, we expect. It's going to be easier. You never know until you're done. It's not going to take you two years. Right. So we've done a lot of the pieces

All the stuff we've been doing so far is like the orchestration. It's foundational stuff. So like our system is based on what's called the plan execute pattern, which means that before taking any action, the system generates a new intended state of the world, which we call blueprint. And then it goes and executes that blueprint. And like all that was really important foundational work for...

building a system that can be operated autonomously, which is also really important for the air gap thing. Because we can go test all kinds of things that can happen with just the planner part without even worrying about the execution stuff. And then we can go test all the execution stuff given whatever plan we want without having to have gotten a system into exactly that state. And it also lets you do all kinds of things like ask the system, what are you going to do next before it does it? And why are you doing that thing? You know what I mean? So these are really important operational things that we just need to have.

That's the kind of stuff that's taken the first two years. So you laid a lot of groundwork. That's right. And so now we're talking like doing what we're calling non-disruptive updates. So this is doing updates without rebooting the customer VMs. We're still rebooting the sleds. So you move the VMs? Exactly. We're going to live migrate the VMs. And now that's a question. I don't mean to oversimplify it, but that should be just a question of policy, which is like we're flipping a bit in that blueprint that says this sled needs to be evacuated. So you have an oxide rack, 16 sleds. Is that typical? Yeah, 16. All right. So I got 16 sleds. And we need to run an update. This is in the new world when this exists.

and I got each sled has, I don't know, 30 VMs on it. Okay, now we're doing math in our head. Oh, he's going to do a math problem. No, I'm not going to. And it comes time to run update. So I go put it on my thumb drive, off my laptop, whatever, plug it into the rack, and it's going to run. It's going to live migrate VMs off of a sled one at a time, update that sled, reboot it, and then move some stuff back to that one probably. Something like that. Yeah, that's where actually... Distribute that load evenly across the other ones in the meantime. That's where a lot of the complexity does come in with the non-disruptive update is, first of all...

how do you like mechanically move these things around? It's like a bin packing problem. But then there's also like, how do we make sure that we have the capacity to do that, right? If we're going to start doing it up, if you've totally filled every sled. You're running your thing at max. Right. There's no way to move them. Right. And so. Buy another rack. Right. Well, that's the thing is like, at scale, people actually don't care about this problem. Because like keeping a couple of sleds capacity free when you've got a hundred racks, it's like a very small fraction of your cost. And it actually makes sense for a lot of reasons. It also allows you to sustain failures and like put that stuff over there. But when you only got one rack, that might be more of a problem. So then there's the question of how do you create an experience for the operator that communicates clearly what the trade-offs are, but also gets this input from them, which is like, what do you want

to prevent you from using all your capacity so that you can update it? Or do you want to have the possibility that you go do an update and we just say, sorry, we're paused right now until you can tell us, just reboot all these VMs or whatever you want to state. That reminds me, or that makes me think of failed updates in the self-service world when this version's out, not the, what do you call it? Undisrupted? Undisrupted, yeah. With the current iteration you're working on now, can you guarantee that an update will...

Finish. That's not what I thought you were going to say. Finish, no, because there's things outside of our control. Like, for example, one of the biggest challenges in self-service update, because the control plane is running, we have these intermediate states I mentioned where you've got new version of software talking to old version of software. And how do you avoid that becoming a problem on our ability to change our own software because you have to do backwards compatibility like forever, right? And one of the ways we've addressed that is to say that there will be an order to the updates. So we will always update, for example, the host OS before we update the control plane that talks to it because the reverse never happens right now.

And so that's fine. But that means that if you're doing an update and one of the sleds is like out to lunch, we can't talk to it. We don't know if we've been able to update it. We can't actually keep going and update the rest of the control. You gotta wait for that sled. Yeah, so we gotta tell the operator, look, you either need to what's called expunge the sled, which means to remove it from the control plane and we'll pretend like it's just caught fire, like it's failed, we've moved everything else elsewhere. Or you gotta figure out what's wrong with it and bring it back. And that'd be a support call. Probably a support call. And then unless they go, they just unplugged it and just plug it back in or whatever. Those sorts of things are always outside of our control.

Yeah. Now, what do you think I was going to say? I thought you were going to say an update that explodes. That, like, you start doing the update, and the control plane is now down, and what do you do? And, like, that's my nightmare. That's been my fear for the last couple of years. That's why it's taken two years, and I'm not done yet. Yeah, but seriously, that is why we've spent so much time on this, like, having the automation take these careful steps where every one of these steps we know is safe. You know, as an example, we've got a CockroachDB cluster that's, like, storing all the control plane database data. We've got five nodes. We definitely don't want to bounce a sled that's hosting a Cockroach node while that Cockroach cluster is already unhealthy, right? That's just, like, a thing we want to make sure we never do, because that increases the risk that...

we actually lose quorum on the cockroach cluster and it's dead and we're in trouble. So we have all these kinds of safeties built into the automation, all this testing, this whole pattern, all this stuff. So that's like one angle. Obviously testing is another example, but that's kind of a given. But it's a hard problem really because part of what we do, part of what is involved in an upgrade is making backwards incompatible changes to data formats, like database schemas and things like that. And once you've done that, the old software can't read the new thing. So rollback is really not possible. So what a lot of software does is they'll have like a point of no return, whether some call it a finalizer or a deferred update or something like that, where you basically get the whole thing.

kind of working before you've committed in that way. And then you ask the operator, like, does everything seem to be okay? And then they hit the button and it's like, okay, go. But even then there's like, there's still risk there because whatever it is you're activating by taking that last step hasn't been tested before that. And there's kind of no way to get around that. And that's kind of a future problem for us right now, but it's something we're going to have to deal with. So it is your nightmare. Has it ever been your nightmare? Meaning it's happened? Like in my career, yeah. Well, like specifically for Oxide and updates. No, no. Any updates gone wrong? No, but we haven't started doing that in production yet, so there's still time for it. How do we do it in production now? We're using the process we used when, yeah, the manual thing. So the way the manual process that I was talking about at the very beginning works basically shuts down the whole control plane, replaces all the software and brings it back up again.

How long is that process? It's actually not that long. It's shorter than the other one. I feel like it's safer. Why? I mean, I get the whole reason. Downtime, man. Can't have it. It's downtime and it's the self-service aspect. Although you could imagine a self-service version that looked more like that. But then the thing is like, if it's self-service, you're talking to an API, what's running that API while the thing is down?

you don't have, there's nothing, right? Yeah. So that's why you kind of, that's why you gotta do what we've done. Gotta be done. Yeah. So downtime's required now. Future is, you said non-destructive? Non-destructive. Non-destructive. Meaning the VMs get migrated around versus shut down, but you still reboot sleds. That's right. You're still rebooting controllers and control planes and stuff like that. Yep. But that's not visible to them because we have enough. Is there a world in which that isn't even required? Which part? What if you don't have to reboot anything? Oh.

I think that's pretty dicey. It's definitely a thing that people have done. Yeah, there's types of updates or patches to a running Linux kernel where you write the new one over here and then you jump over there. It's just a pointer. It is. It's all software, right? That's right. Who says we have to reboot, you know? Yeah. That is definitely harder to do with stuff like the SP and the ROT. Part of it is that the ROT's job is to attest to the software that's currently running. Yeah, if you're going to change it out from underneath it. Yeah, then you can figure out what that means. And then there's also this risk that you're now in a different state than you would have been in if you had actually bounced that thing.

So have you created a time bomb for yourself where if that thing loses power and actually does power back on, is it going to do the same thing that it was doing? That's the thing I always, like bifurcated code paths where you're like, this is the thing we do sometimes and this is the thing we do other times is totally the kind of thing that results in something failing at runtime. Catastrophic failure. Yeah. All right, fine. Bad idea. There's the thing you asked about like my nightmare, right? The upgrade just like explodes and we're toast. Another thing that we've done there is try to create a lot of guardrails around the types of changes that we can make to the software so that we know if we're making a change that's going to break things. So for example, if you're changing the database schema, we know that you're changing that and we've operationalized that one. So that's kind of fine. If you're changing like an internal API.

we make sure that you're doing it in a way where none of those intermediate states will expose us to a situation where those components don't speak a common version. And that's something I imagine other organizations do do. I haven't actually really seen that before, but I think it's really important. Because that's the way that I've seen this fail in the past. It's like, someone goes and makes a change to the API, they're like, I'm not changing upgrade. So they test everything, everything seems to work. You go deploy it, and it blows up in the middle. It's like we've tested the endpoint, we've tested the beginning point, and we just got unlucky in one of these intermediate states that wasn't tested. And so we've tried to identify the kinds of changes that would cause those problems, and then detect those at CI time for us, and then build time if we can. That's cool.

Any newer novel testing strategies that you've had to come up with as far as this, I don't know, fuzzing or carbonistic testing or anything? That's probably the biggest one. The other thing I would point to is that sort of distinction between the plan and execute stuff. We haven't actually gotten to this kind of thing, but one of the things we want to do with that is like property-based testing on the planner, where you're basically like sending all kinds of different inputs at it and putting constraints on what kinds of outputs can happen and see, you know, make sure it never does anything crazy. Make sure it does that. Yeah.

We're supposed to give a talk today. Yep. They got rescheduled. That's true. We're not going to be here for it. Right. So you have to spill the beans. Yeah. So let's see. What was I going to talk about? So I've been doing this for a couple of years. Update on update? We have a surprising number of new faces. And so part of that talk is literally the stuff we were just talking about. Like, what is update? What is it? Well, we've got a couple hundred components that we've got to replace. And, you know, pick your metaphor or whatever. And so part of it is just like laying that out. And like what we do today, the stuff we've been talking about, why this is a problem for customers, what we're doing, current status of that, which is like we're planning to ship the self-service part very soon now. And the non-disruptive is coming after that.

And then the rest of it was probably, I don't know if that's really interesting to a broader audience, but it's kind of reflections on what it's been like to run a project for such a long term. And I don't know, maybe it is more generally interesting. But like, I have a lot of fears about update at runtime, but my big fears about update as a project was that it would feel perpetually a year away. And we would make decisions day to day and week to week that ensure that it continued to be a year away. Because when something's like a month away and somebody asks you to do something else, you're like, sorry, I got to do this thing. Like, we're shipping it in a month. But when something's a year away, it's very easy to be like, well, here's a really important problem over here.

And like, it's hard to know what the next thing is to do on update. That's been really a challenge the whole time. It's like, what's the next step? There's so many steps and there's so many circular dependencies in those steps. They're like, well, I got this other important problem over here. Maybe I'll just like kind of solve that. And that's fine. Sometimes that's the right call. But if you make that call, even 20% of the time, right, exactly. You just never get there. How close is it? How close? You said very soon, didn't you hear him? Yeah. Update. Yeah. In the next month or? That's the plan.

That's not a year away, though. No, no, no, no, no. This was two years ago I was worried about that. Two years ago, I think I was, like, perpetually a year or two away. I saw the roadmap this morning. Yeah. I saw last year's roadmap. It's not a roadmap. It's an important thing. One of these was update. Yeah. And then the priority this year was on going. Accurate. But almost ready. Yep. Almost ready. That's cool.

Why you? So we had a side conversation. He's been with Brian almost his whole career. Sun to Oracle, the acquisition, and then Joyent. And then what makes you uniquely positioned for this task, this quest? I don't know if I'm uniquely positioned, but right after we shipped the MVP, this was one of my big worries about the products. And I was like, this is something I think I have a lot of experience with in terms of building distributed systems and reliable automation and things like that. And so I thought this was a good opportunity for me to swing in and try to create the vision that I want us to get to and then be able to execute that.

I was interested and I asked and that's how a lot of stuff works around here. What has it been like casting the vision for it and getting feedback from the team? Because I'm thinking culture, I'm thinking the process of getting feedback on any idea because you're leaning out all this groundwork and you're doing all the work to kind of get to a direction. What is it like to put that idea out there, that vision out there and get that feedback? start moving on it. It's great. I don't know if you all talked about the RFD process, but... The RFD process, yeah. Yeah, so when I started on this, the first step was writing an RFD, I think it's 418, if you want to find that. And it's basically... Is it public, the RFDs? Some of them are and some of them aren't. I don't know if that one is. It probably could be, if we want it to be. There's nothing particularly sensitive in it. But it's kind of laying out...

where we are. Lay of the land, this is where we are, these are the problems, this is where we're trying to go. And like, it was very specific in some ways. It was like this idea of playing execute pattern and the automation has to be safe and all this stuff. But it was also very like, we have a lot of stuff to do and I don't know what all the pieces are yet. So that was the first step, is getting everyone aligned on the vision. And that RFD itself was a team effort. You know, I drafted this first version, but people are looking at that. And I think broadly, there were no surprises there. Everyone was like, yeah, this all makes sense. And then that process just keeps happening. You know, you have more and more specific designs and say, okay, let's get some feedback on this. And I do enjoy that part of it. And I enjoy the collaboration and it goes pretty well. Like it's not...

It's not the sort of environment where you're worried about what so-and-so is going to think about this and is someone going to be unproductive about it or something. Sure. We know that Rust is a foundational language here, obviously. Yeah. We talked about this being somewhat of a novel problem. How is Rust uniquely positioned to help solve this problem? What about Rust makes this problem easier than another language that you may choose to do this with? Yeah, so the big thing for me about Rust that I really love and I think that's been huge for Oxide is its ability to help us...

ensure things, especially at build time, that need to be true of the system. And so that sounds really vague. But what I mean is you can catch so many problems early. And everyone talks about the obvious ones. We talked about this earlier, but everyone talks about the obvious ones. Like it's the borrow checker will help you find memory safety problems. But it also allows you to create abstractions for the rest of the team that can't be misused, right? So you can say, I'm creating this thing. Like maybe you represent it with an object and you say, you can't.

do these two operations concurrently on it. Well, that's the thing that we can enforce in the type system. And you literally just can't compile the code that would do that. That's awesome. And it sounds so low level, but it's extending that same idea and applying those same tools to do it that allows us to say, if you try to evolve the API in a backwards incompatible way that won't work at runtime, at upgrade time, you might get, depending on how you do it, you might get a build failure. You'll at least get a CI failure. And so, I don't know if we have a minute to talk about it. I mean, so one of the first things that I built here with Adam Leventhal is something called Dropshot, which basically lets you write an HTTP server and then generate an open API spec from the code.

And then we feed that into something called Progenitor, which generates the clients for it. But that alone means that if you make an incompatible change to an API, even before we've done any of the versioning stuff, your client fails to compile now, which is hugely valuable. And that's true not just because it doesn't have the operation in it, but you passed in a new one with three variants and it now has four variants and you need to accommodate for that fourth one or something like that. I call them enums. Yeah. So anyway, we ended up extending that. So now that's how we do this versioning stuff is we have a bunch of these open API specs that are the ones that this thing supports. And then we know if you've changed it because we know if it generates a different thing, then you've changed it incompatibly. So Rust has facilitated all this stuff.

Sorry, the last thing I want to say about that with the Dropshot thing is like you've got your rich structured types in Rust and you're like all happy because I'm in Rust land and I've got my strong types. That carries all the way to the client because of the way it goes. Like Dropshot just takes those types and puts them in the open API spec and then the client generates faithful things on the other side. So you can basically get that strictness all the way through there. We got a little bit more on this then. Speak to your confidence in the code you write because it's Rust. I'll just...

Yeah, I mean, I'm reluctant to make any bold claims about the correctness of the code I write, lest I immediately walk back in there and there's some horrible thing happening. But I mean, I'll say this, there's a lot of changes I've been able to make where I'll go work on the code for like four hours, and when it compiles, I know it's already correct. I know I haven't broken anything. And it's not as simple, you know, I know there's the cartoonish version, it's like if it compiles, it works. And I'm not talking about that. I'm talking about something that's like, it's either a refactor, this happens a lot with refactors, or I'm building a new thing in terms of these things that already exist, and I'm plugging into the middle of it, and it's like, there's no way for this to be wrong at this point, because like it's...

it fits neatly into the narrow interfaces on both sides of it. It's correct. It's huge. It's so huge. My previous experience was in Node.js and it was the complete opposite of this. And part of the reason I love Rust is by the time we got to the end of the road at my last job when we were using Node.js everywhere, every single JavaScript function we had started with like 30 assertions about the types of all of its arguments. And I'm like, why are we doing this? The computer can do this. That's what the compiler is for. And so I have so much more confidence in those things now. And that gets back to why I think Rust is so valuable for this problem space is that these things are so complicated by allowing us to encode all these constraints into build time constraints.

That allows us to evolve the software so much faster and with so much more confidence. Someone can come in here and make a really big, complicated change, and you're not wondering, like, well, what about, I wonder if they missed some caller or something like that. It's like, no, you haven't. You've covered every single case. And it's huge. It's huge. Right on, man. What else? Anything? Tell us about Oxcon. Oxcon. Oxcon is awesome. So this has been something we've been doing for a while, and this is something we, it's very similar to something we did at Joint, these sort of company-wide meetups, engineering-wide meetups. You know, for a remote company, it works really well to be remote basically all the time, but it is also so valuable to have that time in person together to get to...

There's so many conversations that don't happen if you haven't scheduled a meet for it. And we have other ways of trying to have those conversations anyway, but it's just really nice to have the FaceTime with people. And also the small talk, and you're going to get dinner, and you're just talking to people about whatever it is, learn more about them. It's a really good time. So there's a mix of the structure time that we have. Brian and Steve talking about all the exciting stuff over the last year. It's inspiring for people. It gets everyone gymmed up to talk about everything that we've got to go do. And then there are other good company-wide sessions talking about important projects and stuff like that. But then there's also all this breakout time that I started by talking about, which, as with any conference, the hallway track is almost.

It's at least as important as best. It's the best. Yeah, right. So that's huge. Not really. Hallway track only. Keynotes only. No hallway track. That's our move. This is so much more. It's getting more like mature and professionalized. We never had a stage before. We never had like pro AV before. So this whole week is a little bit of a dream. It's like Otcon already feels different. And then we've got the stage there. And then it like rained in the Bay Area in September this morning. I was like, am I dreaming? What's going on here? Yeah. But it's a good time. Thank you. Appreciate it.

Well, friends, last up is Ben Leonard in charge of all things design for Oxide. Here we go. One of the things that stands out about Oxide is design. I think you can have a good company, successful company, but... Every company that's successful is set apart by its design. Can you talk about how you came to be here at Oxide and the design story behind that? Yeah, so I was working at a branding agency for Pentagram. Oxide was my last branding project. I was planning on leaving to work as a freelancer. I'd been at Pentagram for four or five years. I was ready to move on. And it just so happened that Oxide were looking for a designer just as I was leaving. And it felt like a unique opportunity. What would happen is you'd work on these brands and then you'd throw over the fence. And then you'd check in on it two years later. Like, what have they done to my baby? They destroyed it. My boy, my boy!

um and so it was a unique opportunity to continue it on and i'm i'm kind of i was trying to you about this before i'm like relentless that i like i like to fiddle with things i can't work on it yeah i kind of get bored of it as well so i just want to i just want to tweak something so we i mean recently i mean like we made a modification to the to the logo that uh most people wouldn't even notice it's slightly thicker but it's something that's bugged me this inside yeah yeah yeah he was up here i was like steve's shirt has the thinner logo yeah yeah and on stage here it's much thicker I noticed that. I saw it. Right away. I noticed turning. Oh, yeah. So I'm always kind of filling with stuff. But then the beauty of outside is I get to work across everything. So I'm working across. Generally, most of what I do is working on the product. So in my case, it's the web console. So you're working on the design system, the UI. I'm also working on the brand and the marketing and the sales assets and all this stuff and industrial design. The beauty being is that those all inform each other. Because what you tend to happen is you have your product team and your creative team, and they're very distinct.

This is a bit in the weeds, but we have the same UI design system that drives every piece of design. So we have the same colors, we have the same UI elements on both the website and the web console. They're kind of continuously informing each other. And it's difficult because I think there's been a shrinking of creativity in tech design. That is, branding stopped being... What it was is that...

The product itself, so the proliferation of SaaS, the product itself became the design. So the UI language of the product became the design. So Linear is a great example of this, which is the design of the product and the design of the creative are one and the same. What that can mean is you have the world of design within product is much smaller. And so what that can mean is you have everyone's always complaining that all tech looks the same now. And that's probably because product is much smaller. And if your brand looks like your product, then your world is much smaller. So I'm kind of, I'm always figuring out how wide I can go with the branding and keeping it still.

like it comes from the same world as everything else. And yeah, it's a... Yeah, the industrial design, the creative, the product, there's so much to work on. Would this rack behind us look like that at all if you weren't here? It'd look a bit like that. Same shape. Same shape. If I wasn't here, it'd be squatting the other side. Less green, probably. It might be less green. Or any more green. So we consulted a little bit with my old agency, Pentagram. Industrial design is there, and we worked together on that a little bit. One thing that I'd learned from working on a hardware product previously is...

The first version was, it was a small run that they distributed to people, right? And so they're producing this thing, which is initially really magical. And then as soon as they mass produce it, they think, okay, we're going to have to make this cheaper, easier, more practical to make. And then the next version looks really, really bad because essentially they were just trying to recreate what they've done. And, okay, we can't do this complicated tile system. So we'll just print it directly on, it was a PCIe card. And what it was, it was compromised in the worst way. So I think like going into it from the beginning, thinking like, what's the, what are our limitations? Like we're going to make, we're going to make thousands of these, like how.

from the beginning. I don't want to do things that just exist in this kind of first, this initial run, like, and so figuring out what the, what those compromises are. But working, working with hard, like, just working with hardware, hardware is hard. I haven't done very much of it. But yeah, you're, as soon as you're touching materiality, you're, you're dealing with, yeah, I mean, we were speaking about, like, color, color is the bane of my life. Like, we have, we have the, we have painted drive bays, we have powder coated metal, we have, like, plastic pieces and,

And color matching the green, I think is... Yes, that's probably gotta be your thing, right? Is that a little off sometimes? Oh, it's a little off all the time. And some of it's outside your control, so you have to think about how to handle that, so...

What you do is you avoid putting elements next to each other that might be different. If there's a little bit of separation, then visually you can get away with a little bit of difference between those things. One example is originally the rack, it was a black, but it was a slightly bluey black. Like bluey as in a cartoon bluey? Exactly. Bluey black. I'm sorry. I'm bluey black. Disregard. It was a cool black. Cool as in color temperature, not cool black. I thought you liked it. But my thought was, as soon as we start integrating multiple components,

Then I'm trying to make sure that I'm using the exact right black. Oh, possible. That's hard. Yeah. Well, I don't know how much of it is me trying to avoid upsetting myself. Like these things that no one else uses. Someone's like, whatever. Yeah, yeah, yeah. This is like trauma-driven design. Right. Yeah, yeah. You guys have TDD. We have the same. Do you get into the PCB design then, too? Like, how much? Or is it simply this? I don't want it simple. Just the easy stuff back there. Do you get to step into that world where, like, how is this?

Because I think about Apple. Apple has done a great job of branding everything. Look at their CPU, right? Their latest M1 chips are the latest M3 chips. It's a design to everything they do. You get into that as well. I mean, Apple's a good example because they're stunning inside as well. A little bit. Occasionally I dip my toe in and I get shouted to leave. So the PCBs are currently there.

The PCBs are green, and I think once I came into a channel and I was saying, hey guys, can we make the PCBs black? I'm thinking, okay, you change the color of PCB, you just, I don't know, you just order black ones instead of green. I'm always told that's not the case. I think it does have impact to, like, thermo and stuff like that. What did they tell you? They told me some information, none of which I retain. You're like, sorry I asked. Maybe they said it, maybe they went somewhere, maybe they were just saying it because they were like, yeah, yeah, yeah. Black was cool, I've seen black PCBs. But there are issues with, yeah, there are manufacturing ramifications, which mean that it's, yeah, it's a bit more complicated than that. I think with the rack, you have to take, um,

There's a balance. We can't go full Apple and just invest so heavily in details which cost money for a little benefit. But what we can do is make sure that something which is an item which is underappreciated. I like taking boring objects and boring designs and boring, boring things and making them better. It's easy to work with Nike and do some beautiful designs for something that's already compelling. But I think it's much better to take something for which design does not usually feature and try and elevate that.

So Oxide is big on the values. You know, you go to the website, there's values. I'm sure you know all the values. They inform all the decisions, how you act. Do you have like a design language or a value system or something that drives your design for Oxide that's separate or maybe even congruent with that? I think there are ideas. So I think my approach to branding is you have one idea which should then filter through into everything and that's how you make a holistic thing. It's not necessarily that this looks like this. It's that it comes from the same place. So with Oxide, Oxide is an old idea like brought new, right? So it's this.

old crazy idea that you own your own hardware. And so you see that in the design language, you see the ASCII. I mean, the green is like this kind of terminal green. Like it looks, it has like this kind of retro edge to it. And so the design language is like old meets new. And yeah, so I think that's like nostalgia is a key part of it. I mean, you see over there, we have the lowest nostalgia is a key thing, but you don't want to lean so much into nostalgia sketch. So yeah, I think that's sort of referential.

um referential things to kind of old computing is a is it maybe that's why it speaks so well to us because we're all about that yeah yeah where the old beats are new is interesting precisely what gets you excited about what you do here like what makes you be like hell yeah i like to say hell yeah i don't know the british equivalent of hell yeah is um i don't think with that um yeah i don't think with that yeah we're not that immersive Yeah. We'll get you properly excited. Now you're talking his language. I'm probably excited. No, I love to do a bit of everything. And it's the variety, I think, that really drives me. I think that's why. I mean, I've been here more than four years now. I think that's what variety is what keeps it interesting. And, like, outside the company where you get to do that, you have hardware, you have software, and everything kind of in between. That's the thing that keeps me interested is the young. I would expect the opposite. Because at an agency, you think that's where the variety comes. Because you're on to the next project. Like, new.

new company, new design language, new brand. Whereas when you decided to settle down with one brand and do that for years and years, I would expect it to be less variety, but you found that it goes wide. I mean, I think certainly working at an agency, I think it's always going to be much more varied than working at a startup. I think for me, there's this interesting tension between what's staying the same and what's different. And so there are things in the Oxide brand which have retained logo more or less, the colors. But I think there are things that are kind of constantly changing. And obviously, Oxide as a company is changing, so design needs are changing. As an example, I think we are growing, and so marketing and sales needs are growing. And so previously, there was very little collateral outside of the rack, outside the web console and the website.

That was it, so you have that kind of small world. But then when you're kind of entering the world of sales and marketing, then that kind of opens the door to a bunch more design. That thing at the door when we came in was a card with like a chip on it. Did you design that? The coin, yes, yeah. Does that come off the card? It does, yeah. It has a little, yeah, yeah, yeah. It's got two sides. You're afraid to. One that I can actually use the coin for the coin. Yeah, one to sell on eBay, yeah. Design that. Yes, yeah, yeah, yeah. That's fun stuff, right? Yeah, I mean, that's what I think. Those are the palette lenses, so.

The big pieces of merch tend to, they tend to get made just as I finish something that's like, that's like drained my creative energy. Website is the big one because, I mean, I'm usually coding the thing too. So I've designed it, I've coded it. It's took so long that I'm ready for something else. So we have, as you saw those, the little kind of rack stickers. Those I designed right after I'd done like a run of the website. And so those things are a nice little, they're a nice little time. They're a way to kind of refill my like creative cup a little bit. Are you excited about the growth? I imagine with new footprints, we're doing video, we talk to...

A little fourth wall breaking here behind the scenes. We talked a little bit more about being able to put motion graphics into place and collaborating on that. Is that exciting to you to see the different areas you can do? Yeah, for sure. I have a list as long as my arm of things that I will eventually get to. Motion is one of those things. There are things I want to do, but you don't necessarily have the reason to do. I was chatting about this before, which is the size of the gamut of creativity of things that you can do. Product design is much smaller because it needs to be functional. People are using it every day. There's also the zeitgeist of the way that people use products. It needs to be designed in a way that's expected.

and websites a little larger and then the kind of creative space is even larger than that. And then you just think about what are these like one-off excuses to do something different? So like product launches, all those things that you can get a bit more experimental around. We love the work you do. Yeah, it's great work. It's beautiful work. Thank you very much. I think the company's awesome. I think the design really, in my opinion, is like, it's the glue. You can have a great product, you can have a great software, but it's the final piece that says, this is just super awesome. Basically, all right.

the design to me is what sets it apart yeah i mean the design the design is what's like says you care about everything yes yeah it shows intention it shows trust it shows all these things yeah it's just really important so getting to go to oxide's headquarters for this conference oxcon was years i would say in the making it didn't happen overnight and over the years we become closer and closer friends with brian and steve and the rest of the team there behind oxide and just getting a chance to celebrate with them this moment this series b moment this massive new order moment this crossing the chasm moment i know for jared and i was quite awesome so Glad to share that with you. Glad to get this on the pod. Glad to peel back the layers with Cliff and with Dave and with Ben. These different parts of Oxide, these integral, important and different parts to truly see what's on the inside of Oxide. The good news is, I guess even better news really, is I think there's more to come. They like working with us. They like what we've done so far for them. And I think you're going to as well. We're producing a 10 minute inside Oxide documentary for the YouTube channel. So it's kind of cool in the works, in the edit booth right now, in motion as we speak. But man, I've seen previews. I've seen the teaser. It's epic.

A massive thank you to our friends who sponsored this podcast, our friends over at Code Rabbit, our friends at Depot, and of course, our friends and our partners over at Fly. Check them all out. They love us. You should love them if you will. We appreciate that. Of course, the beats are by our good friend and longtime friend, brother, Breakmaster Cylinder. Love those beats. OK, that's it. This show's done. We'll see you soon.

Frequently asked questions

How long is this episode of The Changelog: Software Development, Open Source?

This episode is 1 hour and 16 minutes long.

When was this The Changelog: Software Development, Open Source episode published?

This episode was published on September 26, 2025.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Changelog: Software Development, Open Source episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.