Distributed databases with Peter Mattis

He built the game from Jeter while at college, designed the original storage system via Gmail, and built many more large complex and widely used systems. This is Peter Mattis, co-founder and C2 of Comroach Labs. Before we sat down to talk, Peter told me, for the last 30 years I've always been a prolific coder, but my current output is a bit insane. And this is invite code of junk, but database-worthy, high-quality, high-performance code, thanks to working with strong coding models. Today we cover why B-trees are so important on building databases, and why Peter kept reaching for the data structure over and over throughout his career.

How he wrote 100,000 lines of production code per year, pre-AI, stop coding between 2022 and 2024, and why he is back now. Why he thinks AI agents are lazy about testing, and how this is an easier fix than it looks. And many more. If you're interested in distributed databases, distributed storage systems, or knowing how Peter manages to use AI to produce unusually high-quality and production-ready code, this episode is for you. This episode is presented by TurboBuffer, a ridiculously scalable, fast and cheap, hybrid search engine built on top of object storage by the engineering team I've really gone to like after spending time with them.

The TurboBuffer engineering team is doing something really, really cool. They're completely redesigning their storage architecture from first principles to make storage faster, cheaper, and more reliable at scale. If you follow TurboBuffer, you know that their storage architecture was a massive part of their early success. Redesigning winning architecture is a big deal. It's one thing for your query plans to pass unit tests.

It's another to bring each query plans to performance parity, or better while maintaining correctness and reliability in production. Here's a cool part. TurboBuffer is documenting the whole thing. Their new storage architecture, which they're calling TPUF V3, demands some hard-core systems engineering and their building and in public, sharing design decisions and benchmark results as they ship. They're keeping a work-log of their journey, and the first polls just drop today.

Follow along at TurboBuffer.com slash V3. That is TurboBuffer.com slash V3. Peter, welcome to the podcast. Oh, I'm happy to hear this. It's awesome.

I want to get into, how did you get into tech originally? When did you figure out computers are interesting? I figured that out in elementary school, high school, gaming was a little bit of a gateway drug for me as a software engineer, as many other people. I remember early on, my mom did programming at IBM at some point. I'm not even quite sure what she did, but we had computers always around our house, like Apple 2 Plus, Apple 2GS.

I'm from that era on up and go to the bookstore. I find a book on basic or magazine type and programs, no idea what I was doing. But I was just kind of like, I was a digital. You can put stuff into these computers and get interesting stuff out. And I got to college, and I was like, I didn't think there was any money in computers.

I didn't know anything about it. I started as a mechanical engineer. Following the footsteps of my dad. You started at Mechanical Engineering as your socialization. Yeah, as my major.

I got in there and I was like doing, like, I'd done some computer stuff before. It was real foolish when we did this. But first semester, I was doing these homework assignments in Mechanical Engineering. They were got off, like, six pages for a single problem. And I happened to take a CS course at the same time.

And it was so easy. And then everyone was just failing it. I'm like, I'm in the wrong field. Let me switch. And then you switched.

And then I switched. Yeah. What was the first software that you built? You started college and also been at college. In fact, you were like, all right, this is a piece of software that I'm kind of proud of.

That's a complete piece of software. Well, I mean, the big thing that I did along with my remaining college, we had this course was a compiler's course. And I'd done journalism in high school in my senior year and I'd used stuff about computer graphics and wanted to do something like Adobe Photoshop. So started taking the tires and we built up this program that a lot of people know of called the GIMP. And along with the GIMP, I did a lot of the graphics library GTK.

This has since evolved just massively since then. It's kind of interesting because after college, I kind of stepped away from it. Didn't really stay involved much past my first year out of college. But definitely, people still know me and it led to other interesting events in my career. And it was like starting GIMP.

It was just you just saying, all right, I want to do something like Photoshop, how hard could it be? And then you just, this wasn't through what you learned in college, right? This was like you figuring out how to build, you know, like a graphical, I guess, engine, rendering, drawing, data structures, all that stuff. All that stuff, figured it all out. I remember trying to look at some papers back then.

My roommate was looking at papers. We're just figuring it all out. And it's one of these things. You almost kind of need to be a little bit naive to start anything like that. Because if you know how hard it's going to be, you never started.

So I think there's like, you've had this level of like, if you really, truly wise about effort, you like never get into it. But then when you just keep on snowballing, it gets farther and further along. And then at some point, you're like, wow, this is really awesome. But there's an interesting little tidbit associated with our work on the GIMP, which is kind of a little bit known. I think we've talked about this before, but it's just kind of fascinating where we were getting to the point where we're like, we should release this to the public.

And back then there was a newsnet, news groups where people would post stuff and there's one on graphics. And I remember like just a couple of weeks before we were going to release our first version of the GIMP, someone else came on there like, I've been working on this graphics program and did everything the GIMP did, every last thing. And then some and we're just like, oh, that sucks. I guess we'll just keep on working on it. It's been fun.

And then we released the GIMP. Never heard from this other guy again. And I just got a little less of that. There's always going to be someone else working on your idea. I don't know, but I'm not going to get dissuaded if they pre-enounce it.

Nothing ever comes of it. And a lot of the marking behind some of that stuff he might hear is like, I don't know, if we stole his thunder, I'm not going to show it happened. He never traced down what happened. But there's a real lesson there. So there's a possible future where you read this announcement or someone saying, I'm going to build all of this thing.

And you go like, ah, you know, someone did it. And you kind of go back and just do something else. And it never happens. That's right. Wow.

Yeah. I guess what today was start-ups. Sounds like just do your thing. Put it out there. Not very least, right?

Yeah. I'm not quite sure.

I'm not quite sure. The very first version of the Google logo was down the game. So they knew of us. Somehow one of them you know, woke me up asking me to come interview at Google. I did back in 2001.

This was when Google was three years old. Yeah. Three years old. And I said no. You did not.

I said no. Yeah. No, here's the calculation of mine. I was like, Google's down the view. And I didn't want you to commute.

So I went and worked at another startup for a year. And that one after a year I thought, like, I saw it wasn't going anywhere. And they called me back up and he said, Hey, do you want to interview again? like, we're not going to be able to keep the same stock off because you got a form like, okay, I can't remember what it was. I still can't remember what they offered me the first time, but it probably wouldn't be a lot more money if I take that first.

But I did all right, I'm not complaining, but it's one of these things like, yeah, I think back up, and I'm like, oh, a lot of good stuff came out of it. Some point along the line, Red Hat was going public, we actually offered friends and family stock during their IPO with a lot of people. We got offered, I made a little bit of money off that lot, but it was like, just after college, it was actually quite significant at the time, you know? So, and there's a number of good things that kind of came out of that. And I also used the GIMP as a, you know, I was looking for like, oh, I can't afford Photoshop, and here's the GIMP, and it did so many things.

So like, I'm sure so many people, like, you had a positive impact of being able to use this thing. And the nice thing that I, whatever you liked about it, it was free, but I wasn't like stealing any software from anyone. You see what I mean? This was at a time where free software was not as common as free and open source was not as mainstream as it is today. Yeah, yeah.

So, and the other cool thing, I've heard from a lot of people, because people still have like, I mentioned this, and then some people, you know, software engineers like, I learned how to program, I look at your code, and I'm just like, wow, that was a code I wrote 30 years ago. I wasn't even good to software engineers like I know. And then you got into Google seconds time around, you said yes. What did you start to work on? Yeah, yeah.

Well, I got in there and they said like, hey, you know, this was early day, 2002, it was April 1st, auspicious date. You're sorry, yeah, April 1st, 2002. And I got in there and they're like, well, you know, we're gonna actually build email, Google email. And it wasn't called Gmail at the time, there's a code name internally, it's called Caribou. Caribou, yeah.

And got in there, started working on it, and I was gonna task with working on the backend threading and message storage and indexing system, and worked on that, you know, pretty hardcore three years on it through the launch, which actually happened to be April 1st, 2004. Do you remember this? It was like, is there an April Stole joke? Yeah, I remember. So correct me if I'm wrong, but the launch said, Gmail with one gigabyte of storage or some sort of infinite storage, and I like star one gigabyte, I'm not sure what it was, but back then most email providers would give you about 10 megabyte of storage to free email providers.

And then you could pay for maybe 50 megabytes or 100 megabytes, but that was really expensive. This launch, it looked like an April Fool's joke because who could possibly give you a free email service with 20 to 50 times or 100 times more storage. We were talking about this internally, it was like a shock and awe campaign for the industry. I think it was actually four megabytes for like Hotmail or Yahoo email and then you had this. And not only that we had so much more storage, but it was index really fast.

So you can do a search and come back almost instantaneously. Can you tell me, when the project started, okay, we're gonna do email for Google, how the UN the team arrived to the point of like, okay, we will offer all this storage and we'll do it fast because this was the time, if you can take us back, but what I remember is hard drives were still expensive. They were relatively slow, we're talking HDD, we're not talking necessarily about SSD if I remember, but if you can take us back, okay, can you take us back up what it was like, what the constraints were like and then how you innovated, like to actually do something that's never been done before. So at the time, Google and friendly had this large distributed file system called GFS, Google file system. And we were basically looking at numbers and being like, yeah, we think people on top of this, they had a lot of knowledge about how to do search and retrieval, it ended up being that some of the existing systems that had been search and retrieval, they started building the prototype on there and then that was completely written, that was actually what I got involved in, because I got in there and there was already a prototype and then it was completely written.

So I was on that threading side, we decided to have message threading right from the get go, which is also kind of innovative, that was in common in email systems. And then threading, the message thread, which I assume you needed data structures on the back that needed like storage and figuring out read-intensive little kind of things. Yeah, there's some B-trees involved in that. I think that might be the second time I implement B-trees. And I think I've implemented them like a dozen times now.

How do B-trees relate to email threads? It's just like you have to have some storage there, we have a thread ID, a message comes in, you have to look up, it was using the search index to match actually the subject of the message ID into the thread and there were some other things taking place in the B-trees well, because we were keeping track of the unread counts of threads and whatnot. So I can't even remember all the details now. This is ancient history. This is 2004, what are we in 2026, 20 years ago.

So it's like left my memory, but there was definitely B-trees. There was also this in the next taking place there. All this code has since been completely re-written. What were the economics? In the team, you must have done the economics of being able to offer free email, which of course I'm sure you did some math so how it could be subsidized but to make a terrible, terrible loss.

Yeah, yeah, yeah. We were kind of kicking around ideas. And one of the ideas that had come up at some point was, hey, we can put ads on this. And it was one of these crazy things. So I didn't, wasn't involved in this.

This is Paul Buckeye went on to do some cool stuff at Friend Feed and Facebook and I think I'll partner by combinator at some point. But he just like 190, he's like, I think I can take her to some of our existing ad functionality and corporate in there. And it just went like gangbusters. And there's a huge business that kind of grew up for that, which is kind of incredible. So I think there's a lot of things that it's just like, you kind of have a sense of what we like, oh, this is going to cost per user a couple of dollars per year.

How do we monetize that? And we didn't want to charge for it. We eventually did charge for it because you have the whole Google Works based stuff at the time that was more like, can we do this for free? Oh, it looks like it can be economical. And it took a little bit of a leap of faith.

And do you remember the launch? The reason I'm asking because I remember that it was an invite based system. Like not everyone could get in. And I assume that must have been to control the expected demand because again, you were offering something for free that was paid before. And it was kind of pretty obvious that there would be massive demand.

How did you think about it, kind of monitor demand, decide how many people to onboard? Honestly, this is a team effort. I wasn't involved in that. I mean, I do remember being worried about like load that would happen. And then someone came up with the idea of like, hey, we should do an invite based system.

And this also had like dual roles. So it kind of constrained the growth, but it also kind of drummed up this excitement. Oh, it can get me in a Gmail event. I remember people ask me at the time, I was like, yeah, you always make Gmail invites as you want. And then after you built the system, where did you move on to?

What was your next project? Was it a build system? Well, there was a build system. That was kind of a model to my mind. I was kind of doing that part time when I was doing Gmail as well.

At some point, Google actually had this large monitor repo. I think they actually still had the monitor repo. And they started out with the Google one repo. That was before my time. Then they moved on to Google two.

That was what was there when I got to Google. And at some point, we saw the strains of Google two. And Google two was just one single monolithic make file. I think they actually had some stomach files, but it was this really large and really make file. Someone came to me and was like, I think we could do something.

I'd had some interest in the build systems. I put the foundation in for Google three. There was a bunch of people involved. I was doing the initial work on Google three. And Google three, the initial insight was like, hey, make files kind of sucked right.

We introduced this thing called build files. And I decided to do it. It's just this stripped down kind of Python language. But it was still Python at that point. And what you can think about at the end was a monolithic make file when you didn't have to write.

Over time, this evolved. It became Blaze internally. There's other systems that are safe with now. I don't even know how complicated it's got. I haven't seen it for a long, long time.

Then that became Basil externally. It became Buck. There's some other people who went to Facebook and they're like, that was awesome at Google. What was the reason that make files or make didn't really work? Was it, what we talking about build performance?

Are we talking about maintainability or readability? Yeah. So like, I think make itself is like, it's OK in declaring dependencies. But it's a little bit like kind of assembly language. And you don't want to write, you know, all your dependencies in assembly language.

If you didn't know what you were doing, it was easy to make a mistake in dependencies and whatnot. And so the kind of like, the thought behind the build files is like, they need to express these dependencies at a higher level and just like, kind of clean or semantics associated with it. And from that, you can compile down into your separate language. And then eventually folks are like, oh, we don't have to compile down to that. And then you can implement, you know, the dependency, kind of update engine directly.

And then that's where the performance improvements were able to come in. Because at Google scale, like, when you have a large read, in general, like a minor sounding is that the reason Bazel and Buck are so popular for large code bases is it can help you improve your build performance. It gives you a lot more lovers to play around with, from caching from being smart about cache generation to obviously just raw performance. Yeah, that's exactly it. Yeah.

So you kind of just dabbled and like, OK, I'll make this build file, that that people, that will go. What was your next main focus? Well, so I mentioned GFS earlier, Google File System. And at some point, we realized that there were some limitations on GFS, the LBLD-FALL-DLE, because I've been working on the storage system for Gmail. I actually tabbled it in another storage system, kind of a research thing that never went anywhere.

But because we worked on that, I'm going to invite to participate in the founding team of Colossus, which was the successor to GFS. And as far as I know, Colossus still exists. It's gone through multiple iterations at Google. But it's like the second generation, you know, distributed file system. So what is Colossus?

Yeah. So when I say distributed file system, externally, you might think of something like S3. Kind of blob storage. It had flat namespace, kind of like S3, give names. And there's like a minor hierarchy there, but it's like very limited.

But it's not like a POSIX file system. So you don't have the full directory hierarchy. You didn't even have the full permission system. Well, that stuff got it out of later. But the files are not stored into local machines.

There's a service out there that has all the files. They're writing it down to either hard drives or now SSDs. And your client can access that. And it's all replicated. So if there's any crashes and whatnot, you're not losing your data.

I don't know when at some point, S3 got a ratio of coding. We did a ratio of coding in the classes. That was kind of a big breakthrough. Which coding? We use resolmen.

So for the audience who's not familiar with the ratio of coding, you might think of like, I want to have replicas. And there's this thing in hard disks called RAID, where it's like, I don't actually have to have full replicas. I can actually take a plus, a and b, you can x-worm together. And then you can have this third version. And there's more and more complicated versions.

That read-solmen is kind of, I think there's actually better codes now. But it's one of the known ways to use this. And we had to pioneer internally how we actually make this work in distributed file system. Where you focused on latency, on being able to store data more. It gets more stored in it more efficiently.

So in GFS and other varying costs, it's where it's like, well, you're not storing two copies, or one copy your data or two copies storing three, triplication, three times as much storage you're having to use. And with read-solmen, you can get that down quite a bit lower. Remember, offhand, exactly what we used for re-solmen? I think it was like, it's actually 2x. But you get that with also the redundancy, too.

So it's like, it's smaller and redundancy. Because I guess the naive thing, if you're saying, all right, I want my data to be replicated three places. You take three nodes, three machines, physical machines. And you say, like, copy one, copy one, copy one. I have three places.

Great. If one explodes, I still have two. Wonderful. And then you're saying that the algorithm here is you could take not three exit data, but two exit data, split it smartly across machines, or maybe you could take it to the lower end. You still have a thing where like, oh, one of them explodes?

I still have all my data. Because it's split it off. That's exactly right. And the mental model, if you just want to stand at a high level, which is essentially, you might want to say, I want to have eight replicas of this data. I think I can't remember offhand.

I think S3 might use nine. They've actually talked about this publicly. Yeah, nine chunks of data. But any five of those chunks can be used to reconstruct it. And what this means is you can lose any four copies, and you can still reconstruct your data.

And oftentimes, it's more like the first five chunks of your exact replicas and the other four are kind of parody ones. I've kind of forgot some of the details to keep my mind. But it's a long different line. But when you come up with an algorithm, you can improve that this algorithm will work. Because it's a little bit like I know in software engineering, maths and algorithms a bit out of fashion.

But in this case, this is really important. Because once you can prove it, that this algorithm works. It will work. It will work. And it's like the math behind here is like Go-Off yields over like GF2, something like that.

I never actually understood the full math behind it. I was regretting not doing more math in college. But you didn't have to. You like read and solve and prove to how this work. I think it's like back in the 1970s, something associated with communication networks.

So you just take that and kind of use that expertise, but leverage that and have to do all the engineering behind it to make it work in a storage system. And then one building, this should be the storage system like Colossus. What were other things beyond, OK, you want to store data in a resilient but efficient way? What were other things that problems that you needed to solve? I'm thinking things potentially like sharding and re-sharding or metadata being important, those kind of things.

Particularly for the scale that people want to operate out, GFS can have the scalability limit. I believe it was like you have 1,000 machines and a GFS cluster and a kind of thing. I guess it sounds big until today. It's kind of resistant to small rights. It sounds big for two months before.

And then Google was like, no, we need to have the scale up to 10,000 machines. And there's some bottlenecks. There's this GFS master, there's a single node, and there's a bit of a bottlenecks. We need to have a distributed master to store the metadata in this system called Bigtable. And so Colossus stored this metadata inside Bigtable.

And one of the things I'm kind of proud and proud of, and also a little bit embarrassed by, one of the design choices I went down, but it actually worked out is we want to use Bigtable for the metadata for Colossus. And the metadata just for those of us not as into this reason, what is the metadata? Well, yeah, it's like the names of the files and for each of the files that are broken into chunks, and then what were they? 64 megabytes chunks. And then you have to have the list of chunks for each file.

And then you have to periodically, you know, the master has to be scanning over this and doing repair work. But there's more metadata than that. But that's it in the shelf. So we want in Colossus to have this scalable service, Bigtable to store its metadata. The biggest user of GFS is Bigtable.

So we want Bigtable to work on top of Colossus. So you don't have to have the circlic dependency. Yeah, well, there's a good strapping thing, right? Oh, yes. Yeah, which one starts up?

You need to mock something somewhere, right? Yeah, no. I mean, the way it actually worked at the time, and they've since replaced this, because this is like the way to get started and leverage what you have, and eventually you can't get rid of it. But there is the foundational Bigtable, that Bigtable didn't use Colossus. There's the Colossus using Bigtable, and there's the normal Bigtable sitting on top of a process.

And it all worked for years. So I don't even know when they got rid of that. They got rid of it at some point, but that was it. But I guess sounds like you can make hacks that go really long knowing that they're hacks and they get you off the ground, right? They get you off the ground, right?

Because if we had to implement that Bigtable layer from the get go, what would you just delay how long it took to get Colossus to? One of the things that strikes me about a system like Colossus is it promises, or this was internal to Google, but even distributed file systems that are external, they will promise high throughput, high availability, and low latency. And to me, it's always a bit conflicting of like, well, it's pretty easy to, I guess, build a distributed file system with my limited knowledge. I could probably do something where I have either high throughput but high latency because whenever I write something, I write it out to all the replicas. You already mentioned one technique of doing it, but how do you kind of reconcile?

How do you get low latency while you have high throughput, while you also have a replication going on on the fastest? Well, I mean, these distributed file systems, I mean, the latency isn't super low. In particular, when they're running on hard disks, which Colossus was doing at the time, which S3 does, you actually notice the latency. So like S3 is a high-performance system, Google, GCS, the competitor from Google, which is built on top of Colossus. It's high-performance has incredible throughput, but the latency is like 20 to 30 milliseconds per street, and that is bounded by your hard disk latency.

If you put it on SSD, it gets down to a closer SSD latency, but not actually kind of the state of our SSD latency, which is kind of this crazy thing that's been happening in our industry. It's just like how much faster the hardware's been getting. Reading from a hard drive, maybe five to 10 milliseconds nowadays, and we never touch hard drives. Reading from SSD over NVMe, 30 microseconds, 50 microseconds. So that's a microseconds, there's a thousand microseconds in one millisecond, so we're talking like a huge, huge difference.

Well, there are now startups or infrastructure companies that are starting to take advantage of the fact that they can have an NVMe layer and they pull things up either productively or not, but it has to say, like when the physical reality changes, you can build systems on top of it, that should take advantage of it. Yeah, absolutely. And you know, the disks are so much faster with SSDs, the networks are so much faster, and just kind of crazy how fast the IntraZone latencies are in the Google or Amazon Center, and it wasn't like that, but when we were building Colossus, you know, I can't even remember the numbers, but milliseconds to do a network round trip, and now it's down in 100 microseconds within the zone. I mean, I just look at these things, I'm like, well, crap, you know, the harder guys have really done a good job. Yeah, sometimes I feel that our software should feel way more snappy, and there are some snappy software, but sometimes I almost wonder if we're doing two complaints with all the abstractions or not even doing this, like, now it's like, the Simon Erickson that's a term of offer that talks about a snapkin math word, like you was like, all right, here's the theoretical limitation of the hardware reach from SSD might be, I don't 30 microseconds, and then like, how can I build a system that is as close to this as possible, as opposed to get away around saying, okay, like, you know, human will notice like 20 milliseconds, like, under milliseconds, let's build around that.

Yeah, yeah, yeah. And sometimes like when you're architecting something, you need to think about the human kind of perceptible latencies, but oftentimes when you're dealing with the storage system layer, you know, I know Simon working on TurboPuffer, they're doing great stuff over there. You have to think about machine scale and machine speed, which is a lot faster than human perception. Like a human can tolerate maybe 100 milliseconds of delay, or if you're playing a game, maybe you need to have like, you know, frame rates of like every four milliseconds, but the machine wants much, much faster than that. Simon says, napkin math, I call this speed of light numbers, and like sometimes it's literally the speed of light bottlenecking you, the cross zone latencies, between zones, cross region latencies, is speed of light and fiber.

You want to hear a kind of crazy fact? I love your crazy facts. The fastest way to send capital data across the globe is to send into space. Is it because the speed of light is faster in vacuum? Quite a bit faster.

Or well, it's not fuel vacuum. No way. Yeah. So you cover a higher distance. Yeah, well, actually, I believe that the way to do this, you send it straight up.

And then with start, you send it straight up, you bounce around between start, and you send it down to the other side. So you want to get it out of the atmosphere as quickly as possible. But now, if we're ever telling a speed of light, this will also go, there's like a digital transformation happening. So you would need to calculate how long it takes for that system to process and do it. But you're saying that even if you do this really well, it will be faster than being able to get through an optical cable.

And optical cable slows down the speed of light, right? Yeah. Yeah. So it's only the speed of light in vacuum in every other medium, it's slower. I was talking, I did a deep dive on the hedge fund industry, and they didn't tell me exactly.

They said that they do use satellites and microwaves to some of these things. They will not tell you because this is their thing. But I had a suspicion that they might have found a faster way. And I think this is like some of them, but these are not going to get into because again, but like. Yeah.

So I'm about to bounce in space. Yeah. So one of the things that they do in the high frequency train is like, between New York and Chicago, it's actually not far enough distance wise to make it worthwhile to send it in space. So they were doing microwave beams there. Yeah.

But I mean, if you really want to get faster, you need to build like a vacuum to between them. It's like send, but maybe they're doing that out. Yeah. But I mean, I think they think it's just kind of awesome about performance nowadays is, I mean, there's so many layers you have to be paid attention to in terms of performance. The rabbit hole just goes so deep there.

You're almost certainly running on multi-threaded systems, right? And you're like, oh, I need to have a multi-threaded program. I need to have synchronization in there. Well, you get the best performance and just kind of avoid the synchronization. And part of this is locked for programming, but part of it is arranged so you don't need locks at all.

You need to carry about your processor caches. And there's this whole kind of setup of caches above the CPU. Like you can get registers of the cache, and you have L1, L2, L3, even your memory on the disk. And there's just got to pay attention to all those levels. And if you do, your performance gets way better.

And if you ignore it, you're like, oh, I'm not worrying about kind of the cache accesses. I'm just accessing data all over. Your program will just be way, way slower. And some folks pay a ton of attention to this. The high frequency trading, I mean, they do this all day long.

And so many other places, like no one pays attention to it. And you get this kind of gradual degradation of the performance of the hardware or the software. I didn't want to talk a bit more about low level stuff. It's not about the speed of light, but low level data structures and programming language features. You made some contributions to the standard library, right?

I liked a couple, not quite standard libraries. So I mean, I just like to have been always fascinated by data structures. It's kind of awesome. I mean, I think just the algorithms in general. And you're just like burning outcomes.

They're just kind of awesome. You could probably explain, you know, insertion sort and you know, kind of every. Could you explain quicksort as well? Quicksort's a little bit harder. I'm not sure.

Yeah, I know, but it's a smart one. Yeah. And then you get to these levels of like, you know, it's like, oh, wait, someone really smart came up with this. So one of the things I worked on just as a little bit of a side at Google at some point was a colleague came to me and he was like, you know what? We're using the STL map structure all over the place.

And the STL map structure is a balanced binary tree. I can't remember if it was red, black trees, or one of these other balancing algorithms that any CSCS college students implement. And he came to me and he's like, I think we'll do better, because you know, there's actually a cache problem here. Every time you're every node you're traversing down, you're going to a different cache line. He was thinking about using something else, a skip list to do this, which that's another awesome, awesome day to start your relationship and look at.

But at some point I was like, actually, this feels more like a B tree. So I'd implemented B trees a couple of times before and figured out like how to implement a B tree, that implemented almost all the semantics of the STL map. It couldn't quite do it perfectly. And the reason is when you insert into a B tree node, you have to shift around so you don't get point your stability. This is just kind of fundamental.

But if you don't need that for your use case, you actually can pack more data in. So the thing about B tree, like the real easy way to describe this is like, you just have a small list of items, like eight items. The best way to store that if you want to kind of fast access in sorted order is just sort of like literally not to have a tree at all for like eight items. Just to have it in the very simple list. Very, very, just an array, sort of array.

And then you can either do a linear scan over or you can do binary search. And oftentimes linear scan is faster. And then you think about that, I'm like, well, if I want to store more than eight items, I just have one node that has eight items and then I have another node and then you have a parent node that connects them together. It's essentially like the, you know, you think about building up on them up. You start with just one node of eight items.

Oh, I need to insert the ninth thing. I'll split into two chunks. And the two chunks, one will have four, the other will have five. And then you have a parent node that points to them. And then you just kind of recurse on that.

That's the B tree algorithm in that shell. Everybody go implement it. Actually, nobody should implement this anymore. Because nowadays we have something else that will implement this in all the optimizations. Because there's a crap ton of optimizations that you can do in a B tree.

But I just want to go back to this. Like there was already an existing implementation for maps. And then so your colleague looked at the code and said, I think we can do better. But what I want to figure out is in my mind, someone sitting outside of, you know, I'm not involved in how some of these libraries or data structures are built, I always thought, I think, this might be naive. But really smart people sit down.

They kind of look at the state of VR. They implement it. And there's no way it can be faster. In fact, I've had arguments in the past name. Like, oh, let's write a faster sorting thing.

Like, it's surely it is the fastest. But if you could bring us a little bit of like how, even if it happens and how other people like yourself and your colleague can say, like, oh, what if we would have tried something else? Yeah, yeah. So I mean, my recollection here is he was working on this kind of the big internal system, I think it was called Gaia, that actually had the mapping from, you know, you log in, you have your user ID, and you have to look this up. And they were storing, you know, all like this, the map from user ID and email to whatnot to the metadata about the user in SDO maps.

And he just knows, like, well, there's a lot of them where you sit here and it shows up on profiles. And then we're like, well, what can we do to do better? And that was kind of the genesis of it. And you know, he happened to be working on it and he happened and they were working with me. And like, we just start kind of newly on this problem.

Like, oh, can we do something better? And it's not one of these things like, I think for Google, they have a whole team working on their kind of internal libraries. At the time, it was more of like, you know, everybody working on their own systems and contributing to a shared base. But I guess it still goes back to what you were just saying of like, just go down to layers, try to understand it. If something just doesn't add up, like suddenly like, oh, there's this big explosion member usage, like, you know, just ask the questions, why is this?

And if you're able to or you happen to be like, oh, can we do something about it? And one thing that you know, he deserved it, we all know. I think part of one of the things was it was like a map from integer ID to something else. And like, if you look at red black tree, every node, you have your kind of value that you're showing the map and then you have two pointers. You might have an integer ID that's like four rapites and then two.

Yeah, and you look at that and you're like, that seems like a lot of overhead. You can just a bit. Well, the B tree actually has a lot better spatial locality. That's what made it faster. But it was actually smaller as well at the same time because you had less pointers involved.

You also got to be able to go, right? Yeah, well, that came later. Yeah, yeah, it came a lot later. We talked about that. Yeah, just one of these other things, you know, I pay attention to like, you know, when there's research papers coming out about new data structures and like hash tables.

Hash tables are like the one that are these things you learn about in college and data structures. Like how do I map keys to values where the ordering is unimportant? That's when hash tables come in. And there is like, you know, the very earliest ways to do this, I implement hash tables multiple times. It's like you take your key and might be a string and you put through a function and it's out an integer and then you map that into a array of buckets.

And if multiple things match the same bucket, you have to have a link with link with link. You have a link list. Yeah, this is a naive implementation. You have a link list. You have a link list.

You have a link list. This is a naive implementation. You have a link list. You have a link list. You have a link list.

You have a link list. You have a link list. You just kind of hash it again and move on or kind of walk down to subsequent buckets to find out like your subsequent slots to find out where you should be. And I remember reading about this new technique and came out of some folks at Google. I believe it came out of their Swiss office.

It was called Swiss tables. I believe that's where the naming came from. I'm not sure about that. I remember reading about it. And then I was working on a long period of time and go has this built-in map structure.

And it's a hash table. It's a very highly optimized hash table. Because the go team is very competent. The go run time team. And various folks have taken a template like, you know, putting together Swiss table implementation for go.

And I tested some of them and I was like, this is kind of fascinating what Swiss tables do and I'll explain how it works in just a second. But I would do this. Well, it's really hard to beat the performance of the runtime. The runtime was really good. And I kept on using this for a little while.

And eventually I ended up having to take this business trip to India, to Bangalore. And so I was like, no. Yeah, it always starts like this. And I'm just going to try to pull on this. I pull on it sufficiently that I can get some of the benchmarks to be faster.

And then I'm like, you know, that is kind of like catnip for an engineer, like kind of make it all faster. Figure all the rest of it. Got some help from the runtime folks. There's an issue on the go issue tracker that where other people are attempting this. Because people propose like, hey, let's use Swiss tables.

And like, we don't quite know all the details. We don't have to navigate this and that. And there's some ideas there that combined them together and got to the point where we had a complete implementation that was faster on most benchmarks, not quite all of them, but most of them. And then they go from essentially pick this up and push it over to finish line. And then so you came with the idea, you got to the point where you were able to show an implementation that showed how some of the benchmarks were faster.

And then you started to work with some folks on the go team too. Well, I didn't, I wasn't quite that. I came up with an implementation that we ended up using at my company. It was good for our use case. But actually putting it into the runtime.

It was a whole other level. But then you just showed like here as a implementation. And then they kind of took the inspiration and the ideas. They always were looking for ways to make a runtime faster. And they're just like, you know, oh, wait.

And then so you did this in this contribution. It was a lot of years after you left Google, right? Yeah. It was from the outside. This is awesome.

Yeah. I know that people contribute stuff to the outside as well. You know, we had another colleague he contributed one to CRC implementations, you know, adapting some stuff. CRC. We're done to see checksum.

You know, Intel published some papers about you. You're saying it's simply very fast and you contribute one of the applications. You see a number of those things where, you know, people just like are contributing externally. It's not a lot. Honestly, I mean, I actually don't know the full details, but you know, people regularly contributing to these things.

Peter just described how to Swiss table work came together using an issue tracker in the gold tracker with different people contributing to the work and then the gold team pushing all of this over the finish line. This is where I need to mention our season sponsor, Linear, which is a place to coordinate work between humans, as well as agents. One thing I've noticed about how most of us work with agents is how it's a pretty single player thing. You open a terminal UI, go back and forth with an agent, and it usually produces a PR. But the rest of your team has no idea what happened in that chat unless you tell them or copy the whole history.

And when everyone in a team works like this, a lot of work happens that invisible is the rest of the team. Linear's take is that agent work should be teamwork. Even today, teams already use linear to define the work to be done. Now, linear can already delegate an issue to a coding agent. This agent could be an AI agent that linear integrates with like codus accursor or linear's own agent or custom agent.

Either way, the engineer is delegating stays responsible for the outcome. What I really like about how linear works is how the work stays visible. Your teammates can follow the session of the agent, check out the PR producers, and join their view. We've gone from single player work to multiplayer engineering work with agents. Oh, and one more thing I like about linear, a focus on costs.

Linear agents on a routing chooses a model that is the best suited for the task. It seems can also inspect usage and set limits, track usage, so you can use capable agents without having costs balloon out of control. Hop on board at linear.app.pagmatic. Peter also previously mentioned Gaia, Google's internal system that map logins to users. It's not surprising that Google custom built all systems including this one, but most of us won't build our internal Gaia.

This brings us to our season sponsor, WorkOS. You can think of WorkOS as something like Gaia for the rest of us. Identity infrastructure, you otherwise spend quarters building yourself. WorkOS includes single sign-on, skin directory, sync audit log, role-based access control. Basically, everything a big customer security team asks for delivered is a handful of clean APIs.

It's how companies go from we have a login to we can also fortune 500 without standing up their own internal identity platform. And WorkOS is already building for the next version of the agent authorization problem. Their newest product is Airlock, the authorization layer for AI agents. Think about what happens when you hand an agent a task like clean up the sale opportunities in our pipeline. The last thing you want is for this thing to have standing permission to delete whatever it likes.

Airlock sits between your agent and the tools they call. It evaluates every request against the agent's intent and your rules, and then it allows it, denies it, or routes it to human for approval. You write down the policies in plain language, the agent never sees your credentials, and every call and verdict gets logged. It works with coding agents like Cloud Code and Codex, and with MCP gateways. So if you're working out how to let agents do real work and production without over-permissioning them, take a look at WorkOS Airlock at workOS.com slash Airlock.

And with this, let's get back to Peter and why he left Google after building Colossus. You're at Google. You're building Colossus distributed. The file system is you're at this point, probably working on probably the largest system on the planet, honestly. Why do you even consider leaving?

Yeah, yeah. Well, after Colossus, I kind of dabbled in this other project, Google goggles for a little while, the glass holes. Yeah. He keeps coming back to the idea, by the way. So, yeah, yeah, it's still here.

It seems like Google was early. Yeah, yeah. I think that was a technology for its time. I don't think it was ready to do that point. It looks like the actually doing the glasses is quite harder.

You know, the Android phones we were trying to power it on were not powerful enough. And then, you know, I just kind of got wander-lessed. You know, like, I'm just kind of stagnating here, which is a strange thing to say, but, you know, the model of people field as well. Inside, go off and try my hand and other startup. That didn't work out.

We got Aquahired by Swear. So, yeah, I want to pause for a second. So, this company, what was the company name? The company that we found is called Viewfinder. Viewfinder, yeah.

It was in the mobile photo sharing space, which it sounds familiar. This is like Instagram. This is like, it's not. This was in 2012. Yeah.

Right as Instagram and all the more we're taking on. Yeah, yeah. We were right there in the play. And we just didn't have the right go-to-market kind of like, how to track the users, how to get my growth kind of thing. Because from the outside, like what I read, when I checked, you know, the story, I just like, oh, you know, you co-founded a startup.

It got acquired by Square. Hooray. Like, it sounds like you had bigger ambitions. And this was a decent outcome, but not the dream. Yeah, yeah, it wasn't a dream at all.

So, the term I used was Aquahired. So, sometimes, a company will get acquired, get bought for, you know, their IP, for the product, for the business. And other times, they get bought just for the talent. The people, the people. And we got bought just for the talent.

They acquired the IP, but I don't think it's wherever to anything with it. It wasn't kind of like where they were working. But we built up a kind of strong technical team, and that's what we were hired for. And you know, like, I can't remember the details. We'd raised a small amount of money, we were able to pay our investors back, make them whole.

Maybe they got a little bit better, but maybe they actually got a little bit of a... But it was essentially they got our money back, which is like, you know, as a founder, you know, investors are big boys, they're used to losing their money, but you kind of feel bad if you lose a lot of money for them. So, you know, getting them paid back kind of makes you feel a little bit better. Yeah. And then you spend a little time at the company, like, where you were just square, and then you started itching a little bit again.

Yeah, yeah, yeah. Because, you know, we've been working on these, you know, distributed file systems and storage systems, you know, classes, one of the kind of sister projects to classes is Spanner. And how is Spanner different to Colossus? Well, Spanner is essentially a distributed database. Colossus is a distributed storage system.

And the way I think about the difference between distributed storage systems is distributed database. You might think about both storing data. I was about to ask you, because this will be a database will at some point be a storage system. Right, right. So what's the difference?

Yeah. So for Colossus, you know, it was targeting large files, large appendowing files. You can't update. Yeah, large appendowing files. So, you know, 64 megabytes, maybe up to gigabytes in size.

But if you're a database, you want to be storing like kind of small, like, you know, if you're using SQL or like the relational data, you might have table, you know, millions of billions of rows, those rows are broken up into columns. The columns are tight. That just has a very different nature to the engineering challenge for database than it does for a distributed storage system. And usually distributed databases are coming on top as some kind of distributed first system. And that was the relationship.

So Spanner was implemented on top of Colossus. And so the design decisions in Spanner kind of directly fell out of the appendonly nature of the files in Colossus. You can't update a file in place. So you have to make the files mutable in your database. And this is where like, you know, log structure and merge trees, you know, come into play.

And you weren't invented at Google, but Google really popularized them with level DB, which emerged out of the work on Bigtable and Spanner that popularized in Rock City. I subsequently re-implemented one of these things and this is what we use at Cockroach Labs. It's called Pebble. So I'm very familiar with the internals of that. It's kind of all based on this idea that the data is kind of stored in the mutable files.

So how did you decide to found Cockroach Labs? While we were working at Square, my co-founder and I, it was actually three of us. We were all at Square and one of them was Spencer. I mentioned earlier, he was working on the GIMP, he was a college roommate. And he was also Google.

He was also Google. Another guy, Ben Darnell, he was also Google. He joined us, a viewfinder and it up at Square. We, you know, like we're kind of just noodling on a project to do and we actually had this design back in viewfinder, like, ah, we didn't, we looked around for a database to be using in order to build viewfinder on top of. We didn't really like the things around there.

The technologies inside Google looked better. We had the Bigtable, it's Spanner and whatnot. We were looking around, you know, like, each base existed, but I wasn't quite happy. And there's some other systems like React and others. You know, at one point, we just kind of like, you know, came up with the design for Cockroach to be an initial design.

And I was like, no, guys, we're doing a mobile photo sharing site. We shouldn't build a database. So we put it on the back row. Which I think was absolutely the right thing. Maybe we should just pivot away from doing the mobile photo sharing site, given the way things work out.

And then we got to Square and we saw some of the same problems that they were experiencing with data systems. And Spencer is very convincing, managed to convince us on the management that, like, hey, you can just work on this part of time, you know, to see if it had life behind the design. And then he kind of conscripted Ben and I into it. And eventually started getting attention externally. And we're like, hey, can we go and spend this off in a company?

And that's what ended up happening. And so you started a company, but I understand you didn't raise VC funding initially, right? That was the case at Viewfinder at, we did differently. At Viewfinder, we kind of eschewed the VC money. And, you know, in hindsight, I wouldn't recommend that.

What? You wouldn't recommend? Yeah, I would recommend taking the VC money. Because my experience, the VCs are very, very intelligent. They can help you navigate a lot of challenges.

You know, I think sometimes there's this perception, you know, the VCs will push you into various areas. And maybe there's some bad ones out there that do. The VCs I've had experience with, just like some of the sharpest people, you know, I've ever met. And I've been mentoring you, given you guidance, telling you what they're seeing. They give you advice, seeing what they're seeing in the market, where things are going that is very hard for sometimes for founders.

Especially the technical founder, right? You're focused on the engineering part. Yeah, yeah. No, we actually took my right away for Cockroach Labs. It was almost like as soon as we left, we got a little road show, you know, kind of in the Bay Area.

And got some interest in, you know, got an investor right away. I have to ask about the name though. Yeah. Other name of Cockroach. Yeah, right.

So, you know, we named the GIMP. Yes. That was mine. Pulp Fiction came out in college and I'm like, which we named this thing? I think we're the Image Manipulation Program.

I think we're the Image Manipulation Program initially. And I'm like, oh, GIMP is obvious. And it just stuck. And at some point, you know, we're newly on the new database. And you're going to give things a name.

You can't just say, oh, we'll work on the stupid database. You kind of need something. And this is not just like, oh, Cockroach Eevee. Like Cockroach is runn' tillable. I want these things to be untillable.

Cockroach is going to survive. The nuclear apocalypse. So that was where the genesis was. And it just stuck. Yeah.

Another bunch of nodes goes down. This thing will still be up. Yeah. Yeah. And you know, this is where we're at today with Cockroach.

I'm like, oh, crap, this is awesome. You can kill a node. We did this whole campaign last year, which was really just to prove out something that already been present and the campaign was performed as a university, but just like you can run or work against it. You can kill a node. You can sometimes kill a whole region and the system keeps on going.

And it's like stories like that. You know, what we did on the working side there, but also we hear this from our customers too. They've had fires in data centers and all the other data systems go down and Cockroach dev keeps on going. I'm like, that's awesome. When you started out, who worked company startups that who wanted to use Cockroach dev and how has it changed since?

Because, you know, like just making the case, like, okay, I'm starting a startup, it's a small startup. I'm running it on node. At what point did you see that typical tech companies are like, okay, this is not enough for me that it's running on a node either because it can go down or because I'm out growing. What was it they are growing? I'm trying to get a sense of like at what point did companies say like, tell themselves, like, we need something distributed in the database.

Yeah. I mean, oftentimes we have companies calling us up after they've had a disaster. So, like a node went up or hard drive failed. It's not quite like we're ambulance users, but if you see an outage, like a big outage from some company, you know, like sometimes we trying to knock them up. But also they will call us, you know, I mean, like there's a very big bank who's now a customer.

Don't think I can name them, but you can go read. They had a very serious outage due to weather event. And after that, we should probably knock down. I'm assuming a region or database or an networking table or tree fell on something. I can't knock down a whole region, you know, the region wide power out, you knock down the region.

And there's a mandate from the CEO. It's like, no, we just have to, you know, be able to drive these things. That gets pushed down all the way. And you see this in other places where, you know, one of our early customers, they're running on AWS and they just got to the maximum size. You can run in a rural instance on.

And then what would typically happen at that point is then you have to charge your database. You know, it's a very standard practice. You take out your single note database, you create 10 or 20 or 100 charts. And this is what Google has done for some period of time. And that's a heavy burden on the application developer.

And the way we always phrase this is like, I mean, the application developer has become a database developer at that point. And they're doing it poorly, you know, they're trying to implement distributed transactions or indexes and whatnot. We've all been working for that along some of the database developer. Can we talk about automatic charting? I think it's safe to some, most of us will know what charting is when you're, but actually let's start from like manual charting and then how you can implement automatic charting.

And if you can tell us like, you know, tactics that a database like CockroachDB can do to actually just take that low off. Yeah. Yeah. So I think the very basic form of charting is a little bit like the hash table. Let's say, you know, you know, you have a fixed number of shards, like, let's say just the hundred charts.

Your data model is a user with a lot of data to say what the user is taking a user and you say like, Oh, they map into one of the shards and you know, you kind of just rely on the hash function to get like fairly even distribution that the problem with this is at some point, you know, one of your shards will get full and you have to kind of of re-charge. That's a very, very owner's project. I'm sorry, re-sharding it. It's just a hard drive. I don't know per node where you write the user data gets, it gets full and you're like, okay, well, I now need to split it somehow.

I need to move it. I need to remap it. I need to read my metadata, which knows where the data lives, that kind of stuff. Yeah. Yeah.

And depends on exactly how you're doing that mapping from like the user idea or whatever your shard key is to the shard, you might have to remap them all. Right. This is very typical. So this happens in hash tables where oftentimes in order to grow the hash table, you just have to essentially create new hash table with double size and copy all the data over. Now, that's kind of like very basic, straightforward way.

And there's various levels of complexity on it. And there's various techniques to do this. It's kind of fascinating like how they all work, but in system hashing, you can add additional node and then it only moves a fraction of the data from each shard over there. There's various systems to do that. And I believe this is like when it lies Cassandra, the way CockroachDB does it is more, can the big table more can expand or more can the H base where instead of actually hashing, you actually take, you can imagine all your keys in a system.

And this is always true in a system. You can imagine this one big continuous key space and then you kind of partition continuous spans of that and then you have to build up an index on top of those continuous spans. And what I just described there actually sounds a lot like a B tree. So there's this index on top that is like that maps you from, you know, like, I need to have this range, which node is it on? And this is a little bit like a B tree.

You know, you kind of squint, you know, it's like, I think you squint and everything's either a B tree or hash table and, you know, that index structure. But it's now I'm starting to make sense because when I remember when I read it might have been the Wikipedia article on B trees, it said this is data structure that is frequently used in databases. It's not going back to me because I didn't think much of it. I'm not some of those databases. But now that we're talking, we just like organically keep touching on these trees again and again.

Yeah. And the other place that comes up in databases, so this is where it kind of comes up in distributed databases and no one ever really calls it a B tree. I just kind of squint sometimes and I see like actually kind of a B tree. But the other place that comes up in databases is for your indexes. So if you have a table and you have like an index on your email address and you want to be able to scan over those email addresses in order, that's a B tree under the hood in a database.

And, you know, like any kind of index you have usually provides sorted order or hash indexes, but oftentimes it's a B tree index and they're ubiquitous in databases. There's actually a paper called the ubiquitous B tree and, you know, basically, I just identified that. I think that paper is written back in the 80s and they're still ubiquitous today. They're the foundations of single node databases and pretty much every data system I worked on is had B trees at some point in placing them. I want to ask about strong consistency.

So, CockroachDB offers strong consistency. Now for people who are a bit more newcomer to distributed systems, can we talk about the consistency models and then why strong consistency is important and why it's hard to implement it? So, I mean, there's multiple ways to kind of approach this, but I mean, if you use the database, you can't afford the transactions. And transactions are a way to perform a whole bunch of mutations atomically. So, databases talk about atomicity, consistency, isolation, durability.

The durability is really easy. It's like when I write the database, it has to be durability written. So if anything crashes, it comes back. The atomicity is just referring to the fact I want to do a whole bunch of changes. I want them all committed or all aborted at the same time.

I don't want to have like some kind of partial operation. And why is this important? Why is the atomicity important? Well, the atomicity is what gets you to the point where it's like as an application, I can do a bunch of operations and if there's an error, an error occurs, it all kind of gets rolled back and it's a much simpler development model to work within. And then there's the consistency in isolation, which kind of get, you know, kind of muddled.

The isolation is referring to isolation between transactions. I don't just want to run one transaction at a time. That's easy to do. Right. I want to run a lot of parallel and make it so that when they're running in parallel, they are running as concurrently as possible.

But you want to have the appearance when they're running as concurrently as possible, that there is kind of a serial order to them. But this is like kind of like the whole track, that kind of the gold standard for isolation is called linearizability. Don't worry about that. There's a step down from that is called serializability. And that literally refers time to serial order of your transactions.

But you're having to construct that in a way that you're doing everything as concurrently as possible. And the benefit of this, the serializability is again, it's a very simple model for the application program. They don't have to worry about weird kind of defects occurring in their program. And some of the ones that can occur, it's like the classic description is of a bank, right, where I might want to read, you know, like have an operation that reads and says like, do I have $100 in my bank account to transfer somewhere else? And you could like arrange for less isolation levels that you might be able to subtract that $100 twice.

And that's bad, right? You know, you want to keep accurate track of your bank account or, you know, what's in your shopping cart or, you know, it's kind of like the use cases are endless there. And you do this wrong and you have very egregious bugs. But now going back to, to weak consistency and strong consistency. Anything less than linearizability or serializability might be considered kind of weak consistency, but there's also like, you know, kind of strong consistency and eventual consistency.

So the eventual consistency is like sometimes I can do an operation and it might not be, immediately, I might not be able to see all the updates. The read result will not necessarily be the current update. But they'll come back, you know, at some point, you know, I've written part of it and the rest of it will show up at some point. And often it's the easiest thing is your credit card balance, right? Yeah. Yeah.

And it's faster to do it that way. Faster in terms of just what the performance you can get out of the system. But again, it's a little bit harder for the application to deal with. One of the places this often comes up in, in distributed databases or databases that have any sort of replication, is that I can write to the primary replica and then read from the secondary and stuff there. Yeah. This is not the eventual consistency.

Yeah. That's eventual consistency. And you can't work around this, but it just like puts bigger burden on the application. Because you have to pay attention to that. And then with talk or DB, you have strong consistency meaning as soon as you're writing it, when you're reading from the database, you already get the written value back.

You get the written value back, you know, it's like you read whatever you wrote. It doesn't matter if you're reading from the same, you know, node you wrote it to if you read it from another node, you still actually get the data you just wrote. Is the trade off logically not that you would have higher latency because clearly to implement strong consistency, you would somehow need to in the naive approach, you would need to write all replicas, right? Yeah. What are you doing inside of our code?

Yeah. Well, we are writing all the replicas. So, well, yeah. Yeah. Yeah. You're just doing it fast.

You're just doing it fast and making that efficient. I think this is one of the things that kind of also fascinates me about the software industry is we keep on finding ways to be more and more sophisticated in order to provide, you know, like do things that make it easier to write the applications, but do it in a very high performance way. And we've got, you know, very, very good at this over the years. And this is the area I know about the databases. This is happening everywhere.

Like I'm just fascinated by how fast graphics have gotten. Where would I enter the industry? Like you were literally like running out each individual pixel. And now you have these GPUs that are doing like billions of triangles per second, whatever current numbers are. And I'm just like kind of standing like how much sophistication has gotten into every area of computer science, where you look at it.

One more thing on Cockroach DB, I want to talk about is the rough consensus. What is the rough consensus? Yeah, I mean, consensus protocols, the original consensus protocols called Paxos. And it was famously hard to implement. Raft was, you might think it was a variant of Paxos.

It was kind of like an alternative to Paxos. But in my mind today, and then because this is algorithm being that you have like a number of nodes, like data, a lot more. And then how do you get them to agree on? We'll do it typically agree on. Yeah.

So you agree that the writer occurred. So like the way to write a current consensus. So you might think I want to replicate data and I write it to a primary and write it to a secondary. You can't actually have consensus when you only have two replicas. And the reason you can't have consensus is if there's a crash and I come up like if I'm on the secondary, how do I know if something was written to the primary?

If I'm on the primary, how do I know it was written on the secondary? Right? And you're always going to be in this kind of confusing place where it's like you either have to roll back a little bit or like, you know, you lose some data. And consensus requires at least three, but you can have consensus across more than three replicas and the idea with consensus is I'm going to write to three places. And normally you don't actually, when you're doing a read, you don't read from multiple of them.

But if there's a crash and you do recovery, then I'm reading, oh, I can read from two of the three. And I know I can kind of determine what had happened previously. And it's usually just on the recovery time that you're actually doing that consensus read. So reads are typically just happening from one replica. You have to write to all three and on a crash through that kind of failure is when the consensus read occurs.

And inside Cockroach, do you have any replicas? Do you choose for your consensus? It's typically three. For some, some tables can be five and then customers also have control of this. At a database level, you can write to five, seven, five is like, you know, if you're really concerned about the data durability, you might use five.

But there's a slowdown, you know, the more you're running to, it's like the more storage space that we're talking about. We're talking like off the side with nodes, but of course, if they're like between regions, there's now you have a lot more resilience for like, say, an earthquake or power hours or whatever, but now you will have additional latency, just speed of light to basics, right? Speed of light latency, right? And it's, you know, tens, you know, about hundreds of milliseconds or even higher if you're going across the globe. So, you know, you have to be very careful with that in terms of how you architect your queries.

And one of the things that is just a general truth with distributed databases is you don't want to have a lot of back and forth. You want to kind of like do all your reads and one kind of parallel read set, get them back, then do your writes, right? But if you're doing like kind of serial operations, where I read a row, I read a row, I read a row, I read a row, I read a row, I read a row, I read a row, I read a row. I mean, the link sees just add up. We talked about founding CockroachDB, but how is the company grown in?

Where are you today? Yeah, yeah. I mean, we're being used in like, we're powering mission critical applications. That's our bread and butter. Can you elaborate on mission critical?

Because it's like, yeah, if you're not, you are in the industry where you know what this means, but from the outside, it can feel hard to put it on. What is mission critical? Is my SAS that is like showing us a mission critical? Probably not. Yeah. So mission critical in my mind is like these kind of the other term of art is tier zero applications.

It's the ones that are like just the core crown jewels of what's running a company, like a trading system. You know, your trading system can't go down. The trading system goes down. This is kind of a critical problem. Or the firm is running the trading system, you know, banking systems.

But also, you know, like we work with DoorDash, you know, like some people might think that delivering to a breeder was kind of a mission critical system. And certainly is for DoorDash, right? You know, if that goes down, it's problematic. We power shopping carts. You know, others like this where it's like, well, if the shopping cart goes down, you know, that company is losing, you know, hundreds of thousands, millions of dollars per hour.

So that's kind of the critical thing about it. Yeah, I guess of course that they're losing, but this is like when the other customers are also like, they're used to this just working like running water. And when it's not the same thing as when your till you broke breaks, right? Your water else just is out. You'll survive, but it's not what you expected.

Yeah. Yeah. Yeah. And everybody's like, what age are we living in? The electricity goes out.

We're kind of this ingrained in our heads at Google. It's like, Gmail cannot go down. We're going to move to other systems. And it's like, you know, like, in some ways, search isn't as much critical, except, oh, wait, every single search is that dollars behind it. And you actually notice the blip in the revenue.

And it's not just the blip in the revenue. It's the blip in the reputation as well. I mean, I think that's the one that really poisons companies is like, if your bank is down for a serious amount of time, the reputational damage there will be horrifically bad. And, you know, like, we often talk about like, oh, then grandma won't be able to pay her rent and she'll get a victory. You have to take this like responsibility really, really seriously.

No, but also like just Gmail be mission critical just on the way here. We only exchanged numbers later, but we were communicating over email like, oh, I was telling you that you were telling me that you're here at email. And I never for a second thought that it could go down. And I think we were like responding within 30 seconds, right? And it's just, I just know it's there.

Like I didn't bother setting up a secondary communication line. So yeah, yeah. It's like when you have that kind of level of trust with your users, you got to maintain it and invest in it. But then it leads to this kind of freedom for the user as well, where you just don't think about it. Like, I don't have to worry about it.

It's just going to work. And then in terms of the company, like how many engineers do you have roughly? We have some hundreds. I don't actually know the precise engineering number 110, but there might be 150 in R&D overall. There's other folks besides just engineers.

I'm a clear engineer, I'm an engineer, I'm an engineer as well. And then you know, like we've been grown steadily. It takes quite a while to build distributed database. Not for faint of heart. So it takes a couple of years.

You've done it a couple of times. Yeah. Yeah. Well, I did distribute a storage system. I did a distributed database.

It's not for the thing apart, right? So there's a lot of work getting it to a level of stability and then a level of quality beyond that level stability being all the bugs out. And then you can put more performance into the system, adding functionality to integrate better and within enterprises. A revenue has been kind of steadily growing over the years. And it's at this place now where we see the past of future success as well.

And I want to ask about your coding habits. So when you've co-founded the company, how much code did you write for the first few years? A lot. So I've always been a very prolific coder. I wrote a lot of code early days.

And early days, I mean, I was kind of, we were all technical co-founders, Ben Spencer and I. And we were already on a lot of code. And I always know exception. But you know, if I look back at my GitHub output and it was like peak years, maybe a hundred thousand lines of code in a year, which is a lot. Yeah.

Yeah. So I'm like, we're on PreAI. PreAI, right? This is when you're back to when this mainly, right? You know, at some point, you know, we started out using a system called RocksDB, which is an all-a-some.

At some point, I think it's back in 2019. You ran the limitations with it. I decided we want to rewrite it and to big push to, you know, write, I might have been like 40, 50,000 lines of code and then a bunch of other people have come up and helped. And like, you kind of look at that output. I'm just like, oh my goodness, that was a lot to keep in your head.

It's a lot just a type. Yeah. A hundred thousand lines code. The average kind of like the industry talks about is three thousand lines of code from an engineer in a month. And so if you multiply that out, maybe 36,000 in a year, that's good, right?

So I was doing a lot. I kind of look at that. It's like, there's kind of a max that you can hold in your head at a time. The tool's got a lot better since I first entered the industry. We got better debugging techniques, better testing techniques, but still quite significant.

And you were CTO from the beginning, co-founder of CTO, but there was a time, sometimes around like 2022 when you decided to kind of be a bit more hands off, right? Yeah. Can you tell me about that? I mean, the general rule of thumb for engineering leaders is, well, you got to have your team, you got to manage your team, and we had a VP of engineering. But I was kind of getting to the point of like, okay, is my co-indays done, you know, like I just direct from a higher level.

And, you know, I got this advice from hard-coded time and I pushed back on it, but, you know, I kind of acquiesced at some point. And I think it was right, but I'm not saying that the advice was wrong at the time, but there was time for about 2022 to 2024, whereas like my output declined. I think I actually did the Swiss table thing in that time period, but I wasn't doing much on the core of the business. Yeah, the core of business. You know, I would get in there and do some work, but it's really hard that if your meetings all day tells you to do code.

I think this is the fundamental tension. So you kind of took on the kind of the meeting bird and the coordination bird and the stuff that was, if I read it correctly, before you spent a lot of your head in the code, and now you're spending a lot of your head like above the code, the business, engineering, or whatever customers, that kind of stuff. The customers and just being executive as well. Yeah. So very hard where all this had simultaneously. And then I got back into it because AI started to emerge.

So how, when did you get, when did you start using AI, when did you start to find it useful in terms of coding? Yeah, it was interesting because you know, those initial versions of like, how glorified auto-complete came out and we were talking about the GitHub compiler, the gurster. That was when I had the first exposure to you, we'd have a cursor at the time, but they're all like glorified auto-complete. And it was kind of crazy that you could start typing something like, the rest of the function. You look at it, and you're like, wait, I kind of got that right.

This is crazy. Right? And we're trying to encourage our engineers to use this. And at some point, you know, I can't remember if this is my idea or my co-founders or someone who basically, like, you know, in order to really guide people about how to use it, you have to be easier for yourself. You know, I think this is true of engineering management in general. You want to manage engineers, you have to know how to be an engineer.

If you don't know how to be a good engineer, it's really hard to manage other engineers. I feel like you will have a hard time just relating to them at the very least. Exactly. Exactly. So it kind of took on me, like, no, I mean, this is like, it was clear very early on, like, this is probably going to go somewhere.

It wasn't quite clear how far, how fast it would go. And you start dabbling this and I was like, oh, okay, well, it's not quite good enough. It's not quite good enough, but, you know, I'm starting back into the coding very rapidly. You know, you start seeing the signs of life, like, you know, kind of the opus model is coming out. It was on it first, then the opus, and you're like, you're looking at these and I'm like, oh, wow, okay.

Well, they seem to be able to do quite a lot, but, you know, code is still not great. But then it was just like the just this cadence of continued improvements. And, you know, I was starting to a lot with sonnet and then starting to use opus and, you know, I had that same moment everybody else did. And this was last year, last year, November, November, December, or winter break. Yeah, it was Thanksgiving. I distinctly remember it because, you know, you were not chilling. You were coding.

You were agenting. I was agenting just like everyone else. I had this thing I'd wanted to do for a long period of time on Cockroach TV, which is like, I wanted to like test like all these configurations of Cockroach TV across, you know, different vertical scaling, like how many CPUs you have on a node, how many stores you have on a node, how many nodes you have on the system and test across all this huge matrix of it is one of these things I couldn't ever quite get prioritized to predict because it never seemed like critical, but I've always had this intuition. There was something there. And then over this like four days span, the code just like materialized as I was using.

I think that was opus 47 or four five, whatever the number was, like the recollection I had is, you know, like a pretty fast type. And then I just remember having this feeling of like, well, the code is just like materializing before my eyes. You know, you would ask for these things. I got out of the habit of actually typing. You just kind of like ask for something materialized.

If you've ever seen like some of those, you know, a movie where they have this kind of archetype of a software in here, gets in front of the keyboard and they start typing, show the screen, it's just like, it's going way beyond human speed. You know, it was, it was, it was, you know, that, right? Like, I think the software engineers like we used to laugh at these. Like, I still remember Swartfish, the thing when they're visualizing, the things are going forward or the code appearing. You know, you're seeing that the person's typing and then it's a big line.

And then a software engineer is like, we're laughing because it's not how it is, but it's crazy that like that effect, right? Yeah. Yeah. But now it's even better than that effect, right? It actually works this time. And it's that slow in comparison. It materializes faster than that.

Like you can literally go like we've been talking about B trees a whole bunch. I implemented another B tree in the past months. Yeah. And it took about 30 minutes to implement probably 10,000 minds of highly optimized rest. I mean, it's just like it follows mine.

I mean, we should look up later 10,000 minds. It's like a crazy amount. You physically cannot type that fast. You started getting back to coding. Like, are we talking about kind of vibe coding prototyping or are we actually talking, you started to contribute like proper production ready code that is up to the next?

Yeah. Well, it started with this tool that was like this kind of benchmarking tool that tested this matrix. I have a CTO. I have an office to the CTO. The office to the CTO is to innovate. And I was looking for places like we can have innovation. And one of the things that we want to innovate in was better auto scaling of a copper TV cluster.

And under the January timeframe, I came up with like, what I would think is like kind of a research breakthrough, you might say, and it came about because I was badly in this area and just working with models trying to understand it. And I'm not just going to coding. They're also good at helping you explore design ideas. And I think this is kind of the fascinating thing where it's yet to get out of the minds. I know exactly where I'm going to build, but more like, hey, we have this problem. Talk to a partner with me. And it'll be a sparring partner.

It'll be a sparring partner. You know, like, there's this advice you might have heard that like, you know, if you're stuck on a problem, you should go yellow, okay, go rubber, rubber, rubber, I think I've heard his yellow. You know, just go talk to something. You don't even need to respond. Now you can talk to the system, this intelligence, and it will give you back stuff. And it's not always right.

I mean, this is the thing, even today, these models are fantastic. Fables, fantastic, Astra is fantastic. And they're not always right. But they kind of like, they know so much. It's just encyclopedic.

And you can explore ideas super, super fast. And then we point out, well, that doesn't sound right to me. And you'll be like, oh, yeah, you're absolutely right. You know, like, I hate that sixth. And see, it kills me.

You're right. So I'm back on that. That's a new absolute right. Yeah. And it just able to make such fast progress. And I find it absolutely incredible. So quickly moved from just doing kind of side things, building up some tools and whatnot to building, you know, essentially over the last eight months, it's January.

We've been building towards a, you know, a new launch. And, you know, like a lot of that has been produced and gently powered engineers and like the entire company's gone on board with this now where it's like probably 100% engineers are using it to a greater or lesser degree. And a lot of code is being produced. I think it's very high quality code. These models, I'm not testing adequately. They don't look quite intently enough about performance. But you're like, you can guide them the right way.

And I think there's actually a huge advantage to anybody who's done management before. It's a little bit like, you know, manager of people where you're a manager of a large group. You're not looking to reline a code, but you're definitely kind of helping architect the system. I think there's a very. I sometimes push back on this analogy. Reasoning I wasn't an engineer manager. My take is that working with these agents. It's not like management because management has so much of the human stuff that like, as a manager, I think of all this, all this stuff. I dealt with which was the people side of things, the conflicts between people, the performance reviews, the meetings, et cetera. And you have none of that. You do have the orchestration. Like, and this is like almost like, I guess a really, like naive way of management where like, you know, they don't push back. They start to do it. Sometimes they're like unreliable, but you know, I almost like to use like orchestration a bit more because I feel management is so much more involved. Like, I think this is a thing where like the tech lead who has no management responsibilities, but they have a group of interns, but they don't need to do with their performance with their anything.

Like it's a lot closer to that, if you know what I mean. Yeah, I do agree. I use the engineering manager shorthand. But it's really about being a tech lead for like a 30 or 40 person organization, you know, or being an architect for what I think the term architect, excuse me, a little bit of a distaste. But if you know architect, he's also on the ground hands on architect hands on architect and you don't have any management stuff, which is a blessing and curse, but it's kind of remarkable that you can spin these things up.

If they make a mistake, you can keep on cracking them until they get it right. We've always been able to do that on the human side in action to a faster. I think the ultimate result for me is you just have to be more ambitious about everything you do. You can produce more higher performance, higher quality, more secure. So our ambitions have to raise up. You also mentioned with AI, like you're ever using and you're building stuff, but also with Cockroach TV, you're now building something that is also related to AI. Can you talk about that?

Yeah, yeah. We're building multiple things. I mean, yeah, it's the future. It's here to say. Yeah, I mean, like one of our PCs, which is not crazy. Every application in the future is going to be written by AI.

You know, I think there will be some kind of bespoke software, you know, handcrafted software. We'll see that. Can you do exist just as people, right? Assembly still, you know, but it's going to be the machine size. So, you know, asymptotically approaching 100% of software will be written by AI generated, right? Yeah.

It's hard to say if like when it's going to end where the humans are the agent, you know, kind of like spine agency and the vision behind it. I think that might exist for many, many years, but like I think the code will like fundamentally be written by the AI. And I think we're going to just see this explosion of applications. And we're seeing that inside Cockroach Times or seen it elsewhere. Earlier this year, we can roll out this internal platform where non-engineers write kind of many applications.

I mean, it's like you're sharing this in other companies. We did the same thing. And over the course of just a couple months, you know, 500 applications, 1,000 applications by non-engineers. Like non-engineers, primarily by non-engineers. And I thought it was awesome. Like our HR team is building like these little applications.

Like, this is the thing I've always dreamed about doing. And like, they were never serviced. Like your CFO is all over the place. And ours is no exception. Producing dashboards that they can never created before. And I think it's very empowering.

I'm married. I have a wife. She needs software. She cannot produce that software on her own. And I've never actually helped to produce the software, which is my own failing.

But I think there's a world in the future where she gets everyone's game custom software built for them. And you're just going to also see greater and greater applications and higher quality systems being produced by every company as well. Today, what is your stack? What do you work with in terms of harness model, how you run agents as one agent? Is it multiple? What can terminal do you use?

Yeah, yeah. It's evolved over time. So, you know, like when I first started dabbling the AI stuff again, it was GitHub co-pilot. It was an Emacs user for like 20 something years. I convinced the move to VS code. That's all gone now. At some point, I moved to using quad code.

That was the thing that whatever reason I just got started using it was quad code in the terminal of late. I used a mixture. So I'll just grab a current setup. I use the Cloud desktop app. Cloud code via the Cloud desktop app. It's fantastic. Good job.

I also sometimes use the codec desktop app. Just so I have an alternative model to turn to for some very critical things we're working on. I will get one of those models, sometimes like the best model. Oftentimes I'm using the best model like Fable. I've recently started using that sometimes as Astra, sometimes as the other ones. But you have one pretty sub sign.

You have the other one being like, hey, my colleague produced this. Can you tear it up? I've already reviewed it. It's not always perfect. But I think there is utility, especially for something that's super super critical. We're pushing the words on launch really soon.

And rushing to work with the finish line. I'm telling folks on the team, especially our very senior folks, like, you should use the best model right now. It's worthwhile to do that. I'm generally just using the best model. In the part that we say, I don't actually know, that I'm getting a lot more intelligence from it. But I don't want to have the cognitive overhead of deciding on case-by-case basis.

Should I use Sonic? Should I use Opus? Should I use Fable? Should I use Solar Astra? And I think maybe if you really need the speed, I wouldn't make that decision. But oftentimes I'm doing things in parallel. So you're asking how many agents I'm spinning up?

Well, it depends. Oftentimes there's a certain number of sessions you might be using. I find cognitive overhead is about five to ten sessions concurrently. But sometimes, some sessions will have many sub-agents doing things. And it'll depend on what I'm doing.

So, as I was coming here, I was on the train. I spun up something to do a little bit of research. And that was just probably using three sub-agents. Other times, it might be 20 sub-agents, you know, with an anthropic heads-dynamic workflows. I can't remember what the codecs thing is called. But they have various ways of spinning up kind of graphs of agents and prosecuting them.

I mean, I think sometimes I'm probably using 100 sub-agents. Others have something like five, and sometimes I'm not doing any of that agent stuff because I'm often thinking about something else. Since you were an EMAC Studio for 20 years using the terminal, right, well, EMACs. But how come you moved to graphical interface with agents? A lot of people still use terminal UIs.

So, I moved from EMACs to VS Code. I was just like, a colleague was like, hey, these ides are really good. You get to try them out. I was like, EMACs is an ID. And then I moved. And some of the stuff is just more seamless.

And EMACs keep on catching up. It always feels a little bit six months through your behind. And every time they have a new version, my setup would break. And I just got tired of that. So I moved to VS Code.

But then I was like, I have many terminals open in VS Code. My setup would always be terminals in VS Code tabs. The terminals in tabs have eight of them open. And then they start taking over. It used to be just clod code in the terminals.

Yeah. And at some point recently, it was like six weeks ago, a colleague was like, well, if you tried out the clod desktop app recently, it's really good. I was like, yeah, no, no, I'm very, very happy. And I made the switch and it was a little bit awkward at first. And then suddenly I'm like, holy crap, this is awesome.

You can like manage the agents a bit better easier. Yeah. And the sessions there, the sessions are down the side, you know, it's like they're kind of like tabs in some regard. And you ship it to the whole integration that Antropics have been doing. We did open eyes doing the same thing now.

By the way, like I'm not just an cursor and factory in condition. They're all pushing the same thing. It's incredible how fast these systems are innovating right now and evolving. What do you think good software engineering looks today compared to like four years ago before we had it? Has it changed? Has it changed or does not really change?

Yeah. I think the ambition has to increase, you know, I think quality has to be higher, security has to be higher. I'm glad you mentioned quality. Can we talk about it? Because I'm seeing across the industry just quality declines, which you cannot fully put your finger on AI.

But oftentimes it is people pushing up more and more and just not paying attention to just small regressions here and there. Again, like you're building a state of history. Have you noticed any or have you gone and feedback on any quality regressions? Or if not, like how come? Because like when, like this is just basic, you know, like we talk about the law of physics.

This is an observed thing that when you start to have more output, you increase your deployment frequency. You often, not always, but you often have like more regressions and more bugs. If you produce a certain number of blinds code, you're probably going to have a certain number of defects per line of code. And now you can produce more blinds or codes so you probably would have more defects, right? But the thing that pushes against that is that you can be telling these agents, you have to give them kind of firm hand at this.

This is the thing that I hope the model providers are listening to. You need to write up for him. They kind of get a little bit lazy on the testing side. And you have to make sure the tests are comprehensive, but also they're using all the testing techniques. And there's a lot of testing techniques out there. We have a lot of knowledge about how to do testing well.

And you know what, the agents are lazy. Humans are a little bit lazy. So humans to actually be very disciplined about their testing is also challenging. It's actually easier with agents. You know, you can kind of instill that. You can get them set up.

You have to give them the guidance. Use property based testing. Use metamorphic testing. Use like these advanced testing techniques. Deterministic simulation testing. You know, there's like technique after technique you can use.

And humans are always like, I'm lazy myself. Like I'm pretty good about being disciplined about testing it. At some point, you're just like, okay, that was not a good ship. And like now you can get a little bit like, you know, a little bit stricter and firmer. And the same thing applies over on security side, the performance side. I mean, we've always had at Cockroach in the history.

We know security coding practices. You can literally have every single line of code on every commit reviewed by a security expert. Doing like an adversarial security review by an agent or multiple agents. Or multiple agents, right? We're going through a tough time in the industry right now with like kind of hacks and security leaks.

And we're not. There's only a little bit of a number of bugs that can be in software. I think we'll get it added on the security side and on the quality side. And then also just on the part, I say quality such as bugs, but it's also like, you know, just a little thing. It's like, oh, that UX element is wrong. And there's no excuse for that right now. Like fixing it is so, so easy.

And what we're also starting to see of all this like, you know, designers using Figma. Well, I think that should be a thing of the past right now. Our designer just deals with HTML and CSS and JavaScript directly. And sometimes it's even just producing pull requests, you know, PRs, which she loves. We all love everybody's happy with this. There's none of this waterfall.

So she's producing pull requests for the production code base? Yeah. Yeah. You know, this is on UX side. Not on the core data.

Of course, but this is the area that she owns, right? Yeah. Yeah. It's just, I mean, she loves it. Everybody loves it. I mean, there's no downside.

Yeah. This is not new. Yeah. For sure. Yeah. What about code review? What's your take on code review?

It's starting to get a bit controversial. Like, is it going to stay or not? Because it's been a practice that's been around like, think about it. You worked at Google. Google has been so big on code review. As I understand there, there are two code reviews, right?

The domain experts reviews and then there's language experts reviews. So you went through that. How did you think of it and how are you thinking about now? I used to be part of the, I was one of the early members of the C++ cutting. I'm asking, right? Okay. You will have strong opinions. Tell me.

Yeah. I mean, I see that code review, I think, was fairly essential before the age of AI. But now we're getting to the point where I'm reviewing and looking at code. I look at most of the code that the agent's producing still. And yet, it's like you're not getting at the same level.

Right? And this is always in the case. If you get a pull request from your engineer, you have to get more screen. And if you get from your senior engineer and you ask any tech lead, any engineer manager or any software, they'll say, yeah, yeah. So the junior engineer or someone new to the code base, you know, you just have to get more screen.

And when I find you as the agent, you're having to give less and less screen. You do stop to give some things. It's like, it's like the testing will be incomplete. Maybe they didn't follow the security stuff. Maybe, you know, like there was a performance rush.

And I think it's like, you know, it's showing there was kind of constraints on the system. So it's hard for them to do the wrong thing. I suspect that, you know, I don't know if it's going to be this year or next, that we're materially going to stop looking at the code in the same way that we don't look at assembly anymore. Like, you will. We trust the compiler to produce pretty good assembly.

We trust the compiler to produce good assembly. Unless you're one of those people where it's really important, you're maybe a game developer or someone and you look at it and but there's two of you are those folks. Yeah. And even then, I think what will be migrating to is why does the human have to look at the assembly? The AI should look at the assembly. Like I'm doing something right now where I want to have a zero overhead abstraction,

you know, for doing something that is done test time and what is in production is compiled out. They will set up from your little system where you actually looks at the decompiled code to verify that there's only a few action instructions that's been in place. I never would have put that in place, but now it has this guardrail. There are every time it changes this, it can verify that there's no regression. And I think you're going to see more and more of this where you kind of put guardrails in place.

I think the model kind of likes it because it can work within that guardrail. For like 20 plus years, you were writing so much code. I'm sure you were in the zone. You remember being in the zone and just turning it out. You wrote a lot of like production ready code. Now that you're coding with AI.

Like, do you get into the zone? Yeah. Absolutely. And how is that zone? Is it the same as a different?

It feels a little different. It feels a little bit less intense, but you're managing more things cognitively. Can you describe me? Like, what is it right now when you're in the zone? Yeah. Well, I'm thinking of ideas that normally would have taken me a week to experiment with.

And I think multiple of these experiments and I fired them off simultaneously. And I'm kind of like reviewing, like, what else should I do while that's being complete. Sometimes I'm reviewing, like, they'll be giving me progress updates. Like, oh, hey, this is coming in. We're seeing this stuff.

And I'm like, well, that doesn't sound right. Hey, what about this? Or do we do this correctly? You know, maybe I had design and it's not implementing the design quite perfectly. But I have this little feeling that, you know, I haven't been a college professor, but maybe I'm a, you know, I was a college professor. I had a whole swarm of research assistants and they're all off doing things.

And it's coming back, just really rapidly. You're not waiting months or weeks. I'm not waiting months or weeks. And then I'm iterating. Like, oh, that one failed. That's fine. You know, you just got to let go.

And this is the nature of the software. I build. I think there's other pieces where it's like you just whip out a website. You can whip out something that doesn't have this level of kind of scrutiny, you know, very quickly. But I talked about this in a whole bunch of analogies about, like, the future.

Let's talk about it. Just do you have about using AI, right? I mean, the one I was advocating for, I had kind of chewed that I was advocating for, like, late last year and this year, which is, you know, AI is coming for us. It's here, right? And it's like, you know, you're producing software, like walking down the road.

And sometimes, you know, some will pass either running. They got an efficient gate and whatnot. But everyone's in their own locomotion. And these, these agenti coding agents came and it was like a car pulled up next to you. And again, the car, you don't have to drive. You don't understand the controls. You got to get in.

You start getting in the gas pedal and it takes off and crashes into a tree. But you have to learn how to drive. You know, I think using all these coding tools, it just doesn't just happen naturally. It's learning how to drive. I think we might be in the era of F1 driver right now, which is like really good people can drive these systems a lot harder, a lot faster than people who just picking them up. If you've never used an agenti coding tool, there's a fast difference between someone who's like really expert in them, knows where they break.

Can pay attention to that versus someone who's just picking up for the first time. One analogy I've heard is we used to talk about 10x engineer. You remember, like, this used to be a debate for a very long time. Is it or is it not? But now what I'm hearing is the 100x engineer. And so you're saying that you are seeing some folks who maybe less than 100x engineer, but like this F1 driver who is just really good at it.

Like, how would you describe a person who we've seen this? Is it just like rock saw engineering basics and they picked up, they lean into using these tools or what are they like? I mean, there is quite a bit of a, it feels like, you know, it's directional that they're a good software engineer before. I mean, I sometimes think it's like, you know, everyone's in this kind of spectrum of capability software engineering. And this is just like, you know, taking that line and spread it out. And it's not quite true. You know, I think it's helped some people more than others, but you know, it feels like it's just stretched it out.

So, you know, your ability before is now amplified. I'm always interested to learn that, for example, Boarders' Journey, T-Bo at OpenAI, they both have been really good software engineers. Boris wrote one of the first types of books, the first types of book from Arilie. He built some massive system, same with T-Bo who built it. They're kind of seeing, oh, these people are building all these tools and innovating like, yeah, they really go before us.

Well, I mean, this is a, we have two other analogies to give you about like, what feels like an AI. You've only heard about the term pair programming. Yeah, the idea behind pair programming is it's good to just like, you know, have one keyboard, one monitor and two engineers at it and one of the keyboard and the other one is inside them. Like, look over the shoulder and giving guidance. And I think there's an aspect of that feeling where I actually got chached to do a little image of this where, you know, it's like the Android is out there, computer typing and you're just there giving instructions.

That was maybe the way it felt like a year ago. I think it feels a little bit different now. The one I just started recently saying is that I feel like the domain experts, the people who were really strong before are now massively amplified. Have you seen all this mathematical stuff coming out like the crazy proofs? Well, I don't understand it, but I understand it either.

Yeah, yeah. So I told you it wasn't very good at math. Yeah, it's not complete. I just was like, I stopped in the freshman year of college, right? But you know, I kind of like watched along with these advancements in, you know, turns to how he's like, most famous living mathematician, you know, super genius.

He actually posted this session. There was this recent breakthrough. It's called the Jacobian conjecture. I don't even know what it meant. But posted this chat, he became session where he's interacting with the chat team and you can see him interacting with his intelligence.

And it was crazy because he's talking to it as a peer colleague is responding and literally, it obviously looks like I encourage everyone to go look this up. It looks like, you know, almost a foreign language. It's like his domain expertise is being amplified by the system he's interacting with. You can see how he's like kind of learning exploring ideas just really rapidly. Now, there's all this controversy about mathematics, but I feel like the analogy that comes to mind to me, though, is the domain experts, they're a little bit like sorcerers and they're particular domain.

You know, yeah, the earth sorcerers, the earth wizards, the water ones, whatnot. And if you know the magic incantations, the right words to say the right order, you actually get something kind of magical. And if you don't, you just get sparkle stuff that doesn't have anything there behind it. You know, you would probably make you don't know much about distributed databases. If you're good at ask fable or asterisk, build me a distributed database like cocker TV. You get something out, but it'll kind of be ultimately hollow inside.

You're an expert in databases and you ask about distributed database and you can point out all the various things you have to know about distributed database. You have to worry about the storage layer of networking layer. Here's the various data structures and runtime inside. You can actually get something quite magical, very, very rapidly out of it. What would your advice be for, advice before engineers with like mid level to senior level who have been figuring out how to do coding to become strong engineers in this, I guess, H of AI?

Well, the first off is you have the most amazing tutor kind of readily at hand. And I mean, one of the things that I would always do, you know, throughout my career. And now I've kind of stopped doing it. But it's like the reason why it's going to become obvious, which is like, I would always look at other people's code. So I was at, you know, Google early on, you probably heard of Jeff Dean. Jeff Dean was amazing coder. His colleague, Sanjay Gamawat also just named Cobble Coder. And, you know, I would be looking at their pull request.

I'd be looking at their changes. It wasn't called pull request at Google. There's a different name for it. But I'd be looking at the CL, right? Yeah. CL. This is a P4. It's a different version control system. But I'd be looking at their changes and be like, how they do what they did, right?

We'd be looking at their code. Oh my goodness. Sanjay's code is really always very elegant. You know, just as high performance. How's he doing that? You know, like, how's he going about it?

And it's almost just like you acquire VOSMOSAs. But now what you can do is not just acquire VOSMOSAs, but you can really like, I mean, I would be encouraging if you're a junior engineer, you know, if there's a senior engineer nearby, like you could ask them how they're doing what they're doing. But you just ask the AI to dissect what they've done and explain it to you and explain it to you at various different levels. Like, how's this code work? What is it doing? Give me a diagram. Explain to me like I'm five. Explain to me like I'm 10. Explain to me in French. Whatever, like, you know, like, I mean, fundamentally, to some degree, AI is a translation tool. They're translated from whatever, you know, language or understanding. It's in and keep on interrogating it till it increases your understanding.

You were saying how domain experts are very much amplified. I guess one strategy as a software engineer is like, obviously become a great software engineer and use it as a tool. You can get a lot faster. You can go to distributed systems. Like, I'm not a distributed databases expert at all. But I use AI to explain a few things for me to understand up front, which was very helpful. And it would have taken me a lot longer time before. And so I can use this.

But I wonder if there's another part of like as a software engineer, you can use it to become more of a domain expert wherever you're working. If it's a payments company, I mean, use it to learn about payments as well. So you can help the business. You can help your team. And honestly, you'll just learn more, right?

Yeah. I think, you know, I would encourage everyone. You have to have a little curiosity, right? Don't be bound in by the area you're working on. Explore outside of it. I was working on Gmail, but I was fascinated about how the main Google search engine worked. I was fascinated by how the internal big table worked, even though I wasn't directly working on big table. Just explore in. Look at those things. And now it's so much easier because you have this super advanced patient intelligence there.

You explained it. Why do you think it was done this way? And then once you become an expert, you can be an attorney. I see it's done that way. Why don't we change this? Would this be helpful? And that's where you go from just kind of learning to actually contributing back.

I think everyone has to have the personal agency to do this. You know, if you're just sitting there waiting for someone to educate you on how to do this, it's going to be really hard right now because anyone who's coming in explaining how to use AI or how to be a better software engineer, they're going to be out of date, right? You just got to get in there using this tools all the time yourself and using it like using it to learn. I feel like I've learned more in the past, probably in a year than previous five years from buying is weird given your trajectory. I'm given the environment you were working on, right? I mean, like everybody's been in the industry. I'm like, I'm a definitely better coder. I was a better coder 10 years ago than my first time in the industry. It's like, I can look back every decade and realize like, I got a lot better. And I feel like I just got a lot better.

I was really excited to talk to you because when we started to just exchange messages, the first thing you wrote to me and I asked you like, hey, you know, how are things going? You said like you wouldn't believe, but my coding output is insane and it's high quality and it's data based quality level. And those were the things I don't really usually see. I usually see, okay, I'm not producing more code, but it's slob. But again, like, to me, this is a bit of inspiration. Like, look, you can use these tools to just like amplify yourself as a software engineer.

Like you are one example, right? Hopefully, what I'm many. Yeah. Yeah. And I'm not the only one who's talking about this as well. I find it very exciting.

You know, it's a little bit exhausting right now, but it's very exciting. Like, I got a software engineering because I like building stuff and I can build stuff faster. You know, the stuff you might have had to compromise on the in the past and you can take away some of those compromises. I mean, you see this in the UX software coming out, like the UX is a lot higher. You see all the PMC, you like web animations and whatnot. That's only just like the surface level. It just extends way, way, way below that.

Peter, this was awesome. Thanks for coming on the podcast. Yeah, this is wonderful. Thanks for having me. One reason I was excited to talk to Peter is because he's been a very high profile and productive engineer in pre-AI building some of the most resilient distributed systems in production.

CockroachDB has known for resilience and how even if several nodes are destroyed, they'd be so low price while they're data loss. Basically, it's as hard to get rid of as cockroaches are, hence the name. One interesting part of our conversation was how Peter built more efficient data structures than the standard coding libraries had. Thanks to him and colleagues paying attention to parts of the library that seemed slow. He did it for the C++ STL map and then go for the Swiss table implementation.

I found both stories a good reminder that you can improve the existing library or even the language, especially if you measure which parts feel slow. Another part of his conversation that I liked was how Peter came a bit of a full circle. He used to write 100,000 lines of code per year, being a very productive engineer and CTO. He didn't stop writing code aiming to code engineers between 2022 and 2024. And then he started to code again because with AI tools, he wanted to coach his engineers' best friend.

He wanted to coach his engineers better, but it's hard to do if you don't use the tools yourself. And now we find himself being extremely productive. And this time, the team around him is productive as well. And we're not talking about live-coded software, but database-worthy, high-quality code-generated, and committed to production. Peter has convinced that AI amplifies existing expertise, and this is one reason why he probably learned more this last year, building with AI, than the previous five years combined.

And I find it a value reminder that learning and building deep expertise and software engineering is very valuable. And as closing, I appreciated that Peter said that not only is he excited, but he's also exhausted. There's a lot to learn, but it's tiring. And neither him nor anyone I know is immune to this. So if you're also exhausted with all of the things going on with AI, no that you're not alone.

Check this one out for more The Pack-Net Engine and Deep Dives on Google's engineering culture and on the Studio Systems. If you liked this episode, please make sure you're subscribing to your podcast player, and a special thank you if you're liberating. Thanks, and I'll see you in the next one.