Welcome to Changelog and Friends, a weekly talk show about doing big things. Thanks to our partners at Fly.io, the public cloud built for developers who ship. We love Fly. You might too. Learn more at Fly.io. Okay, let's Kaizen, live in Denver.
Well, friends, I'm here with Damien Shingleman, VP of R&D at Auth0, where he leads the team exploring the future of AI and identity. So cool. So Damien, everyone is building for the direction of Gen AI, artificial intelligence, agents, agentic. What is Auth0 doing to make that future possible? So everyone's building Gen AI apps, Gen AI agents. That's a fact. It's not something that might happen. It's going to happen. And when it does happen, when you are building these things and you need to get them into production, you need security. You need the right guardrails. And identity, essentially authentication, authorization, is a big part of those guardrails. What we're doing at Auth0 is using...
Our 10 plus years of identity developer tooling to make it simple for developers, whether they're working at a Fortune 500 company or working just at a startup that right now came out like Combinator, to build these things with SDKs, great documentation, API-first types of products, and our typical Auth0 DNA. Friends, it's not if, it's when. It's coming soon. If you're already building for this stuff, then you know. Go to Auth0.com slash AI. Get started and learn more about Auth4GenAI at Auth0.com slash AI. Again, that's Auth0.com slash AI. We're recording. Yeah, we're good.
Let's just let it go. While we're here, how did Kaizen begin? It was a long time ago in Japan. Yeah. I don't know. How did Archizen begin? 1,300. That was a year. Something like that. It began with this crazy idea that let's improve, but in a consistent way, so that every 10 Ship It episodes, we'll talk about the improvements that we drive in the context of changelog. Share them publicly, and that was our reminder that, hey, 10 episodes, we'll be talking about everything that we've done. Some called it navel gazing. I don't think that's what it was. That was me, probably. Jared. It was good fun. And, yeah, it's been four years. It's been four years. It's been four years. It's been longer than four years.
Has it? A decade. We've been Kaizening a decade. We've been calling it as a Kaizen. So Kaizen one was about four years ago. Kaizen as it is, materialized roughly four years ago. But the relationship, that's why I bring that up. I want to say one thing. It's like, these guys are my ride or die people.
Gerhard's amazing. Jared, you know you're amazing. But the magic and the beauty that's come from this relationship has just been tremendous. And to be here and share that with you all and to share Kaizen 20, this navel-gazing approach to our platform, this constant attention to detail of improvement, and I think particularly with what we'll talk about today is unique to us specifically in that we built some infrastructure that's used by us specifically, where a behemoth, not negatively, is just not maybe the right fit. We've been holding it wrong to some degree. But this, what we're doing here, is just a wild ride. I'm excited for this moment. Is there a way to hold it right? That's the question I can ask myself. It turns out...
if you gaze at your navel long enough, there's cool stuff in there. Definitely. So thank you for joining us. You have to do something about it. You have to have a proactive approach to it. Yeah, we pull out a few of our treasures that we've found along the way. And something that we've built over the last, when did we start Pipely? Pipely was, I think, 18 months ago, roughly. The idea was, shall we do this thing? I mean, we've been talking about it long enough. Shall we actually try doing something about it? And the beginning was, as all beginnings, like, can we do this? How long will it take?
What will it take? Do we know what even needs to happen? And that's how the conversation started. And many of you that have listened to those conversations remember us, right? How we were pondering, should we, should we not? Are we crazy? Three wise men, the question mark was really the emphasis. Like, are we wise doing this? We have no idea. And then along the journey, the best part was the friends that joined us. So it turned out it was not such a crazy idea. And just the idea of improving something in public so that others see how we do it. And it's our approach. And maybe it will inspire others. And I think that worked really well. And here we are today.
With friends. With friends. And that's, again, that's the focus. So improving with friends makes me so happy. Thank you all for being here for that. Thank you. We appreciate you very much. Absolutely. So it all started with a dream. Yeah. A pipe dream. Indeed. On a Kaizen number that I can't remember. 13, I think. Kaizen 13. I think so. When we were lamenting our cash-miss ratio on Fastly.
But they had this really nice varnish as a service that we've been using for a long time. We just didn't like the way that we had to use it through Fastly, which is through a web UI and a strange comment-based versioning control system that we invented on top of it called the name of the person who does the thing and the thing you're doing as you update the config, which would produce this gnarly, I don't know, thousand-line varnish config that would work mostly sometimes. And it was great when it worked, but when it didn't work, it was very difficult for us to have visibility and debuggability in order to fix that. And so I said on Kaizen 13, Wouldn't it be cool if we could just have this 20 line varnish config in the sky, just deployed around the world, and it could just be, just for us, everything we need and nothing we don't.
And Gerhardt got a twinkle in his eyes. And he said, 20 line, I can give you 500 lines. Well, I think it's close to 1,000 now, but anyway. So it was a pipe dream. But that began our journey down this particular path, which we are at the end of, or a milestone at least. You're never at the end of this kind of path. Where we decided on our Kaizen 19, hey, we're ready to run this. We call it Pipely as the open source project. Pipe dream is our instance of that, because this has been our pipe dream. And we're there. We're almost there. We're there now. But on Kaizen 19, we were like right at the precipice. We are there.
And I said, what if we just get together and do it together? And you guys said, yeah. And I was like, how about on stage in Denver? And they're like, sure. And so here we are. There's your setup, Gerhard. The way Kaizen's usually work is Gerhard works way harder than Adam and I do. And he prepares. Yes. And I say something like, Gerhard, take us on a ride. Tell us what we're going to do. And so, Gerhard, take us on a ride. What are we going to see today out of Pipely and Kaizen 20? Thank you very much. So we will start with a little bit of history so that everyone is able to visualize a couple of very important milestones. Then we're going to ask you a few questions.
So you better think of some good questions to get back at me or at us. So let's see how that goes. And then we'll do something special because everyone took their time. This took considerable effort on all your parts to be here. And we want to recognize that effort by doing something special on stage live. So let's see how that goes. All right. So first of all, let's start with the beginning. The beginning is July 26, 2025. This is an important moment. We're all here today. It's the first time this is happening. Kaizen 20 is the moment. Thank you all for being here.
I'm sure that you all know this. Now, it says 20 episodes. Actually, it's 19. One was republished, but it's been 19 Kaizen episodes. And this started actually in 2021, in July. I had to look this up. I wasn't sure exactly how long it was, but that's when it started. So this journey, officially, when we started having these conversations, all recorded, all remote, happened many years ago, and it took us this long to finally do this in person. 10 years. Right. And if you count exactly, when this unofficially started, it's been a decade. It's been a decade. So we're good friends now. I'm stretching it by one. 2016 was the year we launched.
What is now our platform, it's an open source platform. ChangeLaw.com, you all know that. It's on GitHub. Some of you contributed. You're probably in the issues or at least reading them. But it's been a journey. It's 2025. It's not quite 10 years, but I'm ranting up. Give me a gimme. Next year, maybe we do something even greater, right? Maybe we're building up to that. Okay, so the reason why we're doing this is because...
It's about friends. This is a new context. It used to be Shippit. Before it was unofficial. It used to happen just among a few people. And now all of you are here. And we appreciate you. So thank you very much. Because as you know, it's so much better with friends. So changelog and friends is the context. And including friends that want to be here but maybe can't next year, hopefully, or the next time we meet, it's a good reason to make this crowd bigger. But this is how it began. This is the first moment we have met in person. And it's amazing. All right. Now.
You haven't noticed here, but this is something that I pay attention to because the person that takes the pictures is never in the pictures. So we need to acknowledge the person taking pictures. There he is. And there's Aaron. He's over there. He's filming. Hey, Aaron. Thank you very much. So, yeah. Thank you for being here and capturing this moment for all of us. That's so great.
We are friends. I think we can call ourselves friends when we go for a beer and we go for a hike and we share a meal. So that's, I think, what it means to the beginning of a nice friendship that enjoy making things better. That's really what brings us together. And we do it in a public way. We share. And even when we get it wrong, that's fine. Because it's always about improving. It's not about being 100% right or knowing it all, figuring things out. And that makes me so happy. So you all know this.
We post these as discussions on the changelog.com GitHub repository. This specific kaisen is discussion 546. And if you want to do a bit of digging to see what went into making this kaisen, that's where you can go for the technical stuff, for the pull request, for the code. All of it is there. And I think the first step that we need to do is set the record straight. And the reason why I remembered it is because of that very small thumbnail. There's three people there. There's BSDPHK. So PHK is Pool Henningkamp. I didn't know who he was until recently. He is the guy that had a huge contribution on FreeBSD.
NTP time counters, FreeBSD jails, MD5 crypt, and you all know this. Can I say it? Yes. The bike shed. That's it, that's right. Bike shed. He invented the bike shed. 1999, the guy that, so I think that's really important. And we didn't know, a bit of history. This was an important moment. And I think I'm going to switch the screen now to this, because this is important. So Andrew O., he wanted to set the record straight. No, Allgreen, sorry. Allgreen, May 13th, he wanted to set the record straight about Varnish, which is the technology that we use, the relationship to Fastly, and the relationship...
to Varnish Enterprise, and it's all here. So Fastly is not running Varnish Plus, which is a Varnish software product, and they have their own fork. So that is important. It's similar, but not quite the same. What we use is nothing from Varnish Software, not Varnish Plus. We're using the Varnish Cache, the open source project, as anyone can consume it. So we have built, everything that we build is on open source technologies. And that is important to us because that is in our DNA. So there's some history here. That's important. So we are building on Varnish Cache open source. All right.
Back to where we were. Cool. So we did a thing, and this thing is important, but actually we did a few things over, I think, four years. We did quite a few things. Ten years. Ten years. Ten years. Okay, ten years. Just keep that in mind. Adam's not going to let it go. Ten years later. That's a good one. So they were all good. I think most of the things were good, right, that we've done. Most of the things were good, but that's not all things. Right. Shall I fix this now? No. No, it's okay. Sorry. My bad. So what are some of the things that you don't think went as well? Let's talk about that. Out of your...
collective memory. Maybe even someone from the audience can tell us a few things that didn't go so well. S3 cost. S3 cost. Yeah, we spent more money we should have on S3. We're ballooning, and we didn't know it. Egress. Until that one Kaizen episode where I finally looked at it. And I was like, oh, we should change this. We should address this. The bill went from like 15 bucks a month to like maybe 180 at peak, I believe. Which is still small. We're not massive. We're in operation. Yeah.
That's not cool. But now we're on R2 and we're spending like eight bucks a month. They basically pay us. They should pay us. Now it's good, but that wasn't good. I deleted a few things. I should have. As you do. Flying too close to the sun is a good thing. There was one time where Gerhard went into, I don't know what it is anymore, a config or one of my pieces of code, and he changed something.
that to me was inscrutable, like why, what, who, but I had so much respect for the guy and so much imposter syndrome that I thought, surely he knows better than I do. And I must be a fool, but it looks wrong, but I'm going to go ahead and roll with it because Gerhard knows what he's doing. And I found out much later, you had no idea what you were doing. So this is something really important and it's like at the heart of what we do, right? We are figuring stuff out and we are okay to admit it publicly, right? Like we mess things up, but there's no way you're going to learn if you don't make mistakes.
It doesn't matter how much experience you have. It doesn't matter how many things you think you know. You never know. Let's be honest. You don't really know. You're mostly making stuff up. Some things help you, but it's all in the confidence that you will be able to figure it out, will be able to push through. Just stick with it long enough. That's all it takes. All right. So today, we did the biggest thing ever. What was that? A live show. A live show, yes. Yes. Yes. That's one. We showed up in a city we don't live in. Right. People flew here with us. That's a big thing. Is that what you're referring to? Yeah. That's one of the big things. What else? Well. I'm so curious what we did.
What did we do? Tell us. What did Gerhardt do? Let's scratch it. What did Gerhardt do? Yeah, yeah, yeah. What did you do, Gerhardt? So I did something. I'll show you very soon. Trust me, it's coming. But it's the biggest thing ever. All right. So the problem, the fact that it's green, don't let that mislead you. It's a bad thing. Okay? Color, psychology of color is very important. What is this? The 17.93% is the cash hit ratio on our current production CDN, which is low. Right? That's really, really low. It means that less than 20% of the requests get served really fast.
80% plus are slow. And while it doesn't really impact us, I mean, it's all good for us, it impacts you. When you load something, it takes a while to load. And it shouldn't be that way, right? Things should be instant. Things should be very, very smooth. And when something goes wrong in the backend, for example, if 80% of the requests, they have to go back to the backend, it means that they can fail. So the chances of something failing are fairly high. There's something wrong in the backend. And that's the other thing, which I keep thinking about a lot. Yeah. And it's worth noting that our data is unidirectional. I mean, we rarely change anything once we publish.
Episode 1 is still episode 1. 19 years later. We might put the wrong audio in the wrong episode. have to edit something. Or you ship the entire wrong audio, like I've done recently. So we make mistakes, and when you make a mistake, you want to be able to quickly rectify it, purge everything, and get back to where you were. But generally speaking, you put an MP3 up on a CDN, and then you deliver that same MP3 in perpetuity. And so this number is abysmal. That's terrible. We should never have misses. And that's what I've been saying for 10 years. How do we keep having cache misses? And the problem is that, in this case, it's mostly the website. So the changelog website appears slow in a lot of cases, and that's not great. So that was the problem that you were trying, or that was the thing that we're trying to improve.
That's how this started. All right. Episode 26, this is what Jared was mentioning when we began. Should we build a CDN? That was the moment we were thinking, should we do this? I mean, are we really at that point? And it took a while, but the conclusion was yes. I mean, we should at least try and see how far we get. That was 18 months ago.
Pipely.tech, we had this up for a while. We talked about it at Christmas. A new CDN is born. It hasn't been updated recently, but it will be. But this is the home for the open source project that we would like others to use at some point. I think it's getting there. It's not far there yet, but we have made many improvements to make it easier to consume.
Well, friends, I'm here with a new friend of mine, Harjot Gill, co-founder and CEO of CodeRabbit, where they're cutting code review time in half with their AI code review platform. So Harjot, in this new world of AI generated code VR, we are at the perils of code review, getting good code into our code bases, reviewed and getting it into production. Help me understand the state of code review in this new AI era. The success of AI in code generation has been... Just mind-blowing how fast some of the companies like Cursor and GitHub Copilot itself have grown. The developers are picking up these tools and running with it pretty much. There's a lot more code being written. And in that world, the bottleneck shapes the code review becomes even more important than it was in the past. Even in the past, companies cared about code quality, had all these pull request models for code reviews and a lot of checks. But post-gen AI, now we're looking at, first of all, a lot more code being written. And interestingly, a lot of this code being written is not perfect. So the bottleneck and the importance of code review is even more so than it was in the past. You have to really understand this code in order to ship it. You can't just wipe code and ship. You have to first understand what the AI did. That's where CodeRabbit...
comes in it's kind of like a think of it as a second order effect where the first order effect has been gen ai and code generation rapid success there now as a second order effect there's a massive need in the market for tools like code drive it to exist and solve that bottleneck and a lot of the companies we know have been struggling to run especially the newer ai agents if you look at the code generation ai the first generation of the tools were just tab completion which you can review in real time and if you don't like it don't accept it if you like it just press tab but those systems have now evolved into more agentic workflows where now you're starting with a prompt and you get changes performed on like multiple files and multiple equations in the code and that's where the bottleneck has now become code review bottleneck every device Okay, friends. Well, good. Next step is to go to coderabbit.ai. That's C-O-D-E-R-A-B-B-I-T dot A-I. Use the most advanced AI platform for code reviews to cut code review time in half, bugs in half, all that stuff instantly. You got a 14-day free trial, too easy in the credit card required, and they are free for open source. Learn more at coderabbit.ai.
This is something that has been bugging me for years. We run on fly.io, and fly.io has points of presence all over the world. But our application only runs in a, well, in Ashburn, Virginia, because it's closest to the database. Of course, it's going to be close to the database, right? Because data has gravity. But we wanted to distribute the application for a long, long time, but it was never the right model. With a CDN, that's exactly what we want to do, right? We want to get those instances all over the world, so finally, we can say that after all these years, we are holding fly.io right. And it's been working pretty well.
Is that the big thing? I think it is a big thing. Is that the big thing? No, no, no. It's coming. It's coming. I'm just waiting for that moment. This is one of the things. We are holding it right. I want to say one thing, too, real quick. Leave that there. That slide's good. That slide's good. The next one's good. Next one? Yeah, the next one's fine. Let me show some things. But you see Fly here. They're not here. We didn't make this about sponsors. We wanted to be about you all. Us doing a normal live show together that wasn't like, hey, let's charge a normal spot for tickets or whatever it was. We just want to go somewhere, have some fun, get together, and just share the story. But I do want to recognize that Fly and Kurt and the team there have been extremely supportive of us. Not saying you should use them, but they love us. We love them.
And what we're building is really on top of the best platform we believe. So Fly is amazing. Fully agree. Yeah, fully agree with that. Jared? All good? Yeah. All right. So this happened about six hours ago or seven hours ago. This is 1 a.m. last morning. OK, something went wrong. So things will continue going wrong. You will never really get there. It's all about the mindset of, can we do it a little bit better? And again, we are figuring stuff out. So this was yesterday, last night. So this is a question for the audience.
Who would like to see us improve this specific crash live? Yeah? Yeah. All right. Let's do it. Cool. So what do you think happened here? Let's do a very quick understanding of what the problem is. What do you think happened here? Can you describe the architecture of PipeDream in terms of what's happening out there? I can. So it's Varnish instances. I mean, Varnish is really at the heart of it. There's a couple of other components around it, but Varnish is the heart of it. Varnish.
makes requests to backends. The backends in this case would be assets, for example, we store static assets. We were mentioning MP3 files, PNG files, JavaScript, CSS, that kind of stuff, which rarely changes. Then there's a feeds backends. Feeds stores generated feeds for users, plus plus members, and shows, various shows. Every show has its own feed, and then you can also create your own custom feed. So there's something like on the order of 600 to 800 feeds, I would say.
Eight of which are way more important than the others because they are publicly consumed by all the podcast indexes. And those feeds get the most requests from all the platforms, the podcasting platforms that consume the changelog episodes, and they distribute them to their audiences. Or to your audiences, but through the platform. Our audiences. Our audiences, yes. Our audiences, for sure. And what this is, basically, we get these instances that are distributed around the world so that the delivery of that content gets accelerated. The one thing which I haven't mentioned is the applications, the changelog application, the website, which is an important one. That's where many users go to, for example, look at the homepage, look at news, things like that.
And that is the one which is most sensitive to latency, because as I mentioned, it's only in one location, close to the database. So we need to accelerate delivery of that website to users which are around the world, including Australia, South Africa, South America, all over the world. We have a very diverse audience. And we want those users to have just as good experience as anyone else that's maybe in the North America. So in this case, one of our 10? Yes. 10 instances of the Pipely application. Yes.
Crashed. Misty Bird 4931. Yeah, Misty Bird. That's the one. Misty Bird crashed. That's what happened. Exactly. So now your question to the audience was... Why do we think the application crashed? Why do you think it crashed? And anyone can answer except Matt, Nabil, and James. So they know why it crashed?
It ran out of memory. Yes, of course. Thank you. Someone's paid attention. Someone's paid attention, but why did it run out of memory? Why do you think it ran out of memory? Because there wasn't enough memory. Right. I love some trolling. Seriously now. You're going to make me say it, right? Okay, so as more content gets cached in memory, the problem is there's like a configuration which I wish it was easier to make, but you have to manually adjust how much memory you give to Varnish out of the total memory available. So there's a dance that you need to make so that you know how much is enough so that when more memory gets allocated, the thing doesn't fall over.
I wish this particular thing was easier, maybe something that we can improve. But for now, you have to fine-tune it and find, if you have four gigabytes of memory total, how much should Varnish be allowed to use? And if you think it's four gigabytes, it's way too much. So that's what we're going to do now. I'm going to switch to some live coding and show what happened.
So, actually, you know what, maybe let me try this. How's the font? Can everybody see the font? Can you see what's there? I'll make it a little bit bigger. A little bit bigger. Okay, so this is the change. And if I'm going to undo this, you'll see what it was before. So we, I thought, let me take responsibility of this, I thought that 800 megabytes is going to be enough. This was the case when the application had 2 gigabytes. So when the instance had 2 gigabytes, 800 megabytes of headroom was enough so the application wouldn't crash because of out-of-memory issues. And apparently, when you have 4 gigabytes, 800 megabytes is not enough. So what happened is 33%.
33% should be enough for this to work. And again, you have to specify this explicitly. I'm sure that we'll improve this at some point. This is what this looks like. So we're going to push this change into production live right now. Here it is. Here's the change. So we'll say increase varnish memory or limit. Let's do that. Limit varnish memory to 66%. 66%, right? Varnish memory can only use 66%. Okay. I commit it. I...
What's going on? Of course, I need to connect. Right. Let's do that. Let's connect. You're offline. Yeah, I'm offline. Let's do this. Again, this is live. This is not recorded. It hasn't happened. Let's see how this... I normally record these things, but let's see what needs to happen. Good time to go live, Gerard. There you go. Let me just read this Bill Gates quote. Okay. You thought 800 megabytes ought to be enough. Yes. Bill Gates, 640K ought to be enough for anybody. Right. Right. Apparently not. So, you know, it's not a reasonable thing that you thought. Nope. Okay. All right. So, what happened is we committed, and we pushed this commit, this one commit, and to deploy something into production...
All that we have to do is tag the repository. So tag and commit. So RC3 is the last one that went out. You can see when it went out, it was two days ago. We're going to do an RC4 now.
And all we have to do is this. J tag, J stands for just, so I'll do just tag, because I don't want to remember the command. It's quite long. So I'm going to do the SHA. The SHA in this case is going to be, sorry, tag. So V 1.0.0 RC? Four. Four, there you go. Nice. We'll just do head, right? Of course, it's going to be head. And the discussion, this is the changelog discussion where you can basically listen about this thing. So this is us preparing highs in 20. Remember that GitHub discussion I talked about? That's what this is going to do. And I'm going to push it now. No, push. Git push.
What happened? Undefined. That should have been fine. Looks like the connection. The connection dropped. Someone's blocking port 22. The connection dropped. Yeah, let me just go back. Let me read a Bill Gates quote. Yes, sure. Another one. Give me another Bill Gates quote. Let's get push again. Let me just make this. It's going to slow. There you go. That pushed. Cool. So the tag went out. And what we're going to see now is actually this one right here. It's going to go to the actions. So this is a...
Live this one right here 100 RC4 and it's going to push this change into production across all instances against roll them live We are deploying this so why is it significant sorry? Let me set the record straight sure Bill Gates in 1996 said I said some stupid things and some wrong things But not that no one involved in computers would ever say that that a certain amount of memory is enough right, so I guess I think back is that true? What is your source of information? Yes, so there you go. He may or may not have said that So let's see so now we're seeing like a live rollout and we'll see so publish and deploy tag is doing all the validation Gone gone. It's moving on. It's going through the changes Yeah, but it's going to resolve itself. It just takes a while. There you go Well a last time I think it took two and a half minutes so we are about a good idea of course Why not look we're already creating the new machines look at us. That's going all right So this is something that I think is important to get right early on and that thing is how long does it take you to put a change into production?
So how long does it take you to make a change and see that change role live? And this is something that we've been working on for quite a few years on this thing. We do push the production. We own our production. We don't develop in production. That would be crazy. Leave it for us. But it's like this mentality of if I'm going to make a change, how long will it take me to see that change happen? And if you can shorten that time, you're in a good place. 10 years ago, it took 20 minutes or so.
It was long. It was a long time. It was long enough that you go do something else. And now it's two and a half minutes, maybe. And this is a global CDN. So this is, we are pushing this change to all the instances of the global CDN that we run in. Where do we run them? Let's have a look. So this is it. So we are Ashburn, Virginia. These are the instances.
Two days ago. If I'm going to refresh this, you'll see the new instances come live. So that was two days ago. That was like the last deploy. Let's refresh. Deploying 57 now. That's the commit. We have new instances coming up. And you can see that we do blue-green. Of course, we need to blue-green. We need to deploy the new ones. The old ones are still there. Two of everything. It's an important rule. And yeah, this works fairly well. It will not have any downtime. And we can look at that in a minute. So Ashburn, Virginia, Chicago, Dallas, Texas, Santiago, Chile, San Jose, California, Heathrow, of course, London, Frankfurt, Sydney, Australia, Singapore, and Johannesburg. So these are all places where we deploy these instances. And they will accelerate.
all the content to our users. Does anybody out there have a deploy pipeline that runs faster than two and a half minutes? Does anyone have a deploy pipeline? Let's start there. Two people, three people, four people. I think we're winning. Five people, six people.
Cool. Two and a half. Two and a half. Nice. It only feels long when you're on stage. Yeah. Well, this is real. This is real. Like, not edited. This is what it feels like. Now, I don't think two and a half minutes is long. I think we can improve it. But you need to think about all the things need to happen behind the scenes, right? The allocation of resources, the health checking. Like, how do you know what you put out there is correct? And you need to wait a while to make sure that the thing doesn't crash. That's why you need to wait at least 60 seconds before you can say, yep, this is good. You need to do a few health checks. Because if things start falling over, you don't want to leave that thing running in production, of course. Cool. So I think we're in a good place.
The one thing which I wanted to show is if I come back here, can you see that? Is that graph looking good to you? Yeah, cool. So do you see this yellow line? This was the instance that crashed. So this one, it's San Jose, California. For some reason, there's a lot of requests hitting this instance. And those requests, they don't look like human requests. So I think we may have some sort of bot situation going on. Some sort of, I don't know, LLM trying to learn. Yeah, there's like a lot of requests. And if I'm going to look at this one, the CPU utilization, this is the same instance right here, the one in San Jose, California. I mean, look just how abnormal this instance behaves. And if I'm going to remove that, you can see all the others. So all the other instances, the CPU usage is fine. But this one is spiking up to 100% because it has a lot of requests.
Obviously, this is in front of the application, so five-minute load average. We can also see here, San Jose, California. Yes? Can we see the endpoints that it's serving? The endpoints that it's serving? We can, yes. So let's do like a tiny readout right now. Jared's like, can we do X? Of course we can. Of course we can. No, we can. We can definitely do this. Okay. So...
I'm going to switch to this view, and this is the last seven days, and I'm going to make this a little bit bigger for you to see. And can you see that all right? Okay. So you can see we're here, right, July 26th. So the application went into production a few days ago. So not all of it, not completely, but we started routing a bunch of traffic to the application to see how well it would handle our users. Is that the biggest thing we ever did? Almost. It's getting close. It's getting close. But this basically shows that...
we had many, many steps towards this moment. And now, together with you, we've shown you how we can update something that's running live. That is serving, I mean, this doesn't sound like a lot of requests, like a thousand, but the granularity is 30 minutes. And we are sending a portion of the traffic, and it's handling it pretty well. And we can see that the biggest users, or like the most requests, are coming from ORD.
Chicago. That's the one. Frankfurt next, London Heathrow, and it's not me running low tests in Singapore. So these are like the, and then San Jose, California right here. So we are running live traffic, not all of it, a portion of it, but we wanted to see will this thing continue working well. Did you know that?
No, I don't think so. Yeah, so technically we're not launching Pipedream today because we launched on Thursday just me and you. Exactly. Sorry, team. We did it. We barely launched it. We launched one out of five. One out of five, yeah. Yeah, so roughly 20% of requests go through our Pipedream as well. So whenever you go to changelog.com, that's it. One out of five requests hit the new instance.
And we were able to see how does it behave with 20% of traffic. And it's working. No one complained. Adam didn't even notice. And that's like one of the best things, right, like to roll things out. You do it in such a way so that, I mean, it is a big thing to us and to people that understand and know what goes behind it. But to everyone else, did anything change? Because if you do this type of job correctly, all people see is maybe little improvements. And if they're not paying attention, even those they will miss. I think you're always as badass. Well, I think we are a good team.
The other thing which I want to mention is that, well, what you see here, again, it feels very, it just happens, right? It's like a small thing. The work that went into it, it was months and months of preparation, months and months of discussion. People joining us, working with us. I want to thank James. I mean, he was the first one that has joined. Thank you, James. Thank you very much. James A. Rosen. Yeah, thank you, James. Thank you.
talking in the various issues, the first one that we had, basically being with us for almost two years now, discussing about problems that we thought were problems. At some point, we were questioning, like, are we maybe too strict? Are we too demanding out of this? And no, no, I mean, we want things to be better, and this is why we want things to be better. So James, thank you very much for all the conversations, keeping us, like, big picture, like, that's all objective perspective as well. That was so, so helpful. And then Matt, right, Matt Johnson, thank you very much for that. Thank you. Just, thank you. What a matter. Thank you.
It was all about like VCL and going a bit deeper just to have like another perspective on VCL. I mean, he has a lot of experience in VCL and Barnage in general, but also documentation, also like a diligent approach to how should we do this so that it's easier for others. And that was like so great to have that help. Now, that was months and months and months of, I mean, again, all of us have full-time jobs. All of us do something else. But in our spare time, we find a bit of time to help others. And this was that. So thank you very much for that. Matt had a good idea last night. Can I pitch it to you? Of course. Yes, please. Yes. So my desire was a 20-line VCL. Yes.
You gave me a thousand lines. Matt says, we can count. You guys can just pull most of it out and do an ink click. Yeah, that's cheating, but yes. I won't look. Yeah, of course. Great. Every time I go to the repo, I'll be like, oh, this is nice. We can give you 20 lines, Jerry. We can give you less than 20 lines. Yes, I think that's a very good idea. Yeah, just look number one. Like step number one. Yeah, exactly. We'll cut in that for sure. So I wasn't prepared to do this, but let's do it. So we have, this is just, we have like a bunch of targets here. There's one that says, how many lines? Oh my goodness. Right, so let's just run that live and see what happens. So how many lines I'm going to tap this.
So let's see how many lines we have. 961. Hang on, let's see what's in varnish. Hang on a second. Oh, no, we don't want that. This is not correct. We just want ours. We just want ours. OK, shall we change this live? All right, let's see what's going to happen. Why not? So we want only varnish. Actually, no, varnish. No, we want inside vcl. So yeah, that's just like one change. So let's go just file. Do you want to do it? Nope. I feel like a step again. All right, so let's see. How many lines? How many lines? There you go. So varnish. Let's do varnish.
All right, let's see that. Varnish, sorry, VCL. VCL. Okay, let's see if that works. There we go. How many lines? There you go. These are the actual lines. All the lines. So, 364, that's like the main one. And these are like the includes. 24 in this one, 106 in 15. Right. How many total? Less than 1,000. I think it's less than 1,000.
500 yeah ish getting to 500 between 450 and 500 a lot of these are just like statically do the math you have to do yeah so again any savants out there do the math 15 that's how we do it 106 plus 24 plus 364 look at that 509 509 so there we have our answer cool so it's still good still good all right so you want to thank Nabil yes of course Nabil how could I forget Nabil of course so one really important thing is that Varnish cannot terminate backends which have SSL in front So if the backend is talking HTTPS, Varnish cannot use it out of the box. Varnish Enterprise and other products can, but the open-source Varnish cannot. We did a bit of digging, and we realized that's how we learned about Pool Henning Camp, PHK for short, and he was always against including SSL anywhere near Varnish because it would complicate things too much. SSL is really, really complicated. So with Nabil's help, he wrote 50, 60 lines of Go code that intercepts all the requests going to those backends, terminates SSL for Varnish, and presents the request unencrypted. A really simple, elegant solution, one of the...
almost like a sidecar that sits next to Varnish and helps it terminate requests which need SSL. So I'm not sure if that's... Is that any justice, Nabil? Genius. Absolutely genius. Let's hear it for him. Nabil. Good job, Nabil. And his SSL termination. Thank you. TLS exterminator. TLS exterminator. That's the one. Cool.
Well, friends, it's all about faster builds. Teams with faster builds ship faster and win over the competition. It's just science. And I'm here with Kyle Galbraith, co-founder and CEO of Depot. Okay, so Kyle, based on the premise that most teams want faster builds, that's probably a truth. If they're using CI providers with their stock configuration or GitHub actions, are they wrong? Are they not getting the fastest builds possible? I would take it a step further and say if you're using any CI provider with just the basic things that they give you, which is if you think about a CI provider.
It is, in essence, a lowest common denominator generic VM. And then you're left to your own devices to essentially configure that VM and configure your build pipeline, effectively pushing down to you, the developer, the responsibility of optimizing and making those builds fast. Making them fast, making them secure, making them cost effective, like all pushed down to you. The problem with modern day CI providers is there's still a set of features, a set of capabilities that a CI provider could give a developer. that makes their builds more performant out of the box, makes the builds more cost effective out of the box and more secure out of the box. I think a lot of folks adopt GitHub Actions for its ease of implementation and being close to where their source code already lives inside of GitHub.
And they do care about build performance, and they do put in the work to optimize those builds. But fundamentally, CI providers today don't prioritize performance. Performance is not a top-level entity inside of generic CI providers. Yes. Okay, friends. Save your time. Get faster builds with Depot, Docker builds, faster GitHub Action Runners, and distributed remote caching for Bazel, Go, Gradle, Turbo Repo, and more. Depot is on a mission to give you back your dev time and help you get faster build times with a one-line code change. Learn more at depot.dev. Get started with a seven-day free trial. No credit card required. Again, depot.dev. We proved it. We're here.
We did it. Kaizen 20. One more thing. This is too far out. This is too far out. Actually, no. We're good. Anything else you want to cover before we do this? There's one more thing that we need to do. Kaizen 20. Any thoughts? Any comments from the... I mean, you're here. Do you have any questions? Do you have any comments? Do you have any thoughts? Anything? We have a question. Can we talk about how we're using Dagger? All right. I can show you. How about that? Yeah, show them.
So there is one very tiny change that I have here, which is using dash dash cloud. This is an experimental feature that exists in the Dagger CLI. The reason why we do this is because we need a remote engine, a remote Dagger engine, so that everything that runs in the context of Dagger is quicker. Over Wi-Fi, it would be over like my header connection would be very, very slow. So how we use Dagger in this case, let's go here.
We have a couple of commands. The one I think that's the most interesting one, for example, for running tests. So let me show you what it looks like. I'm just doing just test. You can see there it does like the dash dash cloud. Let's see how fast it goes. It starts an engine in the cloud. It connects to it. So we need to set up Varnish. We need to connect Varnish to all the backends, the TLS exterminator, and we want to do it in a way that is as close to production as possible. So it's almost like we want to run the system as it runs in production, but we want to do it locally. Okay, in this case, it's a remote engine, but from the perspective of how it gets put together, it's literally just the same.
context of running containers. And because everything runs in containers, we get the exact reproducibility that we get in Fly. So we get the same configuration, the same Linux subsystem, the same kernel, or like a very similar kernel. And what that means, I mean, in this case, it's even using Firecracker behind the scenes, so it's very close to Fly. So we are able to test the system.
most accurately, and we're even able to run the system most accurately as it would do in production. And that part is hard because whatever you do locally, if I was running, for example, on a Mac, everything would be different. Even if I would have a VM, things would be different. So it's like that containers, the container image, the interaction between all the components, and when we ship something in production, it's as close as possible to that image. Even now, like, to the Go version. So right now, for example, what's happening here, we're pulling down, I mean, some of it is cached, we're pulling down various dependencies, and you can see they're all Linux ones. So Linux, Linux, and again, a VM on a Mac, it gets you close, but it's not the same thing. You have, like, subtle differences.
I'm not sure if that answers your question. Yeah? Cool. Good use of dagger. Any other questions before we, in the back, do the big thing? I think the bill had one, was it? Yeah.
Does Varnish only cache to RAM or does it also use disk? So it can use disk as well, but we don't have disk configured. So the configuration is cache to RAM only. It is the fastest one. If we have disks, as you know, the problem is the host, they can't move around. So it's no longer stateless. That has certain challenges about placement. But also disks tend to be a little bit slower. Not by much, but a little bit slower. As an optimization, it would be worth exploring that, especially with NVMe disks, which is what we would get in Fly. So that is something as a future improvement. But right now, everything gets served from memory, which is exactly what happens in Fastly as well. Because that's what gives you that highest performance.
So follow up to that. Our crash last night was because it needed more RAM than we allocated to Varnish. And in Varnish, there's no way to say, just use all the RAM available. So if you tell it that, it uses more than it has available because it tries to allocate more than you have. So you need to set an upper limit. It can't know how much the machine has? Well, it knows, but it doesn't do the allocation correctly. Is that a bug? I don't know whether it's a bug. It's just not working well with the system available to the host.
Maybe, maybe. Varnish is how many years old? 20-something. There's no way that bug is a bug. It has to be a decision. Right. So I gained a lot of respect for memory specifically and memory problems because they're very, very hard. I spent maybe three years optimizing memory allocations on the RabbitMQ team for RabbitMQ in the context of Erlang. And I've learned that there are so many subtle differences between how memory gets allocated on different runtimes. In the case of Varnish, obviously it's C, so it's as efficient as it gets. But it's very difficult to know how much you should allocate ahead of time and what you should free.
timing, something has somewhere to give. And usually what you need to do is over-allocate. Basically, you give it more than it needs so that when you get these spikes, there's enough for it to spike so that it doesn't crash things over. Memory these days is cheap. So honestly, skimping on that is not worth it.
There's there's a couple of ways we can we can go around it like this is one of them limiting what varnish can do but also seeing if the varnish memory allocation can improve right because I know what it takes for example what it took in RabbitMQ to do that it took me years and I was like full-time on that so it took a while including to understand what happens behind the scenes so you need to break it down in a way that you map and observe everything when it comes to memory allocation and then you realize what needs to be tuned to your use case that's the other thing because every context tends to be different so the allocations they are generic to make them configurable to you or specific to you so that they're optimized for your use case, it takes a bit of understanding of what you actually need. I'm probably reducing it down too much, but I feel like you could just say, this machine has four gigs. Just use three, and if you need more, you've got another gig on there. Just don't crash. But my question, my follow-up, I guess, would be, assuming we figure that out, that little dance, what happens when our instances become overwhelmed? Varnish might not be crashing, but maybe it's slowing down, maybe the machine's bogged, can't do stuff. Is there an autoscale? I mean, the answer's more RAM on the...
fly instances or more fly instances in other regions or in the same region. That's right. Yeah. So there is a big difference between regions, by the way. And this is something like, this is the sort of thing that you only realize once you start using it. Okay. So this is, these are like the last 24 hours. I'm not going to refresh. It's a little bit outdated. I did it like you can see here. It was 1042 AM. So it was this morning before we started recording. And what you can see here is this one, the San Jose, California, the SSJC, is the one which shows like the highest fluctuation. A lot of these, like once they load up on memory, like Heathrow, they tend to be fairly stable. But then you have a few, for example, this one, Santiago, Chile. This is like 2.3 gigs. So this one is not using all the available memory.
This next one in Sydney is 1.84. And you can see that the memory is decreasing because it doesn't need. I mean, this is something I'm not sure why does this, for example. I wouldn't expect this line to go down. I would expect it to stay stable. So I'm not quite sure what's happening here. But this one, the lowest one in Johannesburg, it's using 816 megabytes of RAM. So this shows you there is a difference between different regions and how many requests they need to serve. So what I'm thinking is we should optimize for the regions that we have. And the ones which are busy, we should give them bigger instances, maybe beefier instances. But others, like Johannesburg, doesn't need that much.
So maybe we, and the problem is you can't use different scaling strategies for different nodes, and they have multiple deployments, so it complicates things a little bit. But this is a refinement. Yeah. I think we need more listeners in Johannesburg. I mean, what's up with that? Exactly. All right, there was one other question out there, I thought. It was back here. Yeah.
Yes, I can see what the paths look like in Santa Fe. I forgot about that. Thank you. Thank you very much. So we're looking at Honeycomb, and one of the integrations that we have in Pipedream is we are sending every single request to Honeycomb and some requests to S3. So we see what's happening with these instances. So what that means is we are able to, for example, if I come back to the boards, if I come back to the boards again, let's hope that tethering works.
It will be a little bit slow. Okay, so let's go pipetream requests. And I think maybe pipetream content, but let's go requests. We can see the get. Okay, and now we can slice them and dice them. The 404s and the 500s. So maybe let's do gets. Okay, so let's go to the method. And the only thing that we need to do, so this is one hour ago. So these are all the requests in the last hour. And we can say, give me the URL, which is what we've been asking. So group our URL. And what this is going to show us is the get, but also the number of requests. So what you see here is that the most popular request is podcast feed, which got 203.
requests in the last hour. And this is global. So we can do one more. We can say, let me remove that, and let me remove requests, and let me do data center. So we can see.
which data centers hit the most. So let's go with that. And we can see that Frankfurt, 71 requests went to podcast feed. Chicago, the next one, 47. San Jose, California. So these are the most requested URLs. If we zoom out a little bit, let's look at the last seven days so we get a bigger perspective. Add it to seven days. So you can see when it went live. So you can see that for some reason, this one, I don't know who this person is, uploads 9,500 times. Shall we go and check who this is? I don't think you'll know who this is. Avatars, people, let's see. So let's go to changelog.com. No, let's go.
Where is it? Where was it? Not this one, this one. Okay, copy that. Copy pasting, hard. Uploads, and I think this will be CDN. There you go. So let's see who this is.
Oh, yeah. You have a stalker, or if you, who knows? Who is that guy? Yeah, yeah, yeah. You recognize him. Who's this other guy? Like, 6QD. So that's Z4, or Z4, as you say it over here. All right, cool. All right, so let's see. What's this next other person? This is a fun side question right now. Why not? There you go. It's Cable. Cable. There you go. Using on it. Shall we keep going? Do you want to find who this person is? Yes. That's like the third most popular one, so we might as well. Come on, be Adam. Yeah, Adam, let's see. All right.
Let's try that. Uploads. Nick. So something is loaded to maybe a JS Party page way too many times. That's possible. Oh, yeah. But it will show up here if it was the case. True. But then we have like this randomness thing. What's that? Is dysfunctional doing something with?
Something? Oh, on their website? So you know what we could do? We could do user agent. I think someone's hot linking us, man. Let's do user agent, and we'll see where these requests are coming from. It's going to be a robot. Empty string. They don't want to be known. Okay, shall we do? Let's get to the San Jose URLs. This is what we're trying to get, right? San Jose URLs. Okay, let's do that. So let's do, okay, so it's not even in the list if you look at that. And look, this one has an empty string, so that's really interesting. We have somewhere an issue, so this needs to be, or, no, hang on, this was like when we were doing the testing, so maybe that's one not so much practical AI. All right, okay, so let's do server data center.
equals SJC, run query, and let's see what we get. Most popular in San Jose, pocket casts. Yeah, that's really cool, isn't it? Look at that, pocket casts. This MP3 was requested a lot, 428 times. Is this the last episode? Yeah, that one just went out yesterday, right? There you go. It's just a popular episode in San Jose. That's it.
That's better than I thought it was going to be, which is a hacker. Popular in Cali, not a problem. Look, over cost. Pocket cost and over cost and ten of cost. This is good. This is good traffic. We love it. Nothing wrong with that. Cool. Any other technical or otherwise questions from the audience before Gerhard does his big reveal? All right. Everybody wants to see what you got, Gerhard. All right. Let's do it. Cool.
Coming back here. So one more thing. This is the important one. No, that's something else. Cool. Flash something. All right. I flashed something, yes. Right there. So what we're going to do now, we are going to shift all the traffic. All the traffic. To PyDream. Right? So all the production traffic is going to go to PyDream. Shall we do it? That's why we're here. All right. This is the moment. So all that we have to do, this is how simple it is. It's always DNS. No scripting. It is. No automation. It is DNS. It all comes down to that. Yes. We are going to delete these one.
Buy one. These are the A records. Let me get my own recording of you doing this. Go for it. Okay. Kaizen 20 in Denver. Let's do that. This is going to be anticlimactic because we're going to delete those and then we'll be like, they're deleted. Now, if this one goes down, so let's see if the system can handle all the loads, right? Like what can happen? So after this, the only requirement is we walk away, okay? We do this thing and we go for lunch or something like that. That's what's next is lunch, I think. All right. So, boom. We have four more to go. Okay. Three more to go. Sorry. Yeah, after this one, three more to go. It's not deleting. It's not deleting. It's low, right? It's my, it's my, it's my pattern. Thank you very much. One more, one more. Cool.
And the DNS is like one by one. So more and more requests are going to go to this one, which is the pipe dream. There we go. So delete. So we're at one out of three. Now half our traffic will be going to pipe dream.
Look at that. And now 100%. Now, we have one more. So we're 50-50. What's this one? That was it. That's CDN. We have to do the same thing for CDN. You guys like the CDN ones, too. There you go. All right. So this is app requests now. Goodbye to Fastly. This is it. This is the farewell front to Fastly. This is it. This is a travel moment in a morium. Only if it works. Only if it works, yeah. I may need to add these back. If this crashes and burns, we'll have to go back for sure. The good news is they have no idea what we're doing. Yeah, exactly.
And this is not a live live show, so we're not streaming this. So it's okay. It's in this room. If it doesn't work out, we edit this part out. So that's okay. He always says that we never do. Okay. Cool. We ship it all. There you go. One more to go. I think this is one and one more to go. We're at 60 second TTL. 60 second TTL. They should be fast. Yes. All right. If you can get your phone out and hit our website, please. Let us know how it goes. But that's it. We only have these two. Let's generate some traffic. Where's the real-time dashboard showing the activity? Real-time dashboard, of course. Let's do that. Let's do last 15 minutes.
Let's see everything crash and burn. What's the worst of it that could happen? The worst thing that could happen is all our appliances are out of memory. Let's go back to here. Let's go to the memory. There we go. Every one minute, let's maybe go to the last one hour. So let's see. We had like a spike there, but that was, this is like 1220. So I think it's still good. I think what we're going to see, so two things we're going to see. We're going to see, let's come back here to the pipeline request. We're going to see a change. Like these requests will start going up. More and more traffic is going to start hitting. Look at that. Boom. That was like a spike right there. More requests getting resolved here. Status.
Still 200, that's good. So that's all good. And 404 is 500, so things are looking good. Let's see what else do we have. Let's load a few more boards. And the opposite is obviously in here, in Fastly. Nice. So it kind of works. Still. It works on my machine. It's so fast. Fastly service stats. So these are Fastly service stats. The requests went up as well. I'm not sure what happened there. Why did they go up? Because I just told everybody to go to it. Right. I don't think we have that many people here, but still. Cool. So those are the requests. All good? Plays. Plays MP3s. Plays MP3s. Nice. Play the show. Let's do PyDream service.
So there we have a few again. The cache, we can see things going up here, so this is the last seven days. We'll need to zoom in a bit, so let's go 24 hours so that we see the spikes. There you go, that was the spike. Yeah. What is that spike? This one is more hits. That's good. More hits. We're getting more hits. That's good. More hits. So it will take time, because the real question is, was it all worth it? And the answer to that is, well...
do we fix that cash miss ratio, cash hit ratio? Let's check out. That's the actual answer to the question. What's the 17%? So let's go back to that. 17% cash hits. Last 24 hours, this is the homepage, right? That's the one we're focusing on. Sure. So we're looking at the homepage. Last one day, we had 3,700 hits. Yes. And we had 33 misses. I think that's better. That's better. It is better. It is better. We hyzened it. It's better. We did it. No way. No way. Let's see. 3715 plus 33. That's 99.1% hit rate.
That's a lot of nines. Yeah. Two. That's two nines. Way more nines than we did. That's two nines. More that we had. There are some nines in there. It was like 17%. 17%. So this is much better. All right. So obviously this will take a while, right, for all the traffic to start shifting through. It is DNS, right? It's cached. But things, look at that. Things are happening. Things are happening. So theoretically, 60 seconds later, those TLS records should expire. Exactly. And then require a new hit, which goes to a new route, which is through. Dig. Change.com. And change.com, right now, from each, returns a single IP address. That's it. Bam.
Because we're still in Ashburn. No, this is the Pipedream IP address. This is it. So this is one. And if I do changelog.com, this is the DNS that updates the quickest. That is the Google DNS. That's very fast. The Google DNS knows we're up. There's also, I use DNS Checker. Let's see if it's still a thing. It's been a while since I've used DNS Checker, a service that checks, tries to resolve the IP address from a couple of locations, more than a couple. Let's see.
DNS checker, let's go changelog.com. We'll do CDN as well. changelog.com, this is the important one. So all the DNS in San Francisco, it is. I'm going to make this a little bit bigger for everyone to see. We can see all the locations. So all of it is the new IPs. We don't see any of the old ones. The world knows about what we did. Yeah, I think so. They took notice. I think it's been good. I think. Yes, question.
That is a flight.io IP address. Let me show you that. So we have just IPs. You can see why IPs the CDN is using. And you will see that it's this IP address, which is a dedicated one. That's the IP that we're using.
That's it. Should they clap now? Only if you want. Thank you, thank you. Mind-blowing. We did a thing. We did a thing. Thank you. Thank you. Thank you. Because you were a part of this. And we did it with you and for you, and this was so good. Thank you very much. Thank you. So a couple of special thanks. We've already thanked our highly folks. Thanks to a couple of Denver liaisons. Dan Moore. Is Dan here? Hey, Dan. How are you? Dan. Thank you, Dan. And Kendall Miller. Did Kendall leave? Or is he still sticking around? He left. He was here earlier. Thank you to you two for helping us find this theater, for helping us.
Connect with Nora again after all those years. We don't live here. And so I had no idea what I was doing or who I was talking to. So always awesome to have locals and friends willing to help out and make it awesome. So that's our show. One person didn't get thanked. Yes. Jason. Oh yeah, Jason. Where is Jason? Come here, Jason. Jason, come here. Jason's our editor. He's behind the scenes, but he's very much part of this team. He doesn't get seen. He gets mentioned a lot. But critical, critical, behind the scenes here at ChangeLog. Thank you, Jason. Thank you, Jason. Thank you.
All right, anything else, Gerhard? Thank you all. I really appreciate it. Thank you. Really appreciate it. Thank you. Thank you.
There you have it, our first ever Kaizen Live, but I'm pretty sure it will not be the last. You know an idea has legs when you're already brainstorming version two before version one is even out there, and we certainly were. Stay tuned for more. This particular episode is better on YouTube, and we have more videos from the Oriental Theater coming soon, including BMC's live beats show. Subscribe there for clips, shorts, and more goodies at youtube.com slash changelog. And of course, join our totally cool, totally free hacker community Zulip at changelog.com slash community. Have a great weekend. Share changelog with a friend or three who might dig it, and let's talk again real soon.