Welcome to changelog and friends. A weekly talk show about how good systems become bad systems. Thanks as always to our partners at Fly.io, the platform for devs who just want to ship, build fast, run any code, fearlessly at Fly.io. Okay, let's guys. Well friends, I don't know about you, but something bothers me by getting back.
I love the fact that it's there. I love the fact that it's so ubiquitous. I love the fact that agents that do my coding for me believe that my CI, CD workflow begins with drafting tomofiles for get-of-actions. That's great. It's all great.
Until, yes, until your builds start moving like molasses. Get-of-actions is slow. It's just the way it is. That's how it works. I'm sorry, but I'm not sorry because our friends at NameSpace, they fix that.
Yes, we use NameSpace.so to do all of our builds so much faster. NameSpace is like get-of-actions but faster. I'm like way faster. It caches everything smartly. It caches your dependencies, your Docker layers, your build artifacts so your CI can run super fast.
You get short of feedback loops, have your developers because we love our time and get fewer. I'll be back after this coffee and my build finishes. So that's not cool. The best part is it's drop in. It works right alongside your existing get-of-actions with almost zero config.
It's a one-line change. So you get speaker builds, you get to let your team, and you can finally stop pretending that build time is focus time. It's not. Learn more, go to namespace.so, that's namespace.so. Just like it sounds, like it said.
Go there, check them out, we use them, we love them. And you should too, namespace.so. How else would you learn? Let it crash. Exactly.
The best things happen when things fail. Seriously. If it's in a controlled way, right? I think that's like something which isn't said, it's implied. It has to be a controlled failure where you have the boundary and things will not blow up.
I mean, they'll blow up it like the fireworks sort of blowing up where it's a controlled explosion. Yeah, right. Tiny little crashes to learn from. Welcome everyone to Kaizen22 with the incomparable Gerhard Lazu. He's here to let us know how he lets it crash.
It's like that song, let it snow, let it snow, let it snow. Only you know how to replace. Hey, Gerhard, how are you? Hey, Jared, I'm good. Thank you.
Had a great holiday. It was a great couple of weeks where I've managed to finally disconnect. It's been, I know like 20 years since I had two weeks completely off. Even my holidays are only a week. So this was very different, very enjoyable, and I feel so refreshed.
So I'm firing on all sentences. You unplugged and now you're plugged back in. Too much. Plugged in. Plugged in.
I stopped it and I started it and it's like brand new. It's like laid, man. I'm like laid over here, man. Plugged in plugging in. Smell the scent.
The fresh new years scent called 2026 where... Some people are going to say this is going to be the best year ever. I've heard it said. What do you think? They keep saying that.
I'm excited about them. They said that's about 2020. 2020, we have to admit it was off to us. A killer start. I mean, it was really going well.
Right. Putting tended. It was fine. It was COVID. I'm pun intended.
Another start. That was 2020. 2020 was the year of COVID. And everyone's like, ah, this is going to be the best year ever. And then we had three years of misery.
So I think... I just want an easy going here. You know what I mean? Last year, 2025, first of January, we were building shelves. We were doing studies and whatnot.
And the whole year was full on. It was nonstop. Every week there was something significant happening. And this year would be a bit more chill. A bit more meaningful.
So that's what we're thinking. But how about you Adam? How are your holidays? My holidays were filled with barbecue and good times. Wow.
Even in winter. So barbecue never stops. It doesn't seem to stop some text. Actually, just to shower you all with a few of my picks from my most recent barbecue adventures, if you're in Zulip, go to the general channel, look for barbecue with three bangs after it because why do one bang when you do three? Bang, bang, bang.
Some recent ribs. My gosh, my ribs method is on point. My spatchcock chicken method is on point. No one is disappointed at my barbecue joint. Very nice.
Look at that. I had some meat on this slide. That's what happened to the real type. Wow. This is like, yeah, this is intense.
And that's again, just to be clear, it's Adam's barbecue. Okay, so like no joking aside, we're talking about barbecue. I mean, I'm not, well, I think we have to leave you there. Let's move on. I think we have to leave you there.
I didn't show a burger, but I do make a mean burger too. Thank you for assuming that is something I do rock really good. My smash burger is wrong point. Very nice. Very nice.
I'm looking forward to that. So one day, my favorite Christmas tree. This is what we look like. What is that? And for those that are listening, it's a networking cabinet.
There's lots of blue lights flashing. This is happening in the loft. You have many terabits off network throughput. There's some switches, there's unify, there's micro tick. This is maybe five years in the works.
And every Christmas I take time to improve it little by little. So this year, when really crazy, I really like the whole thing. I did, I read it like a whole, for example, DHCP network, VLAN. Oh man, it's beautiful. Your villains are beautiful.
They are. I want to be a guest on your network, man. I'm gonna get blind from everything, okay? Yeah. Well, well, well, there's like a big story happening in the background.
And it is going to be, I think this is amazing. This will be the best network that I have run like in my life. But the blue and the darkness and it's like, that was like one more Christmas tree in our house. And this was it, where I would just go and tinker for a few hours in between Christmas dinner and turn all the Christmas festivities. So it was nice to spend a bit of time tinkering with hardware.
And I'm sure that many of you listening, when it comes Christmas time when things start quieting down, you get like the little projects that you didn't have time for throughout the year. And then you have some fun. So I'm wondering, did any of you, anything fun this Christmas, but nerdy fun, that's what I mean by that. Nerdy fun. Well, I got upset with something and stuff.
And so I decided to just let it roll. You know what I'm trying to say? I got upset with the amount of RAM usage on my machine. And while I like the application, I was like, you know what, I'm just kind of tired of having four gig, I think it was, I was like 1.2 gigs of RAM being used by CleanMyMac. Fancy little utility application helps you tune and pay attention and stuff like that.
And I decided to remake it. And I was like, sorry, I made it. It's called MacTuner. I know there used to be a MacTuner.com, which was I think a Mac magazine, I believe. But MacTuner Fit, I'm like changing it.
But for now, it's called MacTuner. It does all the things, all the things. Analyze, clean up, uninstall, and not just that fake uninstall, the real one, where you get the dirty dirty sounds. You know what I'm saying? The dirty is all the dirty is all the dirty is all the dirty is all the dirty burgers.
I'm the dirty burger. I'm thinking from the dirty here. I'm guessing about this in a second game. I made a little utility that's from you for now. Soon to be open source though.
Soon to be. It will be soon. I mean, why not, right? Sure with the world. Well, I didn't create a MacTuner, but I found one.
I also was thinking CleanMyMac, you know, like how long am I gonna run this thing? And the answer is, as long as I ran it, because I'm done now, I found a tool called Mole, M-O-L-E, which is a command line Mac OS Cleaner that does like everything. So you may get some competition here at it. Maybe you can come on. Like for those blows down.
It's like, here's why I'm better than Mole. It's got two E, it's all command line based, it does cleaning, optimizing, uninstalling, daisy disk, you know, explorer, all from, yeah. I'm feeling it. I'm feeling it. You're getting ready to start.
You start to set? You know, just change these mind, I'll open source them. Here's your domain name I do better than Mole.com. You know, like I'll grab. That's good.
I could do that. So I've been using that. I'm very excited because who doesn't want to just have all the things right there in their command line? And yeah, I didn't spend any tokens on it. I don't have some tokens involved, but he's also works the exact way he wants it to.
Yeah, absolutely. Mine lever does some recast stuff as well. It's kind of cool. So open source sucker. One day, which day is that?
Not today. But it definitely does right now. It's gonna be a one day, but it's gonna take. There's a bigger launch awaiting is all same. There's a bigger launch awaiting, so I'm gonna open source some things.
Been using App Cleaner for many, many years. Now there's no TUI, there's no CLIs, just like a regular app. It's a really old one. Which you can drag and drop onto it, right? Pretty much, yeah, and you also have a list of applications, but it's so always difficult to find it these days.
And it has not been a very long time. So I will check them all out. Moles are really cool. We're installing Moles and you're done. So you can check it out right here when we're talking.
And I liked App Zapper. And I think App Zapper doesn't exist anymore, but the cool thing about that was that it literally make the zap sound. Has it? Yeah, you drop your app on and zapped it. And I just like that sound.
That's the only feature that your application needs to have. I don't know if it's zaps. More than that. So there you make it zap. This is our tag one actually.
Make it zap. There you go. I think that's a very good debate actually. Everything. What about you?
Or besides your Christmas tree, did you? I will come back to that. I will come back to the Christmas tree. Yeah. Oh, man.
Oh, yes. It's like I have to tease them and be very disciplined because there's too much stuff. So I have to be very careful because it will be an hour and I will not shut up talking about this thing. I mean, it's just like a anyway. So we will come back to that, I promise.
OK. Last time when you finished Kaizen 21, this was one of the last thoughts that we shared, which is what's next. So, BAM, remember BAM? That happened live. Oh, oh, and crashes, auto memory crashes and a bunch of other things.
The good news is that only one thing happens. Oh, I'm talking about. But this rabbit hole is really, really deep. OK. All right.
Take us down the rabbit hole. Oh, oh, M out of memory. Who remembers this book? Erlang and anger. Erlang and anger.
Stuff goes bad by Fred Hebert. Furd.ca. Now I remember learning some Erlang for great good, but I did not remember this one in particular. So I'm not sure why. The other one in my radar because he wrote both of them in the but when did this one come out?
So this one, if I look, I just switched to the browser 2016, 2017, while he was still at Heroku. Heroku, those were the days. So about 10 years ago. And Fred, I mean, he just, like, if you don't know his blog, I mean, it's just amazing. I'll just click it very quickly.
Just have a look. I think it's one of the best blogs out there. There's so much goodness here. So much, but one of my favorites is Q's and Q-ing and how Q's don't protect from overload. So Q's don't fix overload.
And this is so relevant to today's conversation as well. But there's a lot of stuff in the Erlang ecosystem. And there's many, many things that Fred wrote over the years that are so relevant to today. So if I click on download PDF, right? By the way, this is like a, it's amazing.
This book is open source. You can download it open source freely available, creative commons license. And I'm going to make this a little bit bigger so we can see what's happening and if I search for let it crash. It's page number one. It's in the introduction.
Page one. Page one. And this idea of let it crash really comes from the Erlang ecosystem. It's very well renowned there because of how the Erlang VM works and how all the processes and the supervision trees, just it was built this way. And we know a thing or two about Erlang, Jared, right?
Because in location, Elixir, the Phoenix framework runs on the same principle. I know a thing and you know two. So that's how I get to a thing or two. And Adam, I'm sure he knows the big one. But we don't know what he's going to share.
It's the point is, the point is, when you think about let it crash, Jared, in your, like from your development experience with Erlang with Elixir, Phoenix, is there any situation, any moment where you could experience it and you realize, huh, that's nice. When I let it crash. When you let it crash. Well, it's nice that the beams need to handle a lot of the problems with like in a crash. You know, just goes again, and there's a supervision tree and things watching each other.
I don't have to think about it very much. I can't think of like an instance in development where I was like, this is really useful, but I'm sure you've become up with one. Yeah. So you know when you write codes, we tend to write code very defensively, typically try catch. So you feel like you need to account for every single scenario.
And the latest crash philosophy is about not preventing failure, learning from it. And what it means is you need to have a context where it's safe for things to crash and the overall system will still remain stable. So how can you build a resilient system and really about resiliency, where the core of the system will remain running and the system as a whole will remain running, even though parts of it may experience failures. But those failures will not bring everything down. And that's really important.
So fewer try catch blocks, don't code defensively, let it crash and separate the code that solves the problem from the code that fixes the failures. And the more you can lean into the framework or the VM or whatever you have the system to deal with failures, the better off you are to focus on the things that are unique to application. And Erlang is well renowned for that. Kind of the opposite philosophy that Go-Took as I write some Go-Code and I write some Elixir code, where with Go it's like handle every error condition right after you potentially raise one. And make sure that it's, there's no error.
And if you're not dealing with it, then you're not writing robust software. And the other philosophy is let it crash and deal with it elsewhere. I think they're both legitimate, the panel which are building. Agreed. Well, in our case, we had a lot of crashes to deal with the deal.
What we are going to have a look at is all the times that the pipe dream has been crashing since our last Kaizen. So since Kaizen 2021, which was October 17th, we had a lot of crashes. And there's a certain property about the system and this is varnish specifically that made these crashes pretty okay. And the property which I'm referring to is when you start the varnish, D, the demon, varnish itself runs as a thread. And you have many, many threads that do different things.
So when we had these out of memory crashes, all that happened, the thread was killed. Which means that the system is a holding crash, the VM didn't, the varcracker VM didn't crash, the application need to restart. It was just a thread that was using too much memory and it's restarted within seconds. As in maybe two seconds and everything was back to normal. Obviously the cache was cold, but it was good.
I mean, the VM and that's why the memory looked a bit interesting and that doesn't release all the memory. The VM doesn't restart, there's loads, there's not many hangs. It restarts and it crashes really, really quickly. So that's a nice property. Well that confused me.
So how to fly know about it then, if it's just happening inside of varnish. So it's looking at the process. It has looking at the process ID, which process uses the most memory and it's the same process that's asking for more memory. So just basically say, it will just send the signal to that process and kill that process. But that is just a thread that maps to threads.
So varnish itself didn't crash. It's just a thread that maps to a process ID that crashed. And then it was restarted by the varnish demon. Okay, so where's fly involved in that? Because fly's aware because I see all these fly notices and I get the fly emails.
Right. So fly's aware that there is a process on the machine that is using too much memory and more memory is being requested. And then it looks like, okay, which process do I kill? I mean, in this case, the process with the most memory will get shot and will get killed. So fly as a platform can actually reach in and kill that process without killing the machine, rebooting the VM or the firecracker, whatever.
So the fly platform, it integrates with that functionality, which is a kernel, it's a Linux functionality. Like that's why I can auto memory crash would happen even if you have a single machine, you have too much memory, you don't have any swap. How do you basically give more memory when there's no memory left and when the system is becoming unstable? So then you get just a single process which gets killed. In fly's case, they surface that.
They surface the fact that was out of memory crash, there was an auto memory vent and they send you an email when it happens. It doesn't mean the machine had to restart. It doesn't mean that he stops serving traffic. It just means that there was something that just had to go away because it was using too much memory. I say too much memory, it's obviously a bit more complicated than that because something was asking for memory.
The kernel didn't have any more memory to allocate, so it just had to look at what needs to be killed so that I can allocate more memory because something is using too much memory and it just happens it would be this process and this thread. So how many crashes do you think that the pod dream had since guys in 2020, 2021, since October. So we're talking about three months, maybe a bit more than that. So Gerhard has presented us a multiple choice quiz. AS20B is 40C is 80D is 160.
Now I know that I personally receive an email and we're trying to just happens. And so I have a little bit of a feeler into this. I delete them so I can't go do a quick search. Adam, do you get emails when these fly things crash? I don't.
Okay, good for you. I'm not saying that if I do, they're in a box that doesn't get looked at. You can save it on some email bandwidth. But thank you. No, because when we send emails, let's go back to this one.
If I click on this one, let's take this one and you can see everyone that gets an email. So I'm just going to make this a little bit bigger. So you can see services, Gerhard, I do get it. So it must be a filter. It doesn't look at it.
Super human saving me. Nice. That's okay. So what do we think? Good thing other people are looking at it.
So I think it's not an item problem. That's the thing. I'm on that to the side. So I feel like 160 is too many. I don't think I've gotten 160 emails since October on this particular thread.
20 feels like not enough. I've certainly got more than 20 emails. So I'm between 40 and 80. And I'm going to think that, gosh, that's a tough one. I'm going to go with 40.
Adam, we do it 40 as well. Oh, I got it. Yes. 43, exactly. The price is right.
That price is right. All right, cool. Yeah. 43 crashes from October to December for the end of the year. Yeah.
I think obviously there were like periods when it had quite a few. So if we were to think about what could be happening in varnish, that it's running out of memory and crashing. So this is us trying to think about the sort of traffic that we serve, trying to think about everything. I mean, now we see every single request that hits change log, the CDN as well, and it's a lot of requests. Yeah.
So there's something in the system. There was something in the system that was using way too much memory. And as a result, the thread in this case was crashing. I mean, I could guess it, but I might even have some insight. So should I just say it or do I have any guess?
I mean, my guess, based on all sorts of emails flying through, but already I would have suspected that we just have too many large files, these 60 to 80 to 100 megabyte mp3 files, load in memory, fly in every which direction. And you just can't load up that much memory without sort of fancy free mechanism. And it's just trying to hold all these mp3s in RAM, I think. And I just can't do it. So that's my guess.
Yeah. That was a good guess. And I think the next question is going to be to the audience. Because we know too much. How are they going to answer?
Well, just think about it. We will give some time for people to think. OK, we'll be like a delay here. So if they have what to call the feature where you pass silences on, they're not going to have any time to think about this. Right.
OK. Quickly, turn that feature off. Give yourself some kind of thing. Go ahead. Or pause.
We can also say pause. Now is a good time to pause. And then what could be the problem? So you're right. They're all those large files.
We had all the mp3 files, many, many mp3 files. They're large, all trying to be cached in memory. And that was a problem. So what is many? Well, we have thousands at this point, or mp3 files across all the podcasts, since the beginning of time, large, large means anywhere from 30 to 40 megabytes to 100 plus megabytes.
So that's, I mean, just think if you had to load the thousand files, take 100 megabytes, that's a lot of memory that you need to have available. And the problem is that once you store these large files, as we discovered, you get memory fragmentation in that. Imagine that you have all the memory available. You keep storing all these files. And at some point, there's no more memory left.
So what do you do? Well, you need to see what can you vict from memory so that you can store the new file. So imagine that you've vict to feel those objects. But maybe they're big enough. And you haven't vict to them fast enough.
So then you have like this big file that can't fit anywhere because the sizes like the holes that you have in memory aren't big enough for this file to fit. And there's no refragmentation or nothing like that. That runs in the background, which means that even though technically you kind of would have space in the memory for the specific files, you may not. And then it's it's a it can't be stored memory. Now, the thing in varnish is actually called a kid U not n underscore l r u underscore nuked.
So I think the connection to the nuke and to the book and the let it crash is right there. So l r u nuke basically it's like a forced eviction. So it's an event where an object has to be evicted from the cash just to make room for a new one because the storage is full. So you can see how many times this has happened. And that's like an important metric that if we look at, we can see we had too many of these events, right?
Like many objects are being nuked from memory to make room for new objects. But sometimes they wouldn't fit. So how badly did it nuke because we can measure this or we can we can look at this. And this is what it looks like from a memory perspective. So you can see that the instance was running about maybe four gigs of memory.
And then we had a massive spike within minutes, like one or two minutes to 16 gigabytes. So that's a lot of data that had to be fitting memory. And you can already see where this is going, scrapers and bots and LLMs and we have so many things happening. And then you can see the memory went up. The thread was killed.
The child was killed like the varnish wants to the memory came down again. And then it went up again. So the graph that we see here, we can see the first spike, just like maybe a minute apart. The second spike, another crash, it took a little while for it to restore. We're talking maybe 10 seconds.
And then we stabilized around 10 gigabytes from a CPU perspective. We got like 100% CPU utilization when this happens. Like everything is full on, everything like the instance is really struggling to allocate and deallocate and free up memory. And more importantly, we have a lot of traffic flowing through. So how much?
2.29 gigabytes or gigabits, specifically 2.29 gigabits per second. Per second, exactly. And these happen so quickly have like a huge rush of traffic coming in. And then nothing. Well friends, I'm here again with a good friend of mine, Cal Galbraith, co-founder and CEO of depot.dev.
Slow builds suck, depot knows it. Cal tell me, how do you go about making builds faster? What's the secret? When it comes to optimizing build times to drive build times to zero, you really have to take a step back and think about the core components that make up a build. You have your CPUs, you have your networks, you have your disks, all of that comes into play when you're talking about reducing build time.
And so some of the things that we do with depot, we're always running on the latest generation, or ARM CPUs, and AMD CPUs from Amazon, those in general are anywhere between 30 and 40% faster than GitHub's own hosted runners. And then we do a lot of cache tricks, both for way back in the early days, way for a third depot. We focus on container image builds, but now we're doing the same types of cache tricks inside of GitHub actions, where we essentially multiplex uploads and downloads of GitHub actions cache inside of our runners, so that we're going directly to blob storage with as high a throughput as humanly possible. We do other things inside of a GitHub actions runner, like we go off portions of memory to act as disk, so that any kind of integration tests that you're doing inside of CI, that's doing a lot of operations to disk, think like you're testing database migrations in CI.
But using RAM disks instead, inside of the runner, it's not going to a physical drive, it's going to memory, and that's orders of magnitude faster. The other part of build performance is the stuff that's not the tech side of it. It's the observability side of it, is you can't actually make it build faster if you don't know where it should be faster. And we look for patterns and commonalities across customers, and that's what drives a product roadmap. This is the next thing we'll start optimizing for.
Okay, so when you build with depot, you're getting this, you're getting the essential goodness of relentless pursuit of very, very fast builds, near zero speed builds. And that's cool. Cal and his team are relentless on this pursuit. You should use them depot.dev, free to start, check it out, one line of change in your GitHub actions depot.dev. So why is more traffic coming into the instance than going out?
So this is the traffic of the instance is receiving. So we're receiving 2.29 gigabits, which we're only sending 145 megabits. Now is good time to pause and think about what is just happening. Yeah, don't skip silence. So let me say the instance, we mean the varnish instance.
The varnish instance, yeah. Like the other instance between our end user, whatever that is, our users, and our application. Yeah, well, actually, and our cloud flare, not our application. All our back ends and we have a couple of back ends. But in the case of mp3 files, it's our origin.
So varnish is receiving a bunch of data and sending back significant in order of magnitude less data. And what's it receiving? I don't know, man. I mean, I guess we'll be like, we're uploading mp3s. Now that's going to go straight through the app to R2.
Just a DDoS. I mean, what is it? I don't know. Yeah. So it is a DDoS, but it's specifically downloading mp3 files or starting to download mp3 files, but never finishing hanging, right?
So you get like all these requests for mp3 files for large files, varnish is going and fetching them as quickly as it can. So pulling all this data in so it has in memory, but the client is never around long enough. Yeah, exactly. So they basically abort. But varnish is still pulling it in all the data.
Now there's a property. It's called bresp.doString true. So what this does, very weird thing, it tells varnish not to buffer the entire backend response if the client is slow, right? So I'm not going to fetch the entire mp3 file if you only want the first, I know, minute or two or a range or something like that. Now this is owned by default.
So by default, that's how varnish behaves. So we wouldn't need to enable this. But if the object is untachable, it cannot be stored in cache. See where I'm going with this? Memory, don't constort in memory.
So keep pulling these files over and over again, and maybe even just fragments of them. So even though the client never receives them, you may be pulling hundreds of files. And the client just goes away. So you're not pulling the entire file, which are still pulling enough and not able to fit it anywhere and just becomes a mess. It reminds me of the 90s when you used to go gene shopping, right?
And you go into detail, the dub and fish, which I would never, you know, never shopped at. But let's just imagine I did, right? I had to go in there and be like, I like all these genes, get them all, I'm trying them all on. And then I just bounced. Yeah, the person goes to collect them all, they come back and you're not there.
Here's the dress room full of genes and Adam's gone by the same. It sort of sounds like you're speaking from experience. It was just like a prank. It just made up is now, you know, just creative like that, you know, on the fly. Free to be.
Well, that's a good one. That's a good one on the fly. Yes. So it is on the fly. I know.
Boom. Well, what can we do then? What's going on here? Exactly. So this was one of the things which I had to deep dive and understand what on Earth is going on.
Like where do we store like what's happening? So there's a lot, lot more that went into this pull request. It's pull request 44. I'm calling it the elephant in the room. I'm going to switch to the browser.
Just have a look at that. So the title of the pull request is storing mp3 files in the file cache. But that's like the tip, right? Like the most obvious thing is, well, you either have lots and lots of memory to give varnish, which honestly will be impractical in the sense that would be way too expensive to store all these files in memory. The next best thing is to have something like a file cache.
And by the way, we're talking about open source varnish. That's really important. Like anyone can use this. Anyone can configure this. You can configure a file cache, which will basically pre-allocate a file on disk.
And that's where these large files will be stored. Pull request 44. The one that we're looking at is in the pipe, play repository. That's what this adds. But there's significantly more stuff in FM going to go.
There's quite a few files. I highlighted a few. So I'm going to look at this one. So it's not just that. You also need to tune, for example, Threadpools.
You need to tune the minimum, the maximum. You need to tune the workspace backends, like how many memory structures get allocated. You need to configure the new climate. And there's a couple more things that we had to go through just to make things stable. Now, I just going to very quickly mention these things.
It can go and have a look at pull requests to see what else went into it. So this was a one file. The other one was the regions. That's another thing. Not all regions would suffer from this.
So you don't want to allocate too much memory or too much CPU to regions where maybe they don't get a little traffic. You would think that this thing is easy. But oh man, I have a surprise for you. You can't mix and match sizes easily in fly. So you can't say create application groups.
This group will be like the small group and that group will be the big group. It's just one application. It's not straightforward. So you have to, again, this is how I solve it. Maybe someone listening to this will tell me, hey, get hard, you're wrong.
I would love to know that. Seriously. The way I solve it is we deploy in all the regions, because you specify the size once. So say, my starting size is the large instance type. It has a certain number of cores, a number of memory.
And by the way, the disk is the same in all of them because that's like another problem. So we will sidebar that or put a pin in that. So when it comes to the initial deployment, you deploy the one size across all the application instances, and then you go and need to check to see which instances should be scaled down. So that you have the capacity, but the regions that don't need the capacity can just bring them down. And you do like a rolling deploy in that you replace one for one.
You have plenty of capacity to handle the traffic while instance are being rolled, all the good stuff. So we have hot regions and we have cold regions. And there's quite a few things here. Again, if someone knows how to do this better, I would love to hear about that. And we have Tomo with the primary region.
There's a couple of things here. We'll come back to services and HTTP services. That's a fun one. We'll leave that for a little bit later. Fly just.
We can see how we do the fly CTL deploy. We disable Have because we want only one instance per region. We have 15 regions in total. We specify the CPUs, the memory, all the good stuff, including environment variables. Oh, that's another thing.
We need to adjust the varnish size based on the memory that instance has, right? We need to say like, hey, varnish, you get 70%. And that's the other thing that this does. The other thing for the file size, you can't take up the entire disk, we tell you based on the disk that we provisioned how much space you should use from the disk that gets created. There's a scaling there.
So that's another good one. I'm going through pull because there's anything else. Oh, man, this was a pain. So recreating, like writing tests for this. Everything is tested in the sense that which requests would go or which, basically, which files would get cached in the file store and which files will be cached in the memory store.
So how do you write the tests? Some varnish logging is included. You have to have anchors. There's quite a few things. So that's assets back end of VTC.
And part of this is a huge refactoring. So if you look at the lines of code, I wouldn't say to that many. 1,500 were added and 1,470 were deleted. So not much changed. I mean, the net is 13 new lines were added.
But there was a huge massive refactoring part of this. So there's again, this was, I think, two, three days of figuring it out, trying things, refactoring things. And if you think, that's an LMM can help you. Well, you try this. And it takes longer to go through those iterations than if you know what you're looking for.
It tends to be easier. It's very dense, very specific, very difficult to make sure that it's doing the right thing. But it's all there. We have the mock back ends. We're reusing things.
We split the VCLs. By the way, you finish like the splits easier to reuse them. So there's quite a few things there. Now, this is Kaizen. So we are wondering what improved after all this work.
Right? We rolled it out, what improved. And to answer this question, we need to figure out which region is the busiest one. So out of all the regions that we serve, we have 15 in total, which ones get the most traffic as those hot regions. We're looking at the fly, the Grafana dashboard for our fly application, the PIPE, the current one.
And we can see that SJC, San Jose, California is a red, nice big red circle, which means it has the most traffic. And also, NRT, which is Tokyo. Apparently, we're big in Japan. Yeah. And Europe, there's quite a few.
So if I'm going to pull this down a little bit, let's see. No, I wanted to go here. New continent. Are we big there? The new continent, Australia?
That was a new one. That was a new one. Well, what's it called? Which is a new one. I don't know.
I thought you were going to get the joke. There was a new continent being announced. Narnia? Maybe. No, no, no.
What's the closet? Right now, even this loss, even this list, basically, if you think about it, it kind of makes sense, right? It's US East, US West, Europe, but we have quite a few instances in Europe. We have four. It's more geographically spread in Europe.
And we have Asia. So these are like the big ones. Australia, Africa, and South America, they're not as busy. They are like, least busy regions. Cool.
So which instance would you like us to have a look at? So I have a queue right here. SJC, baby. Let's go. Let's go.
SJC, baby. I'm using two flags, dash S, which is a short one for dash, dash, select. It'll prompt me, which instance I want to select. And then I have dash C, capital C, it's different than lowercase C, they do different things. I give it the command to run.
It's varnish.1, which will give me all the statistics from varnish at a point in time. So since instance was running, I will select SJC. There you go. And it will give me all this data, which is like all the counters that varnish is incrementing is keeping track of different things of the origins, back ends, the memory pool, the disk pool, the lock counters, there's so much stuff. I'm really, really impressed how many things varnish has.
So this is what we need to do. We, because AI, right? We're going to copy all of this. We're going to ask AI what it thinks of this in a moment. Okay.
It's just too much data here. So let's be serious about this. So question to you, which is your favorite AI, Jared? Which one do you use? Oh, I don't like any of them.
I would probably start with Claude and then I would go to Grog and then I would go to Chet GPT third. Okay. So Claude, which one, which version, which model? This man, give us the opus. Okay.
So we're looking at advocacy of AI is something I've been using for a long, long time. It allows you, I'm only paying $10 per month for it, not sponsored, you know, not to feel it in any way. It's just something that I've picked for myself and I can basically pick any model and I can just run this. So I have something prepared. So I'm going to drop this.
It's all data and we're going to read through something that I prepared ahead of time. You pre prompted this. I pre prompted this. Exactly. I guess engineering is prompt for exactly the prompt.
Not really. Yeah. So we're going to read it. And in the meantime, Adam will think about his favorite element to try and I have mine. So try three LMS to see what they say.
So I'm going to read the prompt now while everybody thinks. Oh, you should be using whatever LMS should be using. You are a varnish, seven expert. You need to prepare four distinct responses and be explicit about the person that you're addressing. One, a seasoned businessman that has been living and breathing infrastructure for the last two ideas.
Be precise, think deeply and approach the setup from a hardware perspective. Two, an Elixir application developer that embraces airlanks, let it crash concept. You need to give it straight, give it fast and keep it relevant to their application. Is the app in the 90 back ends? As is and fees are important, but less relevant.
Cloudflare two, three, the business person that is selling this thing, they care about costs, efficiency and simplicity. Keep it high level and relevant for someone that doesn't care about the tech, but cares about the outcomes. And four, the audience of a podcast where this is being discussed, make it general, relatable, and fun, make it analogies, keep it light and engaging. I have found too many types. We don't want to make it too fun.
That's too many fun. That's right. Well, now that you understand your audience, please analyze the following varnishnet output for the SJC look. I already knew that you would go for the big one. I have the idea focus on things that work well, things that could be improved and anything else that you find interesting.
And by the way, ignore the synthetic request. It will keep mentioning these. I get so fed up with this, we have health checks that run every five seconds, so they are normal. So, okay, I'm going to copy this. I'm going to run this and I'm also going to open a new window for Adam.
So, which column should we pick at them? Which is your favorite? Model? Model, yeah. We just used it, but I'd probably back up to like Codex.
Codex is like GPT-5, Lazy C5-5-1-1-5-2. There we go. So GPT-Co-X, my favorite one is Gemini. So I'm going to drop it and let's see how do they compare. You're in a different tab now, so advocates can't do Gemini.
It might, but I have my own pro account. So that's something else. I use quite a video, I use Nanobanand, quite a few things. Transcrapes is all part of the package. So it can, but that's what I prefer.
Cool. So, Claude, Opus 4.5 for the season system. This is you. This is me. This is me, exactly.
Thank you for noticing. So, what's working well? Rock, solid stability. So, by the way, the instance has been running for 5.4 days. We had all these improvements shipped and we are able to observe how our busiest instance works and that's what this is basically.
That was the window moved. So, after 5.4 days, zero child panics crashes, zero threat failures. This is important. It means no threat died. No threat had to be restarted.
Everything is healthy on this instance. It didn't crash. So this instance didn't crash. Zero lock contention across all subsystems. Your CPU cache lines are happy.
Excellent hit ratio. 93%. We like that. We read like that. We have back end connection pool link with a two to one reuse ratio and memory pressure is minimal.
One, three, two LRUs in the last five days. LRUs. So very few objects had to be removed from memory. Threat pool property, 300 threads, zero queueing, zero drops. That's perfect.
Every scene investigate. Disc storage allocator failures. We have this C fails. We are hitting storage fragmentation. The disc is 97, 97% full.
We have 48 gigabytes used. That's how many mp3 files are stored. By the way, how many mp3 files total do you think we have? Size or file size? Size size.
Well, if we had 1000 episodes at 100 meg each, which neither of those things are true, that would be 100 gigs, right? So, 100 is too big, but 1000 is too small. I'm gonna say 80 gigs. I guess. That map checks out.
I don't feel like a terabyte, but that's probably raw way files versus not. But all the files that we store in R2, and this includes all the assets, but we know that the mp3 files are the biggest. It's close to 250 gigabytes. We may have some duplicates. I don't know.
I haven't checked, but that's how much files we have in R2. Yeah. Well, we also have plus plus for the last couple of years, which means every episode has two files, not just one. So that makes sense. So we should go higher.
Now we use this in every single region. So maybe we want to reduce a number of regions. But I think in the third category called super hot. Super hot. Yes, it's like SJZ and Tokyo.
That's possible. Yeah, there's four, which we know they're really, really hot yet. Yeah, but honestly, this is happening across multiple regions and we'll get to some interesting things. So, okay, synthetic responses, the race hits all goods for the Lixr developer. And I think this is you, Jarrett.
Do you want to read it out? Oh, well, the field the R is barnished at two and it's job. Your app back in is well protected. You want to read the whole thing? If you want, I mean, how it's shielded.
It's 95% shielded. No failures. Zero back in failures. That's because of my code doesn't really let it crash very often. Exactly.
It's just internally, not externally. That's right. My thing is doing its thing. It is generating some un-cachable responses, but you know, we do have some that we just don't want to be cached. One fetch failure, negligible.
Yeah, I agree. I don't know any worry about that. And then the end says whoever wrote this is really good. I agree. And congratulations on such a great hire.
Yeah, I agree. I agree. I think behind this emotion and a bonus. There you go. All right.
For the business person, the caching layer is performing accidentally. Adam, do you recognize yourself or should I continue with this? 93% of requests never touch your servers. Massive cost savings on compute. Do you know how many requests per second the application is serving?
Like maximum. By the way, what's the maximum RPS for this amazing Elixir feeding application for the home page? Probably a lot. Gosh. Thousands?
Tens of thousands. Okay. Jared? 100,000? The database connection is involved.
Currently? Concurrently, yes. I don't know. I'd say a lot, not very many. For our home page.
12 requests a second. Yeah. 17. So if all these requests were hitting the application, we'd need so much compute to serve that. So much caching.
Obviously, we've removed all the caching. We're joking about this because we purposefully removed all the caching from the application. I remember that a couple of years back because we said this has no place in the application. The application gets restarted. We need to store this somewhere.
We need to cluster. It was just really messy to handle it at that layer, which is why we introduced this. Five plus days running without any issues. By the way, this is like the last deploy. So maybe by the next horizon, if we do no more deploys, we'll be able to see how it handles.
Zero phabies on the infrastructure side and three terabytes of data served to users. Three terabytes. So in five days, this one instance, served three terabytes. Without your application service breaking sweat, storage is getting full. So we need basically more storage for the podcast audience.
Oh, yes. They fun. Imagine a really good receptionist at a busy office. This large server is like having someone at the front desk who remembers everything out of 100 people who walk and asking questions, 93 of them get their answers immediately from the receptionist without ever bothering the experts in the back office. It's cool.
It's been running for over five days straight without a coffee break or a single mistake. That's not school to me, but let's go with it. It's served three terabytes of data that's like streaming about a thousand HD movies. This one instance streamed a thousand HD movies in five days. And the experts had only had to answer 7% of the questions.
The one quirk, the filing cabinet is getting full. It's like when your receptionists, their drawers are stuffed and they occasionally have to throw away old notes to make room for new ones, not the crisis, just time to get a bigger cabinet. Okay. I think there are lots of fun styles of 300 workers. I think that's too deep.
Good fun. Do we care about GPT or Gemini? We can only use one. We can only pick one. Let's get some good hotness.
Let's check Gemini. I will see. I was still thinking. Let's see. I think it's finished.
Maybe that's just close that. It's finished. I think it did. All right. So let's go up.
Slow thinking. I did like the thinking. I'm going to have to go as well. Show thinking. Show thinking.
Show thinking. Slow thinking. I'm like, oh, yeah, show thinking. There's a lot of love there. Anyway, we're not looking to that.
So the instance has been up for 5.3 days. The MGT up time. I like it. It's telling me which of those that long list of counters is important. From a system perspective, the threading model is perfectly dialed in.
300 threads across two pools with a zero thread limited and zero thread queue length. The kernel is in context switching unnecessarily and we aren't starting for execution units. However, we have a critical IO and storage allocation bottleneck in the SMF. The file. I'm not sure what Steve Dormens.
That's a weird one. I'm not sure. I'm not sure. Gemini knows what Steve Dormens did. The problem is SMF disk C fail is at 19,000 plus against only request a 73% failure rate for disk allocations.
The G space in the G space, we have only 145 megabytes while 40 gigabytes used. You're effectively out of contiguous blocks or hitting the ceiling of the pre-allocated file. That sounds right. Varnishes forced to fall back to SME RAM or simply not cache large objects. While SME has 2.1 gigabytes free, the disk is crashing.
The hardware, if this is spinning glass, your eye weight is likely spinning. It's not. It's NVMe. So yeah, there's quite a few things there. Interesting.
Is low relative suggesting the lock intention of the shared memory, low is negligible. Nice. You have low lock intention. The bottleneck is strictly at a storage layer. You just need to basically pick a disk.
The Elixir developer let it crash proponent. I like it. Your app backend is rock solid. Well done, Jared. Varnish hasn't seen a single backend fail or an unhealthy event in five days.
The let it crash loss if he's working, if the app is crashing, he's recovering so far as Varnish doesn't even notice. I don't think it is. I think the application is good. The good news is we're seeing an effective 74% hit rate. Now this is different, right?
We had 93% there, so I would like to understand the discrepancy. That's another reason why you should use more than one, by the way. And obviously, double check the data because sometimes they give you things which aren't true. The crash concern that is crashing again, this is not for the application developer. I think this is for the system in me.
I think the straight talk, your main synth is high. So we have health checks. There's a delta responses. So we have a lot of synthetic requests again, sorry, synthetic responses. This is again, like a varnish thing.
The business person, efficiency or currently serving of our traffic from varnish. I think I know what's happened. I don't think it's taking into account the synthetic requests. Those should be removed from the number of requests. I think it's a lot of the right number.
I think so, yeah. I think so. This means very for customers. We have cost efficiency. That's good.
I think this is the bottom line. I think this was the fun. But I think this is the library. I think we can stop it here. The library analogy versus the secondary analogy.
I think that was the best one. I got the barista one. I thought it was like a very good one. Oh, yeah. For Q&A.
For Q&A, the personality shift was very good. This is using books and whatnot. The library has some burden now. That's fun. That is fun.
So I think JB does get a bit funnier. The nightly feeds in the app are still humming along. Nice. So that's what we have. And that was only half the problem.
People friends, this episode is brought to you by Squarespace, the all in one platform for building your online presence. Whether it's a portfolio, a consulting business, or finally shipping that side project landing page. You just have a mean and do. But never get to. Here's the thing.
You mass produce code on the daily. You deploy new services, new infrastructure, new hardware, your versioning or APIs, your simmering all over the place. But when someone asks you about your own personal website, it's like, ah, I'm still working on it. Is that not familiar? Squarespace exists so you don't have to treat your personal site like a weekend project that never shifts.
Pick a template and drag and drop your way to something that actually looks good and move all your life. No wrestling with CSS. No. It's just done. If you do consulting or freelance work on the side, Squarespace handles the whole entire workflow, showcase your services, let clients book time directly to your calendar, send professional invoices and get paid online.
It's the boring infrastructure that you want to build for yourself. And for those of you out there who are doing courses or getting content or educational stuff, tutorials, workshops that intro to whatever series you keep talking about, you can set up a membership area with a paywall and start earning recurring revenue, set your price, skate the content and you're done. And they've also added Blueprints AI. This generates a custom site based on your industry, your goals, your stop preferences. It's not going to replace your design skills by any means, but it'll get you about 80% of the way there in about five minutes.
Here's the call to action. This is what I want you to do. Go Squarespace.com slash changelog for a free trial. And when you're ready to launch, use our offer code changelog and save 10% off your first purchase of a website or a domain. Again squarespace.com slash changelog.
That was only half the problem. So we're like, I was feeling good. I feel like we had an office. What else is the problem? Oh, man. This is like when all the fun begins. So you remember this, Jared?
Yes. And we three requests intermittently hang in Newark, New Jersey. This was our good friend, John Spurlock, who's been on the show before and is a podcast nerd. In fact, he runs OP3.dev and other podcast nerdery things. And so he really knows his stuff. And so he reports issues.
You know, I don't say did you try rebooting. I take it seriously. So I shared it with you. And he actually did some additional digging for us. Go ahead.
So in terms of you tested this, I think you had issues as well. So we've confirmed this for sure. I did. Like certain times, certain files, actually, it'll be all requests as certain times. I assume that that was that particular pop, as we could call them, or pipe in the pipe dream, was hanging. And then it would go away. And he actually had the same problem. He had a Friday
night deploy of friends and he was trying to listen to it on Friday. He couldn't get to it by Saturday morning. He could get to it. So it's intermittent hanging, very difficult to diagnose, very difficult I assume to debug. And then it just comes back to normal. I thought it'd be the out of memory thing.
Like it's just in some sort of fugue state until it reboots. And then it works again. But you go ahead. That's what I thought. That's why like a deep dive on this. This was November and of November,
beginning of November. So November was just trying to figure out what on earth was going on. Just like, I know, like from the sides, I didn't have too much time. But if you look at this response, there's quite a few things there. There's like the initial one, like an investigation trying to understand what's happening, giving a couple of debug headers, like a couple of extra headers that the request can be made of. Sorry, it can be can be run with. So we just get a bit more details.
Forcing regions as well. So there's quite a few things there. I was checking into that. This is Don McKinnon. He also had issues today. So he pasted some results. So thank you.
Thank you, Don, for adding this. This was helpful. So I'm still scrolling. I'm still scrolling. There we go.
Super helpful. I have confirmed that the request I've been hanging. You were getting the hangs this afternoon as well. This is three weeks ago. So this has been going on for a while. I had a deeper
and I found the problem. The problem was that in the fly config, we had the concurrency set to connections, not requests. So it's possible to configure an application. Again, you're configuring the fly proxy that sits in front of the application to limit how much traffic hits your application. So requests, how many requests per second should the fly proxy forward to your application before it stops? Because you don't want to get overloaded. So before it starts throttling, it starts slowing
clangs down. And then when you start, that's the only starts thing fly edge errors. Connections, you would use for something that has long running connections, like a database, for example. In our case, it's not a database. It's an HTTP application. So requests would have been the right concurrency.
I have no idea why I picked connections. It was the wrong one. But the effect was, as you can see here, we had 2,700 long running connections on that edge. So on that region. So this in this case, it was, I think, Orange one, I think EWR, right? So EWR was getting had like all these connections opened,
the clients were enclosing the connection, the proxy was full, no more connections could be forwarded to the application. Long running connections, there are usually clients which are not doing the right thing. You shouldn't have that many long running connections. So the problem was a misconfiguration on our side, which meant that connections like slow connections, long running connections were basically blocking other connections from coming through. So that was the problem there.
And I thought that was it, but that was more. So this is last comment last week. We now have a check that runs every hour. And what was interesting, and I'll talk about the check as well, we had responses bodies timing out in two regions. So 13 regions were fine. But even after this configuration,
there were two regions I had an EWR where when we were using HTTP two, and for some reason, this is important, when we're using HTTP two and the proxy, the fly proxy would see this, it would not forward the connection correctly. As in, it would start, it would like serve the response, like we could see the headers coming back from our instances, what we wouldn't get is the body. So the body would always be like zero bytes served. And we could see this happening. We could see the connections that by the way, they were opened, they shouldn't have been opened because the application changed. So these connections should have been dropped. There was something not
quite right. My suspicion is with a flight proxy layer, because when we were forcing, when we were forcing HTTP one, everything was working fine. And by the way, the flight proxy only talks our varnish instance, using HTTP one, and you can see that in the headers. So the proxy to the varnish was fine, but the client to the proxy was not fine. And HTTP 2.0 is a very complex protocol. There's
so many things which just don't work the way people would expect. So anyway, the issue fixes itself. So opening this. Yeah, that was very nice to see. And there was something my eye on us. How would you read this? My eye.
And there's there's someone on the fly community forum that was very helpful. They noticed that we had a misconfiguration in our fly tunnel, and we're using services as well as HTTP service. And this is very, very bad. So everything was happy. We could push this config. We had these two things together.
It was apparently creating some issues. And all we did, we were explicitly setting the idle timeout. And the idle timeout, that's the one where if after 60 seconds, the connection is doing anything, it will be forced to terminate it by the proxy. So that was important. So anyway, we made the change, we pushed the change. But even before we pushed the change, the proxy started behaving. And now,
their small request, 49, has like, we've right sized it. We made a few changes. I capture all the details, the configuration, the commands. It's all there if you want to read it. But most importantly, now we have a check that runs against all regions every hour on the hour.
CICD is using hurl. And what I'm thinking is, shall we try running that locally to see how it behaves? Because that's how we started it, like I was starting to do it locally. So on the left hand side, I'm back in the terminal on the left hand side, I am monitoring my internet connection. Remember that Christmas tree, this is related to that Christmas tree time at the top of the Christmas tree, I'm at the gateway, the core router, it's a microtic CCR 2004. Pretty good. 10 gigabits per second,
now my internet connection isn't 10 gigabits, but it's 2.5, which is plenty for this test. So we, every second is showing me how many packets and how many bits we're receiving and transmitting. We are recording everything's happening live. So you can see jumping right as river side, we're pushing more data to river side. Cool. So I'm going to run now, just check. And just check
by default, it's one of the commands, the just command that we have in the pipeline repository. And check all it does, it runs hurl with a couple of flags. It downloads an mp3 file, it downloads feeds, it basically connects to all the different backends. And it sees how quickly it can get data back. We're transferring about those quick, those eight seconds, I'm going to run it again, as I run this, pay attention to the left hand side, it will go to 120 megabits per second. So
that's the mp3 file being downloaded. So every single time this runs, a full mp3 file is downloaded alongside a few other things. Okay, I can open the reports, we're not looking into that because we're going to run something more interesting now, we do check all and what check all does, it runs the same command against all the regions, I'm at 2.3 gigabits per second, we're downloading all the files, we can see the responses coming back, EWR just sped by, I had sped by, so all the different endpoints are returning. Now I'm based in London, obviously the further way you are. So for example, this was South America, those LAX, so a couple of instances are slower to respond. And all this
happens via headers, you can, when you connect a fly, you can tell it, hey, I want to connect to a specific region, and then that's what routes to request to that, to that region. That's cool. Again, it's all captured in the pull request, and you can see what it looks like to check all one, Johannesburg, that's usually slow, and the slowest one is Tokyo for me, Sydney's welcome, so we still haven't received the responses from there. We should skip that shortly, because he's pulling now 50 megabits, 20 megabits, it's just slowing down, and it's just the connections between them there, the last one where there was Tokyo. In 60 seconds, I pulled about two gigs, roughly, there's a lot of data that gets pulled down, the feeds between between all of that. And anyone can
write, I recommend you not to run this, because we have to pay for this bandwidth, but our CI runs it just to make sure that everything works, and if we look at every hour, I think I'm going to tune this down, you can see there were no more connections hanging. So we go to the bottom of it as well, if it ever comes back, it went way on its own, it comes back on its own. We know that. Exactly. Now we have a system that is able to inform us when there's a problem, so let's go to three, we're on page number three. This one, for example, took more than five minutes,
right? So sometimes when the connectivity is a bit slow, some regions can be slow, that's when you get these timeouts, this is capped at five minutes. The last one that failed was a while ago, so you can see we're January 5th, there we go, this one that failed, January 4th, check all instances, so let's see, run, and we'll see exactly which region failed, execution, and RT, that's Tokyo, and as you can see, we have 100 seconds, right? So if after 100 seconds, it doesn't download, it just times out, and we were pulling data, but it didn't finish down the main power mp3, we're downloading 100 and something megabytes. Very cool. So I mean, not cool, I didn't finish, but cool, that was a while ago, and I can actually
test this. Now, do we need to be doing such a large file? Is that part of the test? Or can we test a smaller file and still get the same results? We could, yes, this was a file that was reported, so we need to find an mp3 file, absolutely, we can also reduce the frequency, we don't have to run it every hour, this was always in preparation for this conversation about episode 4, 5, 6, that one's coming up, that's the deepest rabbit hole, so I'm leaving that, that's for last, that's coming up.
Let me ask you just, though, in our, in our Zulu, but I don't think this is, I didn't check to see if this is even a thing, but to validate, you know, if the fly CLI could validate the tom file for you, because you could have been, you could have checked the tom file for syntax errors, or just do some don't essentially, and it didn't. It does have a validation sub-command, syntactically is correct, the config is valid, I mean, it was applied, but because it combines two things, it shouldn't, so at least I would expect a warning, like, hey, you're using HTTP services. Yeah, validate syntax and validate, you know, expected, you know, true tom file config, you know, don't combine or conflate two values or overwrite one or, you know, just that kind of thing, that's how I would defensively do something like that, and see a lot to protect my user from a port config, because I've just not been holding it wrong for so long.
Yep, I agree, just the impact of that configuration indeed, yeah. So, this is something we can see in the same logs, we can see, we can see here, we can go likes to 15 megabytes per second, that's 500 megabits. When we have these peaks, when you see this in the fly, how config, we can see this when you want the benchmarks run, or like when the checks run, because they put significant pressure on the instances, we can see them, we can pick them up straight away. Oh, so that's what this is. All right, so remember this guy, this guy was saying March 29, so it's almost two years ago, when this guy was saying, we will run into all sorts of issues that we end up sinking all kinds of time into.
So this guy had a good hunch, this is Jared, March 29th. And we just went through a couple of examples of issues that we had to deal with, part of this, but because of this, we understand the traffic and we understand how the application behaves and the backends behave at a very deep level. So you're right, Jared, we did some sort of a Hummy Lines, let's see how many lines do we have now. So how many lines? So how many lines?
Five hundred and ninety lines. Five ninety lines we have in total of varnish config, it's more than 20 lines. By the way, we have like the roadmap to 2.0, this is one point that we've had shipped, it's sold like a lot of issues, but that was the easy stuff. Okay, so for everyone that stuck with us, something really goes kind of go. And Adam was already mentioning it, episode 4.5.6, there's something special about episode 4.5.6.
So what is special about it? What stands out to you, Jared? Oh, just getting around with downloads. So episode 4.4.5.6. Oh, off, it's complicated.
Was that, by the way, this was recorded in 2021, it was published again, August 2021. For some reason, it's been downloaded a lot in recent months. It has over 1 million downloads. This is the most popular episode on the changelog ever. The most downloaded episode.
It's crazy. It's crazy. So, also you guys looked into this. We did, yes. We dug into this.
Okay, I didn't know you guys were doing this. So we just had a quick look to understand what is happening here. So we have Honeycomb open up, remember, every single request, which comes through the pipe dream through pipe. Every single request we sent to Honeycomb. We're able to look at it.
This is the last 60 days and I'm filtering down in such a way so that I'm only looking at this one file. How many times has this filed been downloaded in the last two months? And you can see the peaks, right? You can see, and by the way, this is gigabytes. So, and this is the period is four hours.
At about a hundred, we actually it was like this because here we had 200, almost 300, 300, 400. Anyway, close to 400 gigabytes in a four hour period. That's just too much. I think so. I know this is a great episode, great conversation.
I remember that conversation. It was good. Like who is downloading this file? 400, I know, times or actually more than 400 times every four hours consistently for months on end. Super fan.
Super fan. So we can see like different regions now. This is spread across the entire world. It's not just one region. This is really, really big.
I think if there was a DDoS attack, I think this would cost this one. And like in the last six months, in sorry, in the last two months, 60 days, we served 30 terabytes in San Jose, California alone. In Tokyo, we served five 15 terabytes. This is a big number. And if you look in this column, the distinct IPs, the client IPs, we had over 10,000 IPs downloading this file.
So this is not one or two IPs. This is thousands and thousands of IPs, which keep downloading this file over and over and over again. So I don't know how we would block about 10,000 IPs. Right. That would be the VCL would be crazy.
Well, that episode was starring Aaron Perrecki, who is a very talented person. And he is the co-founder of the webcam and a big fan of the indie web, as well as OAuth, obviously. So my hunch is Aaron's very interested in being the most downloaded episode ever. And he controls a fleet of machines from all around the world. And he points them wherever he wishes.
And he thinks, you know what I'm going to do? I'm going to get the number one spot on these guys download charts. And so I'm thinking Aaron Perrecki is the man with a mask on. We pull a mask off and send this whole time. What do you think, Aaron?
I think that we need to speak. See, I don't want to say the specific language. I think we need to go to Asia. I think we need to visit a couple of cities in Asia. OK, find the IPs, which are responsible for this because this is a crazy amount of traffic.
Asia, it just so happens if we look at, so Asia is the basically the continent which where we are getting the most dollars from because of this one episode. And this is actually traffic being served. This is not like head requests or get requests. These are bytes being sent to thousands and thousands of machines in Asia every single hour. So whoever's doing this, please stop.
Please. So we need to knock on doors. We need to go over there and knock on some doors and say, excuse me, is this IP address at this home? And then they might say, yes and say, would you please stop? And what's going on over here?
What can they possibly benefit from this? Like what could they be getting? Maybe we're the speed test. Someone's using us to speed test their connection. He knows.
Yeah, maybe that's the only thing I can imagine. But that's a lot of IP addresses. It is across multiple regions, which multiple data centers. Yes, so multiple regions, fly regions are serving these IPs. They're all coming from Asia, by the way.
I don't want to mention any names because there's no bad guys here. Right? We just want to assume that someone left the oven on. So the blinker on when you're driving that was like, hey, you're not turning. It's not a turn of blinker off.
The way I can see us mitigating this, and this is a hard problem because of the number right of IPs we're trading is we can basically start blocking entire network blocks. Unfortunately, some genuine listeners might be caught in this. And basically, a change of what will not be available, or at least the MP3s will not be available to a portion of users. The other one is obviously we can and we should. Right?
This is like the next problem. We should enable some throttling because there's more stuff happening here. So we don't have any sort of throttling. We assume fairness, we're assuming goodwill, we're assuming decency, and we're not seeing that here. That's the internet.
So to be honest, like whoever is doing this, and it's not LLM, so we have that problem as well. But in this case, it's not LLM. This is something completely different. So my hope is by someone that listens to this episode, maybe put this in the intro. Whoever is downloading episode 456, please stop.
Because we'll need to take the next step. It's a bit of cat and a mouse game, but that's what's going to happen. Because we need to pay for this bandwidth. This is only like varnish, right? This is only the cache layer that's happening.
This is only cache layer, yes. Yeah. And so what mechanisms are in varnish to do throttling or rate limiting or just anything like that whatsoever? There's V-Monts, which are basically modules, advantage loads that just give it extra functionality. One such V-mode and I've looked at this, it is free and open source, it's the V-month throttle.
Now that means that we need to start keeping track of IPs, and if we'll use a bit more memory, that's okay, we have more memory. And then we can need to start basically applying limits to how many downloads specific IPs can do. And we can limit it to mp3 files only. So if we have a bolt or if we have, for example, like, and ours as aggregator or something like that, we can, we were okay serving those requests because again, that's what Vaunter is meant to do. The problem here is that we're serving a lot of bytes for mp3 is the same mp3.
That cannot be real traffic. Yeah, I mean, even in this case, you can like, I potentially just to this mp3, like you just said, which is none of the all mp3 scenario. Like if you request this mp3 with this kind of like request signature of X per whatever, I mean, I didn't examine the actual signature of the request. That's how I probably investigate it is begin to isolate. Does that require us to write a lot of defensive code against that kind of scenario?
I don't think so. It's just some configuration. We just need to add more configuration. And back to Jared's point, we're just chasing now new problems that we didn't even think we would have. But we have what looks to me like an actor that's not very, I want to say this in a nice way, an unfriendly actor, but he's not very happy.
And they are very angrily downloading our mp3 over and over and over again, thousands of times across thousands of IPs. And this is not cool because ultimately we end up paying for this bandwidth. That is not helping anyone. But that's one. It's not the only one.
So we have one more. You can see here, for example, this is the last seven days, we have seven terabytes that were transferred in the last seven days, seven terabytes. Maybe that's more than that. It needs to be more ended, actually. Geocode does not exist.
Okay, I was expecting to see more than that. Anyway, Asia is the one that we can see like the pattern, but we also have like in Europe. Sometimes we have these spikes and it's like this spike, which I wanted to focus on. We know that someone in Frankfurt, like an extra Frankfurt, downloaded the static 5 icon 170,000 times in the span of an hour or two. So they now do this like two, three hours.
So it gets requests like this that are pretty stressed on these instances. I mean, a lot of potentially that was like a pass request as well, which means that they went through the cache, which means that they must have had a cookie set up or something like that, that basically was preventing the cache from working in this case, which again, that's how it's supposed to work. So anyway, I think that was unfortunately not the best thing that we could have ended on, but it's a thing and it's something which we thought more work to be done. There's many things that we didn't get to talk to. We didn't have time for example, we didn't talk about the nightly.
By the way, nightly now is being served by the Pypjim as well. And the reason why we had to do this because that sometimes will get scraped, will get hit really heavily. It's a very small app, it's EngineX, but if I open it, so let's just click on that one and that's pull request 46. Before it was basically topping up at 141 requests per second, now it's with 1,300, so it's almost like a 10x. In order for magnitude faster, the latency went way, way down.
So it's just the only thing we had to do is basically put varnish in front of it. Nice. That's one more thing there. You can go and have a look how it works. Like a benchmark here, a small benchmark here.
That's it. We have, we've lost one for the road, but before we do that, anything else we want to talk about before I share one last thought. I suppose what we do, if we know these demos are happening, we're here on the podcast, just politely asking to stop. We just want to keep happening. Well, we could set up some sort of throttling.
I think it'll be the easiest thing, and I will impact everyone. I don't want to start blocking again IP ranges, let blocks, because we don't know who's going to be caught there. They may change to other IP blocks, so that's entirely possible. We don't know how this will work. We can't block an entire country, an entire continent, especially if it's a big one.
I don't think that's reasonable. So really throttling is, I think, the fairest thing, and then we can throttle MP3 specifically. Because we do have, for example, I see them, like, for example, we have an Asian, a Python client and a Go client, that every week they come and they download all our MP3s. I don't know why they do that, but every seven days, they basically request every single MP3 that we have. So they're like scraping the website and then pulling everything down.
I don't know why. Yeah. Again, the closer, like the more I was looking at, and because I was working so deep in this, I started noticing these behaviors that you would normally not see. So it's one of the advantages, I suppose, being to working so close with a traffic with all the requests and having this level of understanding and visibility into every single request. So it really helps down to the IP level.
Something like that, though, like the Go client and the Python client, where would you, would that be a honeycomb thing? Where would that be? Yeah. It's on here. You can filter by user agent, for example.
And you can see that there'll be, for example, say, no, I don't want to show any IPs or anything like that. So that's why I need to screenshot that. But once we start getting into that, you can say group by client by agent, and you can say filter by MP3s. So like, URL contains MP3. And that will be able to group.
And you can say, oh, and by the way, only show me where there's more than, for example, 100 downloads. And then you'll start seeing, like, the outliers, which are the clients that start downloading certain MP3s or MP3s in general, excessively. Now, that could be spoofed. That's the other thing. Like, we have, for example, the request agent, like the user agent, it's empty, it's an empty string.
That also happens, right? Because you don't have to send the head if you don't want to. Yeah, you can also send whatever you want to. Yeah. Yeah.
It's like whenever you build systems like this, and then even when you observe them, I guess, you don't expect. That's why I originally thought that you kind of hope that clients, aka people, behave. You know, they're going to use the system for the system's purpose, not to once a week download and scrape the entire thing. And I mean, in that case, somebody could have their own web archive, and they could have altruistic reasons for it. I think that's kind of silly.
But once per week download the entire content, and somebody's disk seems like, I want your thing. I want to keep getting your thing. And if it ever changes, I want to make sure I have that snapshot. I don't understand it. It doesn't make any sense.
Like, what would make anybody do that? What is the purpose and motivation to keep doing that? To even commit the compute or the script or the time where they're getting from it? I don't know. We need to go over there and knock on some doors.
I'm going to ask them. Yeah. Why don't you do that? That's for you. Every door for you in Asia.
Do you listen to the changer log? Yes, I do. How many times? Yeah. Tell us about 4.5.6.
You don't want 5.6. We don't do that. Yeah. Yeah. It's just, see, this is how so this is, I think, a really delicate and really important point to discuss.
Because this is how good systems become bad systems. It's true. Yeah. You have to treat it really bad. Exactly.
Like, we don't want to be doing this, but we are forced to do something against something which is good. So it's not benefiting anyone. And we have to step in and do something about it. Now we have to do it. It's been, I was expecting this to stop, but it's still even to this day.
We made varnish. I mean, now that it's stable, it's able to serve more traffic. It's able to be like, we just had the biggest spike because now the system is more stable. But it means that bad actors, again, I shouldn't be using that. Unhappy people.
Unhappy clients. Use it. Unhappy clients. Yeah. The only person you're going to offend is the one who's doing this.
And I'm fine with it. Yeah. Maybe not enough. Here's a cut. This might be a cudgel.
But if we're trying to solve the problem of they're taking our bandwidth for something that's no longer relevant or interesting and it's been out there for years, what if we could just toggle certain episodes? And this might be a cat in that scheme as well. But like a certain point, it's like, well, just give them the R2 URL and not the CDN URL. And just let CloudFlare deal with it. You know, like just let them download directly from CloudFlare.
And we're just out of the equation then. We don't care about the stats. We don't care about anything. We're just like, you know, we've served this file plenty of times for our CDN. Now we're going to just let R2 serve it.
What do you think about that idea? I can see for this specific episode being a very simple fix, right? Because we can just serve basically a location header. And we just redirect. And that's it.
We're done with it. So it'd be like another synthetic response. We could question is that they're actually malicious and they switched to an episode and started that one. Exactly. Exactly.
And we have other clients, which are, for example, we've seen the past where they're basically busting the cache and purposefully going to R2 directly. And just varnishes like a proxy in this case. So we have that as well. We have every now and then we have like this random client that comes and downloads all the episodes. And that's not a problem.
So I think that some sort of a throttle would make sense, which would keep the system fair to everybody. But the throttle needs to be high enough so that it doesn't impact anyone else. Now, if, for example, our requests or like, if our audience grows or we become more popular and we get more requests, obviously, we need to be aware of where the limit is and start increasing the limit. So once we are throttling too much, maybe, that seems to be more like long term. And it seems a more, I don't know, like a well engineered approach in a way.
But certainly the simplest thing would be just like, take this one URL. I mean, that could be done in minutes or all it out. And then we would stop this abuse for specific and B3. That would be the easiest thing for sure. So yeah, I can see the how pragmatic that approaches.
And I like the pragmatism. Well, at least worth checking to see if, you know, the mouse is still alive over there. Right. And if they are, then we'll know that this is a cat mouse game. But if it's just like somebody left the blinker on, which is going to turn their blinker off for them.
And see if just the problem goes away. And if it changes to a new MP3, then yeah, we need more generic solutions. Yeah, we may not need that at all. I do have to say that the internet these days is very different from the internet even like a year ago. With the rise of LMS and AIs, I'm starting to see patterns in our traffic, which are unlike any other time.
We have these very big spikes when a lot of data is being requested in very short periods of time from, I mean, these are agents they don't make much sense. I mean, I know they're smoothed. There are many IPs which are being used. So it's almost like there's like a, I know some for some system, which once a lot of our content is in silly things because some requests, they just don't make sense. Like for example, what benefit does static 5 icon have?
Like what's up with that? That just makes no sense. It's a small file that gets a heartbeat or a version of a heartbeat. Maybe, but this is the first time I've seen this specific file being downloaded this many times. I haven't seen this before, which makes me think, is this a trend that we'll start seeing more and more requests that don't make sense.
And then you start having to set up some form of protection for all sorts of clients that are just doing the wrong thing. You need like a defensive layer by the fault. Exactly. Exactly. And something that would be fair to regular clients.
Like for example, when I was like a benchmark, I mean, sure, it's me. I wouldn't want other people to do that because I'm testing the system, making sure like the real world production system everywhere in the world is working correctly. And I'm aware of what that means and how it costs them by the way. My IPs are removed from all like the stats because otherwise you see those massive benchmarks. So we account for that.
But we can't account for all these like weird clients. It's a challenge. I think it's a good one, but it just sets us up to, you know, when you become older, it's like this like more like like an adult problem. Right. So we like got the thing, barely working.
We got it out there. We know we may be stable, reliable or that. Now we're hitting almost like a new layer of problems. And then this to me is like a hint as to like the next phase. I hope to be a kid again.
Yeah. Well, right. One positive thing I think is the robustness of our observability. Like being able to have this visibility is great. Because otherwise like, you know, wow, pat ourselves on the back, Aaron Perreck, let's get you back on the pod because, man, you are big all over the world.
That's amazing. They can download. You know, so what's your one last thing for the road ahead? I agree. That's my one last thing.
So keep it short. We'll keep it fun. I mentioned about the Christmas tree. I mentioned about the various things which I had going over the holidays. So make it work club.
That's the place. You're there. Both of you are there. So you can join whenever you want. Next Thursday.
Yeah. Next Thursday. I'm going to talk about the 100 gigabit one. The 100 gigabit one. So why would I need such a thing?
It's smoking for sure. So I thought the CCR 2004 has like four CPUs, has like multiple 10 gigabit SFP plus ports. It even has 24 SFP 28 ports, but it doesn't have a switch chip. And people that know about a little bit about hardware, you want to switch chip to the hardware offloading L3 and even L4. So after I bought the CCR 2004, it was like almost like a Christmas present.
I thought surely this will be enough for the rest of my life. I know. I had to get the flagship. So I'll be talking about that. The land has set up quite a few things coming up.
And it just goes to show how much I enjoy the hardware side of things as well, the networking side of things. Like I shaved two milliseconds of my WAN. It's amazing. Like little things like that. You know, like I was already good.
I was like, I already like stopped five milliseconds, but I wanted like stop three milliseconds. It's now 2.4 milliseconds. So the next and what it means, like why would I do this? So first of all, I'm all about improving. And every winter I improve the network.
In this specific instance, I wanted the pages just to be snappier, like things to load a lot quicker. To handle a bit more traffic, but also to not have an impact. Like I was running that benchmark, like 2.5. Look, I'm getting to another one right now. Let's see.
I have a speed test. I have speed test right here. Speed test London. Let's go for this one. So we're recording, we're streaming, right?
And I'm just pulling 2.5, 2.6 gigabits down. And there's no interruption on my network. Right? So it's just my bread and butter. You know, that's how I work.
And by the way, if you see any buffering or any, any slowing down, let me know. I see Adam a bit more pixelated. Maybe he can see a pixelated too. I don't know. But yeah, I just pulled six gigabytes, three down, three up.
It's just what I do every day. I work with stuff and enjoy it. And by the way, this is the slower gateway router. So I'm getting the proper one set up. And I'll talk about that.
And there's so many things there. Like V-Lanning is just quite a thing. I have a new IPv4 block, by the way. So some would say that I'm preparing for hosting something. And maybe I am.
I don't know. We'll see how that works. But I just realized that my home connection, obviously I couldn't serve all the IPv3s, or people like that would really triple my connection if that was happening. But I'm at 2.5 gigabits. The next one will be 5 gigabits.
And the hardware can do it. And the 5 gigabits, I mean, that's like a decent server. And you can do 5 gigabits all day every day. Sorry, yeah, gigabits per second. That's pretty decent.
So I'm just waiting for more internet. I'm going to say you're going to have 100 gigabit WAN, but you're not going to have a connection for it, right? Right. So very few places in the world have that. So if I was in Switzerland, I would get 25 gigabits.
Now, which is move. Would you move for this? Of course. The only reason. The only reason to move is that it's the 25 gigabit connection.
But I know that 100 gig is coming. So we'll see. Are they either a ship it by the time I move or I come, I move and then they ship it. So it's one or the other. The important thing is I have the router to handle already.
You will be ready. Exactly. So I'm a prepper. I'm prepping for that. Prepper in for a good internet.
And this is just like an interestingly, five years ago when I got like the previous router, I did the same thing as like a forum post. They'd like a follow up. So I just did a follow up recently at this milestone. So I've been at this for some number of years. And I'm like optimizing my network and making sure that it's in tip, top condition.
Relentless. I love it. Sorry, that was good stuff. Garehard. Well, that's a happy note to end on, right?
That's a happy note to end on. Observability on 100 gigabit. No. So you go. All right.
Well, the good news for Kaizen is we have a lot to work on. Always. Always. That's what it seems. Yeah.
We know how to bake them, don't we? Oh my gosh. The rabbit hole goes deep. And we keep going in. Kaizen.
My friends. Kaizen. Kaizen. All right. Kaizen22 is in the bag.
Join the discussion in our community's Zulip Head2Change.com.com slash community to sign up for zero dollars. And of course, check out all of Garehard's passions at MakeItWork.club. Thanks to our partners at fly.io and to our BeatFreakin' residents, Great Master Cylinder. Next week on the pod, news on Monday, Damien Tanner from LayerCode on Wednesday, and Techno Tim catches Adam up with the State of Home Lab Tech on Friday. Have a great weekend.
Recommend us to a friend. If you like the show, and let's talk again real soon.