Kaizen! Pipely goes BAM (Friends)

Welcome to Changelog. And friends, a weekly talk show about birthday presents. Thanks as always to our partners at fly.io, the public cloud built for developers who ship. Learn all about it at fly.io. Okay, let's Kaizen. Well, friends, before the show, I'm here with my good friend, David Hsu over at Retool.

Now, David, I've known about Retool for a very long time. You've been working with us for many, many years. And speaking of many, many years, Brex is one of your oldest customers. You've been in business almost seven years. I think they've been a customer for almost all those seven years, to my knowledge. But share the story. What do you do for Brex? How does Brex leverage Retool? And why have they stayed with you all these years? So what's really interesting about Brex is that they are an extremely operational, heavy company. And so for them, the quality of the internal tools is so important because you can imagine they have to deal with fraud, they have to deal with underwriting, they have to deal with so many problems. Basically, they have a giant team internally, basically just using internal tools to stay in a day out. So they have a very high bar for internal tools.

started, we were in the YC batch, actually. We're both in winter 17, and they were, yeah, I think maybe customer number five or something like that for us. I think DoorDash was a little bit before, but they were pretty early. And the problem they had was they had so many internal tools they needed to go and build, but not enough time or engineers to go build all of them. And even if they did have the time or engineers, they wanted their engineers focused on building external physics software, because that is what would drive the business forward. Brex mobile app, for example, is awesome. The Brex website, for example, is awesome. The Brex expense flow, all really great external physics software. So they wanted their engineers focused on that, as opposed to...

building internal CRUD UIs. And so that's why they came to us. And it was honestly a wonderful partnership. It has been for seven, eight years now. Today, I think Brex has probably around a thousand Retool apps they use in production almost every week, which is awesome. And their whole business effectively runs now on Retool. And we are so, so privileged to be a part of their journey. And to me, I think what's really cool about all this is that we've managed to allow them to move so fast. So whether it's launching new product lines, whether it's responding to customers faster, whatever it is, if they need an app for that, they can get an app for it in a day, which is a lot better than, you know, in six months or a year, for example, having to schlep through spreadsheets.

et cetera. So I'm really, really proud of our partnership with Rex. Okay. Retool is the best way to build, maintain, and deploy internal software. Seamlessly connect to databases, build with elegant components, and customize with code. Accelerate mundane tasks and free up time for the work that really matters for you and your team. Learn more at retool.com. Start for free. Book a demo. Again, retool.com.

We are here to Kaizen, which means Gerhard Lazu is also here. What's up, man? In the house. Gerhard Lazu in the house. Yes. Welcome. Everything's up. Everything's up. That's right. That's the DevOps response, isn't it? Or the sysadmin? I don't know what you call yourself these days. Well, it's just, I know, titles, right? Infraengineer. I mean, what is your title, Gerhard? Officially head of infrastructure for Dagger. Okay, cool. Yeah. It's a big role.

Yeah, it is. I'm enjoying it. I've grown into it. Are you on PagerDuty? Always. I'm responsible for everyone that's on PagerDuty. Okay. And I'm responsible that PagerDuty is set up correctly. We are alerted when the right things go down. So yeah. So you literally use PagerDuty? No. No. It's the placeholder for the next phone call. Is that a burn? Was it just a...

It's a fact. It's a fact, yeah. Okay. Well, it's a fact, but was it a burn as well? A burn? I don't know about that. A pager duty burn? I don't know. Maybe. Okay. Maybe. I never really loved pager duty, I have to say. And it's not what's behind it. It's like the whole setup. It's just too complex, I think. I will say this about it, because this is all I know about it. Great name. It's got a great name. Right. That's all I can say about it. I prefer incident. Incident.io, I think that's even a better name. When there's an incident. Really? Why? Because we don't have pagers anymore? Pretty much, yeah.

Who has pagers? That's true. I guess it's a terrible name. But now it's just a page that person, which means call that person or email that person or slack that person or just get a hold of them. However, it means possible. Yeah, exactly. So if anything, if you only use a pager.

It means you don't have a backup. And if something goes down, you definitely want your whatever's monitoring to have multiple layers of redundancy, right? You can just wear two pagers. You know, they slide onto your belt. So you can just clip a second one next to it. But it's using a single network. So you need redundancy. You need cell phones. You need, like, you know, emails, the whole thing. Well, two of everything, I guess. Can't silence it. That's my biggest issue. I forget I silenced my phone. And then I'm like, why did I not get that text? Oh, because my phone is on silent. Do you normally not have it on silent? My phone's been on silent for 12 years.

Same here. You know, I don't know, man. I don't know. That's what I got to watch, right? The watch will alert. Yes. Yeah, exactly. I feel like the phone is such a hard thing, man. I'm just like, when to make it, you know, alertable, let's just say, or like something where it can bother me because I miss critical texts or emails or not so much emails, more like texts or phone calls, you know?

I wore the watch for a couple of years and thought that I needed it in my life. And then the watch broke. And that made me ask the question, do I really need the watch? And I just decided 300 bucks or whatever. I'm going to go without it for a couple of weeks and just see. I never felt more freedom than my watch broke. Oh, I haven't bought one since. I haven't had a watch for over a year now. And I don't think I'm going to go back. What kind of watch you got, Gerhard? It's the Apple Watch. So which one? Which one? The Ultra 2 or the Ultra 1? I bet it's the biggest, most expensive one. Well, it is the Ultra. I was waiting for that. Got it. I love the extra GPSs and everything. So it has like.

a couple of things in it. Ultra 2, that would be the new one and this would be the backup. That's what we're working towards. But I do like, especially when I drive, I love Apple Maps. That integration is really, really good. Not sure if you've tried it, but when you have to take an exit or you have to take a turn, it just vibrates. It's very, very helpful. Yeah. I'm with you there, but I'm not with you there. I feel like I like the Apple Maps and I go there, but I use CarPlay instead rather than the watch.

Let the car be the alert. And she'll just talk to you. She'll just be like, take your next ride. Or just pay attention to the map. Yeah, we got to pay attention to the road, Adam. Also, you got the game on your handheld. You got to watch the game while you're driving. That's right. I'm playing PlayStation 2 while driving. And that's a Fast and Furious throwback, Jared. Oh, I thought maybe it was a Silicon Valley reference. No, man. I got more. Maybe, you know. You're not a single-trick pony? Fast and Furious, the very first episode or the very first, I guess, movie. That's the one that I remember, yeah. Before the race, the kid was playing PlayStation. It was actually PlayStation 1.

in his car, in the console, prior to the race. And it was like a flex, and it was like, oh my gosh, I've got to check out my car. I have to have a PlayStation console in my dashboard. That's not realistic, because that sucker did not have, what was it called, when the CDs would just... Antiproperation? Yeah, like, you know, the old Walkman that took an actual CD, and you walked around with it. It was skip constantly, skip protection. Totally, yeah. I'm pretty sure PlayStation 1 had the same problem. If you're driving a car and playing it, you're probably skipping all over the place. Gerhard, get us on track here. We're here to Kaizen.

We'll talk about movie references the entire show. Kaizen 18. So I realized that this was, or will be, when it will come out, my 1-1-1 episode on the changelog. Oh, wow. You like that number. It's not round, but it's symmetrical. I don't know what it is. It's all ones. It's three ones. I mean, that happens already. Like the next time, like twos, I think it's going to be such a long time, right? If we only do the Kaizens, I think that will last me to the end of life. Honestly. That might. Yeah. I mean, two and a half months, like 1-1-1 divided by two and a half months. That's a lot of years, I think.

How many Kaisens do you think we're going to make it to before one of us, you know, kicks the can? Well, hopefully we'll get to 100. That's what I would like to see. We'll get to 100 at least. Yeah, 100 would be awesome. Yeah, I mean, you know, we won't stop like Ship It at 90. This one has to go to 100. That's right. That's what I'm thinking. Wow. So we got...

75 more episodes ago. And that's, I think, what is it? 40 years? Let's just acknowledge it and move on. Yeah, it's a lot of years. That's a lot. What about yours? Do you know what episode appearance this will be for you? Mostly all of them. Well, we can look it up easily because it's on the person page. It is, yes. I love that page. I don't know if anyone is aware of that, but if you've been as a guest to the changelog, or even if I think if you replied, I'm not sure about that part, but it will show all your interactions or all your references exactly on the changelog.

I use that quite a lot. So changelog.com forward slash. And what is it for the person? Person slash slug. Person. All right. Cool. So there you go. 110 episodes. So I've been on 909 episodes. 909? Wow. Yeah. Wow. That's a lot. Yeah. 909. Crazy. So this will be 910 for me. Or maybe 911 by the time it comes out. I don't know. Because Wednesday show is Adam by himself. So this will be 910 for me. Yeah. Do you think this is the year that you'll crack a thousand? Is this it? Good question. Three a week. No. Three a week? Yeah, no. Three a week times 50. Yeah.

But we're there. It's February already. That's true. Keep thinking this is the start of the year. It's March, actually. Time is compressed. Yeah. Maybe. Yeah, so it's possible. Maybe our final episode of the year will be the 1,000th. Wow. Okay, and 8.02. What happened there? How come do you have more episodes than Adam? What's this on the news? He's got a hack. I was on JS Party for a while. JS Party and news, yeah. Okay, okay. All right, so I'm winning. All right, so far. Yeah, let's see if I can catch it. Yeah, I guess I couldn't catch it, could I? It would probably be pretty hard to do that. You can take over news if you want.

it's like it's not worth it i mean funny news funny news maybe you know scaling is is a people thing so let's talk about uh something that happened let's start with with alone well changelog was down for four hours oh let's not talk about it did anyone notice well i'm really wondering like did you notice that changelog was down you did okay how did it happen for you i went to the website and it wasn't there right okay okay the classic way all right i assume it was signed in people only because i didn't actually check but i'm always signed in and so we will cash with fast yeah we'll cash with fastly if you're not signed in but if you have a sign-in cookie we pass it through to the app every time and the app was down and so i noticed because i went to go share something and wanted to look at something and i don't know it was down although i think i already knew that because maybe you posted it i don't know but i definitely just went to the website and 503 or whatever so for anyone if if you're wondering if changelog is down go to status.changelog.com and you will see what is down, when it is down. So in this particular case, we had a previous incident. There's a bit of a red right there. And this is the origin. So the origin was down. And if you click on that, it takes you to the status and you can see like the whole history. So this is something that I do update whenever there's an issue like this, especially when it's a big one. We had a few small ones, like just like a few minutes, but those don't show up. But this one was significant. And February 16th, 10 a.m. Actually, it was before 10 a.m. So that was a Saturday. Saturday or Sunday? No, I think it was a Sunday. February 16th. Yeah, I know it was on Sunday.

Yeah, it was a Sunday. So that was half the Sunday looking at this. So what happened? Well, if you go to the discussion, 538, that's where all the links are. But basically as it happened, it was a fly issue.

and the fly.io issue wasn't fly.io itself fly.io and i'm going to scroll down to that particular message has providers so in this case one of the upstream networks so let's see let's see where is it i'm looking for there it was a far upstream issue and i'm now looking at um a post from kurt the ceo fly and he was saying that the failure was far upstream from us and a single point of network failure so one of their vendors let them down basically and there's not much they could do about it so this is what happens when you know because we all depend on other systems and other systems are always upstream systems you have internet the internet provider i'm sure you know has um transit links and peering links and all of that some of those can be down if you don't don't run to everything everything in this case they didn't have to have to have everything the switch went down and it took four hours for someone to fix it and it was i think sunday very early morning on the east coast which just only had a bad Sunday. Yeah. Well, lots of us did, but one person in particular probably. Yeah. So their virtual pager went off. Yeah. So that was not great. But I think one of the key takeaways for us is that in terms of how many requests didn't go through. So final impact, I posted again on the Fly community.

For the whole outage, actually, our SLI for successful HTTP requests, that was like the last 24 hours, dropped to 97.40%. So went well below three nines, even four nines. But it's still 97% of the requests were served. Most of them, they go to our object storage, right? All the MP3s, all the static assets, all of that. The website itself, I mean, some of the pages, the most visited ones, they're being cached and they were served from the CDN, fastly in this case. So if you were not signed in...

most likely you will not have noticed this and i think for many people that consume you know the content through their podcast players or from youtube wherever you get the change of content from i think i don't think you'll have noticed this this was very specific to the app and if you have let us know mp3s continue to serve right yeah so yeah exactly unlikely that people really noticed except for the people who noticed yeah well i mean there were some for sure because we can see that you know a bunch of requests failed but in the big scheme of things it wasn't that much now did we run two changelog application instances? We did not. We did not. Actually, well, we did, but they were all in the same region. So this was a regional failure. All of Ashburn, Virginia, in this case, Fly's Ashburn, Virginia, IAD, that's the one that went down. And unfortunately- The major one. Yeah, that's actually the primary one. What made this worse is that Fly itself, the control plane for the machines, was running in that single region, which meant that no one could scale their apps. So if you happened to have a single or multiple app instances running only in that region,

everything would have been down. If you could have scaled it while this was ongoing, you could just basically spin up another application in a different region. But that was not possible. So again, there were a couple of things that failed in surprising ways. And for me, what was surprising is that, well, we did have two application instances, but they were both in the same region and the region went down. So now we have another one running in EWR, I think is New Jersey, somewhere in New Jersey. So yeah.

We're good. So we're good to go. We're good to go. Why don't you put that one on somewhere closer to yourself, Gerhard? Well, if I did, it would still need to go to Neon. That's where the database is. So that would introduce a lot of latency. Now, if you could distribute the database and we could have a couple of free replicas, which is something that I'm thinking about, this would make more sense. Do we want to do that? Oh, I don't know. Any other stuff to work on? I do. I don't think we need to do that. Chasing the nines is fun. Cool. What's next? All right. So there's a thread.

Linkify chapters in Zulip. New episode messages. I remember we talked about that in the last episode and I think it was like a day before or two days before it just landed. How amazing was that? So amazing. Probably the coolest thing that happened in this whole time. Just kidding. So far. So far. Hang on. Hang on. So far. An outage and a feature. Yeah. So what was it like to implement it? It was not very hard. I don't remember. 51 additions and 18 subtractions. So that's a small feature. You know, just a little bit of code to go ahead and Linkify those suckers. So for those who don't know what we're talking about, when...

any new episode is published, our system automatically notifies various social things, one of which is our awesome Zulip community. If you're not in there, what's wrong with you? Change.com slash community. Get yourself a Zulip. It's fully free. And you'll be able to chat about the shows after they come out. And so every time a show comes out, it posts in there, hey, new episode. It has the summary, the title, and the link to listen to it. And we've also now embedded the chapters as markdown tables.

And that was already there. That's not the feature. What I didn't do prior was I didn't linkify the actual chapter timestamps. So you can click on a timestamp and immediately start listening to it. And so that's what I added was I made those timestamp links so you can click and listen from that spot, which was requested by the both of you on the last Kaizen. And so since we have a three day turnaround between recording and shipping each episode, I actually shipped the feature out, I think prior to that episode dropping. Like I said, it was half an hour of coding.

But it's useful. Those are the best features, right? A little bit of work, lots of value. Exactly. I'm wondering if anyone else is using it or if they noticed it. And what do they think? Useful? Good question. Let us know in the Zulu comments. Useful? Useless? Or do we revert it? Do we revert it? Not going to happen, though. Yeah, I just don't see any reason why you take the links away. If one person likes it, we know Gerard likes it, then why not?

That's really cool. Did you click one of those links, Adam, since the feature landed? I would say no. You said you would. You said you were going to click on them. Did I say that? Something like that. Go back and quote me. I want to hear. Jason, pull up a quote. If I said that, I want to know I said it. I'll eat some. What do you say? Eat crow? Yeah, eat crow. I'll eat some chicken. Eat more chicken. Okay. I do like the feature being there. I think that I'm just a go there and do it kind of person, not the stay here and click around kind of person.

Although I do like what I like about Zulip and what I like about what this offers, I believe, is that we tend to have thick conversations, much thicker than we had in Slack. And so one of my biggest excitements, I would say, if that's even a word, happiness levels, pick your 834 in the morning word. This is an early recording time, as you can probably tell. I'm going to do a network talk. Drink your coffee.

Are you apologizing for your lack of sharpness or what? Yes. Yes, I am. I am very sharp in this. The conversation must be boring. Jared, that's what it is. We're not good hosts. Adam's boring himself over there. Jeez, no, I was drinking the coffee. All right, well, did you bring some crow? I got more to say. I got more to say. I got more to say. Well, enjoy. If I didn't, if I couldn't tell, I'm getting there.

is that how thick these comments are in Zoop. So back to what Jared said, you are missing out. Change all the comments slash community. The quality. When you say thick, you're talking about the quality. Thick. Good. Yeah. Sorry. I mean, is that not clear enough? Thick comments? Thick comments actually means they're not very good. Yeah. Like how thick? Well, it depends. If you like thick or not, you know? I think thick is always better than thin. I mean, like, go choose yourself a Reese cup or whatever, right? What? You call a Reese. You pick your nomenclature for your Reese cups, man. I like the big cups, okay? Did you say a Reese cup? What is a Reese cup?

Can you show us? Can you show the viewers what it means? It's this, okay? That's what it is. It's big. It's big. Is it? They have a big cup. Okay, we'll call them Reese. My wife makes fun of me too. I say, I used to say Nike. Had no idea it was Nike my whole life. Okay. You said Nike? Yes. Like a fool. Like a fool. I've never heard anybody say Nike. This is amazing. How many years? How many years did you say that?

35. 35 years? That's amazing. You never realized it until you were 35? I just thought there was two ways to say it. That's a blast. The comments are vast. Lots. They are plentiful. They are thoughtful. And there's lots of commentary in our Zolip. So I think with these comments, back to the links, gosh.

what they provide is if you are there and you're in conversation and you're using that table as a reference point, well, then you're obviously going to be able to go and click directly from there, which I think is super cool because you have the useful tool where the conversation is happening. So, okay. Well, I found a quote from our previous episode. Did you bring your crow? Because you might have to eat a little bit of it. What did I say? Like I said, I could make those links clickable. And maybe I'll do that. And then Gerhard Lazu said, I would love that. And then Adam Sokoviak said, I would concur and plus one that because that would make me click a chapter start time easily. Because it would be clickable for one. And I want to. So you said it would make you click it because it'd be clickable and you want to now. Yeah. And I have. But I'm not like, I'm not a daily clicker. No, I thought you just said you hadn't. I'd click at least one. Okay. All right. Well, at least one. Controversy solved. My gosh, this bus is heavy. I'm under here.

All right. So I just got, I just wanted to close that loop and then we can move on. So I love this feature. Great job. He uses it all the time. I use it daily. I'm a daily active user of this feature. Awesome. Well, before the show, I'm here with Jasmine Cassis from Sentry. Jasmine, I know that session replay is one of those features that just once you use it, it becomes the way. How widely adopted is session replay for Sentry?

I can't share specific numbers, but it is highly adopted in terms of if you look at the whole feature set of Sentry, replay is highly adopted. I think what's really important to us is Sentry supports over 100 languages and frameworks. That also means mobile. So I think it's important for us to cater to all sorts of developers. We can do that by opening up a replay from not just web, but going to mobile. I think that's the most important needle to move. So I know one of the things that developers waste so much time on is reproducing some sort of user interface error or some sort of user flow error. And now there is session replay. To me, it really does seem like the killer feature for Sentry. Absolutely. That's a sentiment shared by a lot of our customers.

double down on that workflow because today, if you just get a link to an issue alert in Sentry, an issue alert, for example, in Slack or whatever integration that you use, as soon as you open that issue alert, we've embedded the replay video at the time of the error. So then it just becomes part of the troubleshooting process. It's no longer an add-on. It's just one of the steps that you do. Just like you would review a stack trace, our users would just also review the replay video. It's embedded right there on the issues page. Okay, Sentry is always shipping, always helping developers ship with confidence. That's what they do. Check out their launch week details in the link in the show notes. And of course, check out Session Replay's new edition mobile replay in the link in the show notes as well. And here's the best part. If you want to try Sentry,

You can do so today with $800 off the team plan. Totally free to try out for you and your team. Use the code changelog. Go to century.io. Again, century.io.

So yeah, so that was a good one. I enjoyed that it landed. We will talk about the YouTube videos for sure. And that's going to come up. We can talk about it now, by the way, because really that's for me, that just like took the highlights in terms of features. Okay. So once the video podcast landed, that was just so amazing. So I am still watching Adam's podcast, Adam's video with Techno Tim. And almost at the end, that was such a great conversation. Thank you, man.

Had it not been for the video part, I would have missed, for example, Tim's background, the little mini-rag that he was building, like the little, you know, like body language. It was just so good. I'm enjoying that a lot more than if it was just audio only because there's so much more detail in that content. Cool. Well, that makes me happy. Yeah. So that's the one that's, again, and it doesn't often happen that I listen to a changelog episode from start to finish. I usually have parts, which is where the links were coming in very handy. Yeah, chapters. But this, exactly like the chapters, but this one episode, I'm like near the end and I just cannot wait to see how it ends. Let me ask you a question. If Tim and I did that more frequently,

do you think that'd be a good thing yes but i think that you need to up your game and start delivering on some of the ideas or like start like implementing some of your ideas to see how they work in practice such as such as so you're saying about building a new pc so i'm curious what did you do about that did you i built a thing you did you build a thing yeah oh wow okay so i got a beefy a home lab right now very nice what are you running oh you want the you want the words here okay fine i'll tell you i'll tell you the words let me see if i can like what's the case did you go for fractal i'm not a big fan yeah i did go for actual i feel like it feels like we need some pictures i mean if this will be in the b-roll um for jason i would love to see that what gp did you go for i'm very curious about that like that was something well so i repurposed i'm as you do anything you start with what you have so rather than go out and spend the five grand that i would really love to spend on something all i did was just go pick up a 3090 and add it to the existing machine i already had so i had a i had just built this beefy machine for my plex machine which was like just overkill i just want to build something so my uh motherboard is an asus workstation level motherboard it's a 680 ace

And it's got four demos of DR5 RAM available, up to 128 gigabytes of RAM. So I've got that. I've got the 13900K. So it's an older generation CPU, but it's still very, very capable. Couple that with the RTX Tough Gaming 3090 and the maxed out RAM and an NVMe SSD, where you've got yourself a really fast machine. And that's my stack right there, basically. That's very nice. Network? It's 2.5 by default. 2.5. Okay, okay, okay. Are you thinking of going higher on the network? So the motherboard doesn't offer it by default, but I can add a card. I don't know if I'm maxed out on my PCIe lanes, though, with my 16 lane requirement for the GPU. Yeah, well, if you have NVMe, it means that you have only one or two. You can't have more than two. There's three slots on the board. I'm only running one. I only have a need for one. Right.

So the reason why I ask that is because as soon as you, I think as soon as you fill the second lane, you'll lose it. You'll half the lanes for your GPU. It'll go from 16 to eight. And I don't want to do that. Those lanes are shared with the NVMe drives. Yes. Actually, in practice, it's not as bad as you would think. I did the same. So I maxed out the NVMe as another machine. And because I maxed them out, I have like four or five. And because of that, my GPU, which is a 4080, dropped to eight lanes. But that's enough. Like the drop in performance is so little because I don't game on it heavily.

yeah it's so fast already what you really want is the storage right you want the vram not so much the speed necessarily unless you really are pushing the speed like uh and you're doing ai stuff and you got a serious parameter you know lm sitting there or whatever then maybe you want those tokens to be as fast as possible because like that's the whole point the actual difference is more like a few percent so if you go from 16 to 8 it's just a few percent this is what i love about these conversations with tim it's just they're so infrequent it's once per year so we're more catching up versus digging deep yeah i would love you to have these more often and especially like you know you had that conversation you said about like some of your plans and um like a lot of things that you mentioned now i remember you mentioning when you were talking to tim yeah and so how did you follow through on that like did you stick to what you said or did you change your mind as you were building it it sounds to me that you haven't i remember tim mentioning 1390 so i'm very curious like did you buy it off ebay because you're mentioning about your good experience with ebay i got it yeah i got it on ebay uh really good experience on ebay i think i got it for like 800 bucks right you know it's not the worst price ever us us dollars yeah

It's basically brand new. It's super clean. You know, I tested it the moment I got it. Like I just did it. I did like all these parameter tests. Initially, I spun up in Ubuntu installation, ran into issues with Docker and GPU. So I went to the dark side. I installed Windows 11 Pro. And so my AI home lab right now is being powered by Windows 11 Pro. I know Jared has backslashes trash over there on me, which I'm cool with. But man, you've got to explore. I love the idea. I've never played with Windows. And like I told my son, I'm like, it's been 20 years since I've played with Windows that long. And I feel like there's a lot of cool stuff there. But man, they got some really terrible warts over there.

you know like just so bad it's like it's developer hostile now not just user hostile like there's ways to clean it up Chris Titus has a script that you can run via terminal the administrator level terminal and remove a bunch of stuff and sort of like make some things nicer which I think is super cool makes it a little easier as a non-windows user to like easily get to a certain state but yeah I played with it first I did some benchmark testing against it I really pushed it as hard as I could to just confirm it was a good buy and it was a good buy but I started with where I was at versus like, okay, let me get a brand new motherboard, a brand new, you know, sticks of RAM. And I would love that. That's the fun side of building PCs is, is like, I really wish there was a better operating system that wasn't, gosh, will I get punched in the face for saying this? It wasn't Linux. I will say that this is the first time I played with Ubuntu desktop.

in a long time. And that has actually come a very, very long way. Ubuntu Desktop, I think, is probably the closest contender to a non-Mac OS operating system that's fun to play with GUI. Now, albeit I have not explored like Pop! OS and others, I just haven't had a... You only test things that you're curious about. I just haven't been curious about desktop-level Linux stuff yet, mainly because it's been the year of the desktop Linux forever, and it's never come, truly. I'm hopeful, though, that one day, I think it's probably the closest it's been in a very long time. So when I started with like my adventure in GPUs, I needed it to do like the video editing properly. Yeah.

I went Linux first. I was saying, you know what? I'm not going to go to Windows. What is the best Linux distribution that has good support for GPUs, like out of the box? It just has the drivers pre-installed, and everything just works. And the tiling manager works as well, because that can be sometimes a pain. PopOS was the one that kept coming up very high, and I said, let's just try it. So I did, and I think I've been running it for two years, coming to two years? Coming to two years. I've been running it, and before I had a NixOS. So this was a machine that went from NixOS to PopOS, and I'm enjoying PopOS more. It's just more, it feels like more like a natural way of using it. What's it based off?

ubuntu okay so it's ubuntu based but a lot of like the little things that in ubuntu maybe they don't work they seem to have a better there seem to be a bit more polished off in pop os specifically the nvidia integration and the tiling manager things are just you know a bit more i know cohesive it's just it's a bit more cohesive so this machine i'm using for a bunch of things and while i started editing the videos on it with davinci resolve davinci resolve itself pop os is not a supported operating system and then i was forced to go to windows so i like when tim said that you have to try them all like in that interview i realized yes i actually went through the same journey so he's like adam you gotta try them all man so i use linux for something i use mac for something else and i use windows for editing because apparently the editing software works has like the best support like codecs and things like that they work really well on windows the operating system itself oh wow i don't know what words to use that would be politically correct but also accurate they make it so hard to do everything Manage your own user. Like even manage your own user. There's like control panel and there's user accounts there. Then you've obviously got system settings or just settings, which you have those things there. It's like there's like three places to do pretty much anything. And you got to do three things to change one thing. And they're all different places. And some of them are legacy looking applications. Good luck even finding it in the sea of things you can find. I just think that's somebody there is not empowered to fix it or somebody doesn't care. I'm not sure which one it is, but they they really could have a.

Because the reason you're saying this you know this out-of-the-box support I had such trouble getting my GPU to play well with the operating system and then being bypassed through to docker I had to like go and add some things and like I think what a documentation was like seemingly foreign I just fell a little lost on Ubuntu Linux to try to get the initial say I want to be and that's my default so I didn't try to go to pop OS or explore I could have but because I had a conversation with Tim he's like you should try them all so I was like well you know I tried you know Windows 11 and Not the worst but man like even I felt successful just as it's aging into I had to post this video to our general channel, and then Jared had to go in there and backslash me. Cool. Love that song. Just because it's such access to SSH into the machine. You had to go and install the OpenSSH server. The client was there by default, but the server was not. And then I think my original username had a space in it, so when I was SSHing into it, it wasn't Adam, it was just something else. I don't know. Trying to find my slug for my username even.

I don't even know if I found it. I think I luckily found something to like swap it out and restart SSH server and I was in. Well, if you have multiple NVMe drives, you could always do a boot and you could try, you know, another Linux distribution. I can recommend PopOS. They have a new graphical manager, I think Cosmic. It's not disabled, but I hear very good things about it. So they rewrote it, everything in Rust. It's amazing. I haven't tried it yet. I'm still like on the old one. What are your thoughts on WSL2 and how it integrates? I haven't explored that deeply, but I have a lot of hope that there's cool integration. Like one thing I know I can do is I can...

SSH from one machine to another, our sync files from that via Ubuntu in WSL2 on Windows, and I can run Linux-y things if I have an operating system or whatever installed to WSL, a distro, I should say. um what's your experience there with that i tried it it's okay i mean it gives you a close enough linux experience it's much better than it used to be before powershell i just just can't get along with it i just wouldn't use it command prompt seriously i was like 20 years ago the thing is still around so yeah legacy but i think the legacy so wsl2 i think it's a good feature but but windows itself is an operating system as a package like the outer package just just feels wrong to me and i use it only for specific reasons yeah so davinci resolve i have i have a decent experience with that if i had to do this all over again I would get a Mac, like an Ultra 2, an M2 Ultra or an M4 Ultra when they come out, like a really powerful CPU and a GPU. But the RTX, like a 4080 or 3090, that is a level of like hacking that you just don't have in the Mac world. So I just wanted to try it out. It was okay. I mean, the Windows workstation, for example, that has a 4090. It's a very loud system. Like really, I don't think people realize how quiet Macs are. Whether it's a laptop, whether it's a studio, whether it's a mini, you know, they're like whisper quiet.

and this first linux workstation which i built is a fanless one the psu has no fans like there's no fan in the system and i love that for it i love it for that um for example the nvme that has no spinning disks which you know it's just just a great fast and silent yeah exactly so it's a great feature in comparison the windows machine is just the opposite it's just loud it's just like it's just hot very hot it's a um 1390 ks it's like the top of the range okay 13 series the ks is clock overclockable i believe exactly yeah it goes all the way back to six six and something the same cpu then yeah yeah except for you got the overclockable version of it yeah um it has also like 192 gigs of ram so it's like fully maxed out nvme like the whole it's like a fully maxed out or it used to be fully maxed out pc maybe about a year ago um so yeah it's okay but like trying that world it is my editing machine and i love when tim said that right you need to have roles for your machines and that's what it is and if it was to break down that's okay there's like another machine to use to to to replace it so your comparison of the fans and the and the noise level i think so the exploration for me is not okay let me and i think for now i'm like yes let's let's make this a greater pc of some sort let's uh explore this world i don't know if i'll stay there forever but i'm i'm enjoying the exploration i'm not enjoying it because it's windows necessarily it's enjoyable because it's new territory it's new found how does this work

Does this fit for me? If it does, where does it fit? I will 100% concur and agree that, wow, this machine, its fans just spin up opening applications. It doesn't need to. It's got this beefy CPU. So for whatever reason, the front three fans spin up for 10 seconds, just enough to hear it. And it goes back down and it kind of cools off. Or if you ask a big question in alarm or whatever, it's obviously going to spin up for the duration of that question. So it's by design doing that. Will the Mac world supersede this in a smaller, easier package that's silent?

power less power hungry that's cool and what they're doing there is super cool but You can't build it yourself, and it's so sad. I know, I know. Anyways, we can probably move on, but I think that's what I love about building PCs, just the exploration of the hardware, how does it work, you know, what works together. Yeah, me too, me too. And I think we're at a stage where it does make sense to have a few lying around. Have a Windows machine, right? If you have to do testing or anything like that, use the Linux machine. You know, I think it's a very eye-opening experience as to what is possible. And then, you know, if Mac is your default, or if not, if you have the opportunity to get maybe a Mac Mini, do that as well. And then you will find the one that you love and the one that's your daily driver, and you have a couple as backups, you know, when something goes wrong, because it does.

doesn't happen cool yeah well talking about podcasts because in video podcast because that's how we started uh i don't think we finished there's like so many new features around youtube and around content on youtube um i think the reactions have been mostly positive there was a whole zulu discuss about it uh which i don't contribute to many but this one i did contribute to and uh february 1st i even got some um love hearts from a few of you nabil and marsh so thank you very much for that but uh what do you think about launching video podcasts like how was that transition how's that new chapter i think it's going pretty well i guess i don't consider it to be over with maybe it is because i guess a lot of what we think about is production workflow and we're constantly trying to improve that and make it better i would say that we successfully went video first now and we have systems in place that we can do that reliably i had to build a few things and we had to figure out a lot with regards to chapters and timestamps and how we handle the videos on youtube versus the podcast episodes and audio and um all the nuts and bolts i think were fine like we just kind of figured it all out and did it nothing really

It was too difficult there. The response has been positive. I think a lot of our audio listeners have a little trepidation because they think, is it going to become a YouTube show? They never want to listen to it on YouTube, which I don't either, honestly. We're doing this for people who like that kind of thing, like Gerard, I guess, and others. And I acknowledge that you all are out there, and we appreciate that you are, and we want you to watch it on YouTube, which is why we came there. But our existing audience, very few of them find much value in the videos, I think, or the ones who at least are vocal don't, and I get that. And of course, trepidation is like, well, will the audio suffer? And will we start to pull a thing up on the screen and have reactions to it without explaining what we're looking at? I don't ever want to get there. Hopefully, we can be self-aware and always remember that we have a listener, not just a viewer.

and explain what we're looking at if we are looking at something so for them i understand because if you love something and it's changing you just hope that doesn't change for you for the worse and so hopefully we haven't done that i think we've had most people who had trepidation at least so far have been fine with the change. They haven't noticed much of a difference. And for those who love video podcasts or watching conversations on YouTube, because is it a podcast actually? I guess YouTube thinks it is. We're there now and people are watching. You know, we get 500 to 1,000 watches on a video. We hope to grow that. And no real complaints there, I don't think, besides your random YouTube troll, which we've had trolls our entire career, so we don't feed them or care about them very much. That's my initial thoughts. Adam, anything to add or attract? I ran into it because I was actually talking to my son last night. I was like, dude.

because my son's nine, and I'm about to give him an Ubuntu desktop machine to play with. I'm going to start teaching Linux. and I was, like, excited because I had just, like, literally maybe earlier that day, SSH into this Windows machine. I was like, success, you know? And I was, like, referencing embedded systems and why it's so cool, how Linux is so cool, and I'm like, you want to see something cool? And I went to YouTube and I searched embedded changelog, and I just searched those two things, and it came up with the embedded podcast we did here that you're aware of. And I go there, and there's this comment that's, like, 500 words deep. And I'm like, I had no idea, one, that this comment was here, and two, that is, like, you know, I was re-revelationed, I suppose, in terms of, like, how cool this move is, is that we've got this new

commentary level. And the person's like, I like this podcast. I love to hear more. And they kind of go into all this stuff. Now, the person doesn't have a username. They don't have an avatar. So that's kind of sad. But, you know, I'm still hopeful that there's more like that that are that are thicker. Jeez, y'all don't like that word. I don't just like it. I think thick is a good thing. Anyways, I won't go back there. It's a it's it's a exhaustive, you know, thoughtful comment that I haven't even read the whole thing yet. But I was like, wow, there's this super huge comment that somebody's like actually talking about relevant things and not how we suck. So that was cool. I love that. I like the.

you know, I was pushing for this because I was like, this is something, this is what we need to do. There's a whole audience there that we can tap into that we're not. And clips are great, but they're not the full length podcast. I'm now sad that there's, that when I share with people, we're on YouTube, that they're like, hey, did you just start producing this podcast? I'm like, nah, man, it's been like forever basically. And so we have this huge backlog that's not there. And that kind of makes me sad because there's a lot of visuals and a lot of just like seeing the reaction, like Garrett mentioned with Tim, just being able to see his pause or his thinking, you know, or my thinking whenever I'm talking or him pointing to his mini stacks behind him.

I think that's it's not for everybody, but there's a large majority of people who are gravitating more and more towards that who do listen on YouTube, pay attention when they want to. But when they want to, they can go and look at the screen, you know, and that's that's that's been my use case for it personally. And so I wanted that for us for so long. And I just felt not so much bored, but there was a missing necessary humanistic component that was visual that wasn't there. And so when you're audio only, I feel like you're stuck in this box. And I feel like we're now like we're like the genie out, you know, we're the cats out of the box, so to speak. We're able to explore the bigger world of YouTube and capture not so much more of an audience. But like, I think there's a lot of people there waiting, wanting what we produce. And now we're there.

in full form yeah so youtube here to stay um a new way to interact for sure and more and more integrations in the website i quite like that for example like the watch button that was something which um was one of the new things to drop on an episode uh it's getting a bit crowded uh maybe this one and but it's there you can click on it you can click on that and it'll pop in there and just look at that how amazing is that cool yeah it's cool right that's the good stuff right there and on the play bar if you go to an episodes page the play bar got a little wider and it has a watch button which is the same thing it'll pop it'll embed it underneath it once you click on it we don't auto embed because you know only when you want it on demand should it say listen this is watch yeah maybe listen and watch maybe yeah play and watch maybe listen and watch yeah that'd be a good improvement yeah but these are nice like you can watch it right here And they just get automatically expanded. I like that we are a commit-driven company, by the way. A lot of the features that get dropped, I just find them through commits. This is so basic. No problem and circumstance, you know, no blog posts, nothing. No, just we are a commit-driven company. So if you want to know what is happening at Changelog, follow the repository and just like look at the commits. That's right. We're very committed. Well, friends, I'm here with Samar Abbas, co-founder and CEO of Temporal. Temporal is the platform developers use to build invincible applications. But what exactly is Temporal? Samar, how do you describe what Temporal does?

I would say to explain temporal is one of the hardest challenges of my life. It's a developer platform and it's a paradigm shift. I've been doing this technology for almost like 15 years. The way I typically describe it, imagine like all of us when we were writing documents in the 90s, I used to use Microsoft Word. I love the entire experience, everything. But still, the thing that I hated the most is how many documents or how many edits I have lost because I forgot to save or like something bad happened and I lost my document. You get in the habit when you are writing up a document back in the 90s to control less, literally every sentence you write. But in the 2000s, Google Doc doesn't even have a save button. So I believe software developers are still living in the 90s era.

where majority of the code they are lighting is they have some state which needs to live beyond multiple request response. Majority of the development is load that state, apply an event, and then take some actions and store it back. 80% of the software development is this constant load and save. So that's exactly what Temporal does. What it gives you a platform where you write a function and during the execution of a function of failure happens, we will resurrect that function on a different host and continue executing where you left off without you as a developer writing a single line of code for it. Okay, if you're ready to leave the 90s and build like it's 2025, and you're ready to learn why companies like Netflix, DoorDash, and Stripe trust Temporal as they're secure.

scalable way to build invincible applications. Go to temporal.io. Once again, temporal.io. You can try the cloud for free or get started with open source. It all starts at temporal.io. Now we've got video first going on. It's time to get CPU officially launched. Turn that front upside down into a smile and an index, you know, something that's cool. Yeah. Very nice. Okay. Any infrastructure that we need to think about to talk about for CPU FM? You can share with him, Jared, what your thoughts are on the application. The plan is just to have a bog standard web app with a...

With RSS feeds, right? Nightly style? Like that is a box standard app. Oh, in terms of the actual software? Do you have a database? Do you need a CDN? What would that look like? It'll be a database. A CDN will probably be smart. But maybe we just drop it on R2. Probably similar to what we're running now for us, only it's going to be simpler and it's going to be a separate software stack. So probably going to go back and give Ruby on Rails another kick down the road and see. Interesting. Just because it's been a long time and I've been in Elixir land for almost 10 years now. And every time I write a little bit of Ruby code, I'm like, you know what? This is my first love. And so probably going to be a Rails app.

deploy it on fly. It'll be pretty simple. Have a backend, you know, write out HTML pages and RSS feeds. That's the plan so far. I haven't written a lick of code yet, so these things may change, but that's the plan. Nice. Keep it simple. Okay, yeah. Neon for the database, I'm imagining. Yeah, I'll probably just reuse all the stuff we've been using over here. Okay. Yeah. Public repo, private repo? Good question. Good question. Probably public. I don't see why not. Yeah. Okay. I'm not going to promise that, but I can't think of a reason why it wouldn't be public.

It's mostly the, the admin is going to be for like just managing the podcasts that are part of it. And then the code, the actual logic of it is going to just be in building basically a super feed for people and maybe custom feeds too. So you can get your CPU pods that you like, and maybe if you don't like one, uncheck it or something. I built that already for us. So rebuilding it over there will be straightforward. Okay. Which will require user accounts, of course, but, or would it? Maybe not. I don't know. I'll figure that out, but.

That's the plan. Pretty straightforward. Not much code. I don't see why we wouldn't open source it. Unless I'm really bad at Ruby now. You know, it's been a long time. It's embarrassing. I think that's more of a reason to open source it. You can ask for help. Contributions welcome. I haven't typed Rails new since probably 2015. So I'm kind of excited to like just type Rails new and see what happens.

Well, make sure you record that. I think many people will be interested in your reaction. I will. See, Gerhard's thinking. He's thinking about the content. That's what he's trying to ask you about, Gerhard. I know. He's like, how can we promote all these cool things? Yeah, that's what I'm thinking. Yeah, I mean, maybe record it. I don't know. I guess if you want that kind of content from me as I build out this new web app, which honestly.

is not a super exciting web app, but it's still, maybe I'll use cursor the whole way, you know, and then I'll just curse my way along and then just rewrite it myself. If you want that, let us know in Zulip, let us know in Zulip. Yeah, it's a new world. And I think seeing how you would approach that with your Rails knowledge, things that have genuinely improved from how you remember it, it's better what is worse because you have a unique perspective, which is the Elixir one running in the Elixir application for so many years. How does that compare to Ruby and Rails? I don't think many people did a switch back. I hear, I keep hearing about people going from Ruby and Rails to Elixir, but going back, I'm not aware of anyone doing that.

It didn't make the hacker news. It didn't appear on the changelog. This would be news. Well, I wouldn't be ditching Elixir because we'd still keep changelog.com over there. So I would be going back for a new app, but, you know, living in both worlds, which I'm happy to do. Yeah, that could be interesting. You know, old man fumbles around in the dark with Rails, you know. He yells at Rails and then he yells some more. Okay, yeah, that sounds interesting. Cool. Cool. So, Pipely. Let's see how long this is going to take. And by the way, this is like where the screen sharing will get into its own. So, that was a question that we had from Tim Huckin.

I'm not sure if I'm pronouncing that right. U-C-K-U-N. Why do you need a CDN if you have fly.io? And I replied to Zulip. That's like the sort of conversation that happened there. And I went through all the various things. So the reasons why we need a CDN, even though we have fly. So we can read it either in this GitHub discussion or Zulip. It's all there. So you can go and check it out. But the thing which I would like to talk about is that we are starting to have contributions to Pipely. And you may be wondering Pipely.

what pipely well what is pipely we renamed from pipedream to pipely why because pipedream is taken we can't get pipedream.com we've already established that that's a big company very successful i think vc funded so yeah so pipely.tech i think it's here to stay and pipely is the name of the repo whichever you go to it will just redirect you so the changelog pipely or the changelog pipedream there's a redirect and now we're having we had two contributions if you go to the road map the first one was make it easy to develop locally pull request seven from matt johnson that took a while to write some docker files explain how the pieces fit together there's a readme so if i go to pipely that we have docs which we didn't have before local dev so all of this explains what we're testing how we're testing quite a few things there so if you wanted to try pipely running it locally there's a doc that explains all of it So thank you, Matt, for this contribution. This one's great. And I'm sure that we will build on top of it. So Matt, did he do this himself and just document as he went? Or did he, like, do you know how he went about this? So I think there were moments when we got together. So we had, okay, let's go pipely.tech. Pipely.tech has the whole story. There's no more three mages or three wise men. It's just a world. So the image has changed. But we had, let's build a CD in part two with Matt and James. So they're there.

and we'll link to the video so you can go and watch it and make it easy to develop locally so this was kind of like a follow-up to that so matt did a bunch of things if you go to pipely.tech you can read the whole story right now it's the second article let's build a cdn part two and this one make it easy to develop locally is the first one so in preparation for that matt had to do a bunch of work right to understand how the pieces work what they are you know try running it locally and he cleaned all of those notes up and he contributed them to the repo so if anyone else wants to try this now they can now that's there So let us know what you think. So that was one. The second contribution, which was completely unexpected, is resolving the Varnish TLS issue.

So this was a big issue. This was a big issue. And we went deep. So Nabil Suleiman, he's someone that you may remember from a Ship It episode. We talked about the case cert. It was a simpler alternative to cert manager that Nabil wrote because cert manager was too complex. And I forget which episode exactly it was, but you can go and look it up. So he heard us talk about the issues that we had when it comes to Varnish connecting to TLS backends or TLS origins. And he wrote something that solves the problem. It's called TLS exterminator.

and now pipely is using tls exterminator to connect to origins that require tls termination how does it work in a nutshell We now spin two processes. We now spin Varnish and TLS exterminator. Varnish connects to TLS exterminator, which then that proxies requests to HTTPS backends. And that does a TLS termination and all of that. So with that, we can now, if I go back to, actually I was here, with that, we can now add feedback ends. Now these URLs, they have HTTPS. We could disable it. We can go via HTTP as well. This is something to discuss. We want to disable it. I think we should keep HTTPS on.

And if we want to keep HTTPS on, we need a component that terminates TLS between Varnish and the origin. So keep TLS on? Yeah. All right. So we're keeping TLS on. Great. Because again, HTTP currently is available, but I think we should disable that. So it's HTTPS only. We learned quite a bit with Nabil about why Varnish doesn't have support for TLS. So if we go to Varnish cache, why no SSL? There's a page on Varnish that talks about why SSL was not implemented. And you may be thinking, who wrote this? Poole Henning.

If anyone doesn't know who Poole Henning is, let's look him up. This was 2011, by the way. TLDR, before we move on, it's OpenSSL is too complex. And when it comes to the implementation, if this was implemented in Varnish, which would have complicated the code significantly, the SSL proxying would have required a separate process, which is basically what TLX exterminator is, a separate process. The difference is that not only would have Varnish been more complicated, it would have been slower. So the whole code that does with SSL would have slowed down Varnish. And that makes a lot of sense. So who is Poole Henning? Poole Henning camp.

He's a Danish computer developer, and he's known for work on previous projects such as FreeBSD and Varnish. So he's the guy that you can thank for FreeBSD. He had a significant contribution, and he is the top contributor on Varnish. Some would say Varnish is his idea. But what's really surprising, so let's go to PoolHeaning again. And he has not a FreeBSD. There's that. Okay, so we'll go through GitHub. phk.freebsd.dk. So FreeBSD.dk apparently is his TLD, and he just has a subdomain. I think that's really cool. So apparently Varnish has a Mower license.

I had no idea about that. And he's very transparent about the accounting. Like he runs a lot of the software behind the Varnish, like docs and a couple of other things. And he's very transparent about how he spends his time and how much he charges for it. And like, I was fascinated as an open source project, how transparent it is and who contributes the most and things like that. So popping the stack and just keeping going through who he is. So FreeBSD apparently.

md5 script, jails, nanokernels, time counters, and the bike shed. He invented the bike shed concept? Yep. He's the guy behind the solar. And look at this. When you refresh it, the color changes. So, bike shed.org. Bike shed.org is what we're looking at. Pool heading camp has to come on the changelog. I think it does. I think it does. And the pipe lead connection is just too strong to ignore. It's so strong. All right. So, that explains why Varnish doesn't have SSL. And Varnish Enterprise does, right? So, there's like the whole...

commercial aspect but varnish open source does not have ssl and there's a couple of ways to solve it um and we may have talked about this on the recording that's not public yet so with nabil so we will wait for that to to land but what does this mean in practice and this is where i go to the terminal so we're looking at pipely everything has been merged and anyone can follow along and we'll do the same here okay so let's do this alias j is for just right so just is something that i love and i think i mentioned about it just do it right it was like one of those reasons 16 i think not the last one the one before last so there's a bunch of recipes that people can run And just debug is the one that we'll look at now. This is in the context of Pipely. So Pipely, as you download it right now, today, this is what it has. So what it does behind the scenes, it's using Dagger. And the reason why it's using Dagger is because it needs to create a specific environment with different tools and it has to wire everything together. So we can use, in this case, we're using Dagger to publish, package the container and publish the container, even deploy the container. So deploys are a thing now. We have deploys wired up. So any commit to Pipely will go out and it will deploy a Pipely application.

and we'll see that in a minute but now i just want to look at debug so what debug does it adds some extra tools on top of the application container so what are the tools Let's just open it up and just have a quick look at what debug does. So debug, actually, I forget it's not here. It's in dagger, main.go. So debug. So for example, we get curl on top of the application container, which has just varnish. Tmox, HTOP, NeoVim, HTTP stat, Sasquatch, which is an interesting utility. It's like watch with some extra features. Gotop and Oha. And Oha, you will remember. And then just obviously. So it's just a way to interactively debug the container and then try a few things out without polluting your system. I think that's the key takeaway there. So let's just run that. Let's run debug.

And the terminal function is what puts us in that container. So I ran the command and right now I'm in a container and I have a bunch of two links available to me. So what are the two links? If I do just, again, just is there. I have a couple of commands to run. I could run these things locally, but really I just want all that to be wrapped because typos and a couple of other things. So what would you like me to run first? Just backends. Just backends, the first command. So let's see what just backends does. So it just wraps varnish ADM backend.list. Because varnish isn't running, there's no backends to list.

What would be a back-end? Back-end would be for example the changelog origin the changelog application the back-end would be the fees origin or the assets origin So this is where back-ends get plugged into varnish and varnish provides caching for those back-ends cool So how do we start varnish? Let's see if just up look at that see and that's why we have something like this So just up boom there it is. It's tmox is a terminal the terminal So there's quite a few things there. So just back-ends. There's nothing there. And if I do just check that's the one

it does the first request uh it fails and the second one we can see we got a 200 run it again it's really fast and again this is messed up all right it'll have to be horizontally i'm sorry it will have to be horizontally so just check it won't fit a lot but there you go there it is we can see http 200 okay and we can see that the request came from we got a hit it's a second hit from local so this came from varnish cool and if i do we do just just backends we see the two backends which are healthy cool what other commands should we run let's do bench cdn i think that's that's where actually bench origin so bench origin and you will recognize this this is going to oha and we are benchmarking that's beautiful yeah we're bench it's not as good as it runs locally there's a bit more detail but it's it's pretty decent i have to say so we have just benchmarked this is what we're doing here so this runs locally the benchmark runs locally but we are benchmarking the change log origin application which is production right now this is production okay yeah so we're benchmarking production how many requests per second 93.7 93 so about 90 requests per second now i'm in london

this is actually split between new jersey and ashburn virginia so there's two data centers it can go to either one it goes through the edge and then eventually connects there which then it has to connect to um i think the database it hits the database so 90 requests per second not great but only the cdn goes to the application directly so let's bench the cdn and we are sending a hundred thousand requests per second sorry hundred thousand requests not per second hundred thousand requests and let's see how long it takes so that took just under 10 seconds and we completed 10 000 requests per second 10 000 requests per second so the cdn we can see it's doing its job i'm connecting to it locally the latency is low and this is our changelog.com cdn all right so now let's benchmark let's go to cdn2 cdn2 is pipely deployed on fly that now proxies to the origin. So this is the new pipeline we're setting up. And I think we already had this, but how does it behave with the TLS proxying with all of that? Right now, we have all those things in place. And we're almost complete. Remember, we had about, I think, 10,000, 11,000, something like that. This one has 4,000 only. So it's slightly slower. It's going to cdn2.change.com. Now, the application itself has a shared CPU, only 256 gigs of RAM. So it's like the smallest, lowest, cheapest CDN instance, sorry, cheapest fly.io instance application that we can run.

So we could make it quicker, we could make it bigger, but that's not what I would like to show. What I would like to show is if we benchmark Varnish directly. 73,000. 73,000 requests per second. actually it's quicker it's a hundred and thirty two thousand the problem is the benchmark we're only sending a thousand requests so let's just make it a little bit more I just sent a bit more let's send varnish let's send to it let's go via HTTP one one just need a couple of things and let's go let's go a million so let's go a bit more so there's a benchmark varnish I missed something up let's see what did I miss up bench matters one one there we go I just made a typo all right let's benchmark varnish so we are sending it a million requests per second Where is this running? Everything's running locally. It's running inside of Dagger. Main request total. How many requests total? I think. A million request total, you said, right? Is that a million request total? We're sending a million. Oh, I made a typo. Actually, 10 million. We're sending 10 million requests to it. So let's see how does it behave exactly. Oops, we'll send 10 million requests, and we're more than halfway there. So if I do, if I go to this instance, remember, the same PopOS instance, and if I run Vtop, I can see how this instance, like what's happening here. You can see the CPUs. There's a lot of red. So this is now CPU bound, actually.

and everything is local so there's no network because it happens in the same container in the same namespace same everything which means that this is really as fast as you get it and there's our result that's how many requests per second varnish can serve 211 000 211 000 local so varnish isn't slow it's caching well we can look at the distribution and because we're right there where varnish is there's tls exterminator that needs to talk to which terminates tls right so that's an external process which that connects to the origin and it can connect to multiple origins so right now we have only changelog configured but we'll have feeds we'll have a couple more this will run next to varnish and i think the pieces are starting to come together any thoughts this is cool man would we like and here's a question would you like to scale those instances up to see how much faster they will go if we provide bigger instances well what are we getting right now against our so current setup so our current setup which uses fastly we're getting yes between 10 and 11 000 requests per second so we're about halfway there we're about halfway there yes

with the cheapest smallest instance we're about 4 000 there you're saying that's all fast gonna do is 10 to how many thousand 10 000 it was about 10 to 11 000 so there's a couple of things that play this is like the the pop which is closest to me i have seen it go faster so i've done a couple of other benchmarks sometimes goes to 16 000 17 000 so it can go faster um i think it just depends on network conditions load on their system but we are sharing right network with everybody but if i can push 11 000 requests per second that's a lot of requests per second by the way I think. Yeah, it doesn't matter. Like, it's 4,000. Yeah, so how fast is it? If we were to do here, you can see that right now, I'm downloading 1.27 gigabits per second. And my network connection goes more. My network connection goes all the way to 2 gigabits. Thank you. Right, so right now, I'm about 10,000, 11,000. So, basically, Fastly is limiting my one connection, I mean, I say one connection, one IP, to about 1.something gigabits per second. And maybe we could benchmark it elsewhere, but the point is, you don't want, like, one user to use all your available bandwidth, right? So you need to apply some throttling.

so let me show you something interesting if I for example let's just do bench so I do just bench and if I do bench let's do remember bunny we have bunny changelog.com remember that the other CDN this is how the other CDN behaves 1,700 so let's go to and I would like to go let's go 100,000 requests per second let's see how that behaves 100 000 requests total 10 000 requests totals you keep saying per second so you do that's what adam was trying to fix earlier too yeah yeah sorry 100 100 000 requests we're sending 100 000 requests and we want to see how many can you serve per second and what we see here is that it stopped at 2000 at about just about 3 000 requests they block you exactly they throttle And this was a surprise. This was a surprise, right? So they have some sort of protection because I could be DDoSing them. Sure. Imagine if it would be like a hundred of us doing the same thing. Yeah. I mean, we'd be sending hundreds of gigabytes and they would just, sorry, hundreds of gigabits. They have to consume that bandwidth too, like as an infrastructure. Exactly. Yeah.

So as you can see, I mean, I'm not liking this behavior. I mean, I can't benchmark it. So from a benchmarking, now it's resumed. See, so there must be... And then it stopped again. And then it stopped exactly. So it was just about 100 requests it led through. So it's just blocking me and then letting more requests... Are you doing this via an API key? How do you authenticate this? Is this just a... I'm not. I'm just hitting it as public like anyone. Okay, I was going to say, if you can do this via an authenticated way, then you can always just like pass a benchmark flag or something like that to get past this. Maybe, but... I'm just hypothesizing how I would build it if I was building it. I would allow somebody to benchmark my system because there's going to be times you want to benchmark your system.

I would definitely look into that, but I didn't have to do any such thing for Fastly or for Fly. I could just send all the traffic. Is that a good thing, though? I wonder if that's a good thing, because I'm just letting anybody benchmark him. You just sent 10 million requests to them. They can handle it. They didn't just break a sweat. You almost sent 10 million per second. No, no, no, no. That's too much. I need more computers. I need my Windows machine for that. That's right. Everything's not enough in that case. So my question, brass tacks, is probably fast enough. That's the question. Correct. And I feel like 4,000 requests a second versus the 10 to 11,000 you get on Fastly. Is that going to noticeably impact anybody?

And I would assume the answer is no. He's scaling that machine right now to test it out. I know he is. I'm doing it. He's always scaling stuff. Let's pause for one second. Remember earlier in the show when we were talking about Fly and being down and stuff like that? Right. That wasn't hate. That was just facts. Okay. What he's doing right now is in the moment of having a conversation is essentially upgrading that machine to be more performant. Taking it from a cheap box to a slightly more expensive box. Yeah. And this is all via the Fly command line. So cool.

It is the coolest tech, man. They really are doing some cool stuff there. I love it. Yeah, I'm just wondering if you were to replace Fastly with Pipely, how well it would behave? Would we get the same thing? And how does it compare? But for that, I promise one more thing. So I'm going to deliver on one more thing, right? All right. Did you notice anything different about my setup? Like in here in the terminal? No, no, no. Like looking at my camera, did you notice anything different? It's super black behind you. Are you in a whole different space? Well, it was black before. Yeah, it was always black. A better camera? Yeah, well, the black, it had some more detail, right? So before, the black wasn't quite as dark. Okay.

Whoa. You just green screen yourself? Not quite. All right. So something just went behind your head and it looks like Grafana. Yes. Grafana's behind your head, Gerhard. Yeah. So that's one of the early birthday presents, which I couldn't wait to open. Yeah, your birthday's coming right up, isn't it? It is. Yeah, it is. I think by the time this will be out. It'll be out. All right. You're listening to this. Fine, Gerhard. Tell him happy birthday. Thank you. I appreciate that. So I always wanted to have a big ass monitor. Like really, really big. So big. A bam, as they call it. A bam. There you go. I always wanted to have a bam. Bam. Bam. He's got one. And that's what happened behind me. Like the whole.

screen like the whole background is actually now a one giant screen this is a real screen back there yeah is it a tv screen or is it a tv screen it's a bam it's a hell's more about this yeah i've heard of this so what is it it's a it's a samsung s95d okay and it's a 65 inch tv so it's big And what it means is that I can talk to anyone and I can see exactly what's happening across every infrastructure. Right now, I have the changelog infrastructure running there. And do you see those spikes right there? Do you know what that spike is? You, just now. Yeah, exactly. That's me just now. I did that. I created that spike. That spike is so proud of himself. I did that spike. Yeah, you did. So that spike right there is the benchmark that went directly to Fly. Now, on the left-hand side, as you guys look at it, that's all the metrics coming from the Fly application, our Fly changelog application. On the right, it's all a honeycomb.

And because it's a bit blurry, you can't see the details, which is exactly what we would want, right? We don't want to advertise all the details. But really what's interesting is the shape of it. So honeycomb, I can't figure out how to automatically refresh. Grafana in Fly has that capability. So I just need to manually refresh. I have to click the refresh. So let me do that. There you go. He's leaning over. Leaning over and I'm hitting refresh. And then you should see that other half refresh, right? Like half of the background refreshed. Yes. And actually it's the same timestamp. So I need to go to the last 24 hours. There you go. Now we're looking at the last 24 hours.

do you see those spikes there yes those spikes is the benchmark which i did against fastly against our cdn so you can see that we never hit those levels under normal operating conditions right that's like 100x what we normally operate so maybe being able to serve 10 000 requests per second doesn't make that much difference since really we never hit those those levels i feel like we've gone to target the three of us y'all took me to the toy department and you said pick a toy I chose my toy. We went to the checkout. Target is a popular store. Here, by the way, if you didn't know Target. We checked out. We successfully paid. We've left. We've gone home, and you've not given me my toy. Right. Where is my toy? Well, I can't get them on it before you. You need to get it for yourself. No, I mean Pipely. Pipely. Pipely is the toy. So Pipely, if you go to cdn2.changelove.com, it runs. It now uses a component that terminates TLS to Origins, and now we need to add more Origins. While it's half as fast as the current CDN is,

we know that it can sustain all the load that we need to replace our CDN. So the toy will work. In terms of what comes next, we need to configure more origins. We need to get, for example, the feeds one, the assets one, and we need to scale the instances in such a way so they can handle the traffic. Right now, we only save the assets or we only save the responses in the actual memory. So we need to configure disk. There's a couple more things, a couple more knobs to configure, but this is getting closer and closer and closer. I feel like the real toy is the Samsung S95D 65-inch OLED HDR Pro glare-free with motion accelerator.

Gerhard, that sucker is expensive. Well, nice eBay. Half price. That's what I say. Brand new. So you just need to shop around. Do what Adam does. You know, do what Adam does. Well, I was going to let you know, you know, my birthday is like, well, just in case you're wondering. Mine's sooner. March 17th. I wasn't really jealous of all your computers you were talking about earlier. But now your screen is amazing. Holy cow. Yeah. All right. So you scaled up our Pipely to the Performance X1. Performance 1X. It didn't work. It failed. So maybe we're trying to scale too much.

let us down oh dang man it was cool until i didn't know i didn't know what would happen maybe we uh yeah let's see if you do fly ctl let me just do that let me go fly ctl machines list let's see what's going on uh live debugging why not so we see performance one two only two really but the rest could not be scaled and i don't know exactly why so let's just do that again let's do vm scale updating machine See, this other one just couldn't update it, and I'm not sure why exactly. That's the one in Heathrow waiting for machine to be happy. Sorry, to be healthy, to become happy. A healthy machine is a healthy machine. A happy machine is a healthy machine indeed. That's right. So that's still waiting for machine. Okay, so now it's moving to the next one. How many instances are we running right now? One, two, three, four, five, six, seven, eight, nine, ten, ten. So there's ten of them in different regions around the world. Yeah. And we've got two of the ten upgraded to performance 1x. The other ones are on shared CPU 1x. Exactly. Which has also...

10x the ram it looks like so the shared cpu is at 256 megabytes whereas the performance 1x is at 2048 exactly yep that's quite a scale yeah it's about 10x and i'm wondering if we do that 10x like how will it behave and what would that affect the bottom line of running pipely because you're now 10x our costs probably because you're upgraded every instance around the world i'm not sure how much it changes the cost i mean we can check it exactly to see how much that would cost and maybe we don't need 10 maybe we need just one per continent maybe that would be enough or one per like east coast west coast like this is like you remember like the old one so there's a couple of optimizations which we can we can change there so what's the question how much will it cost well i was just wondering how much extra it is we don't get the exact answer okay these are just concerns that i have as we move forward and then is 10 even the right number is a question i mean maybe it's smarter to have to leave it at the shared cpu 1x but have 30 of them versus 10 at the performance 1x for instance yeah i think so we can see that like that we went with the cheapest one right smallest one um so shared cpu 1x uh you get i think you get the bandwidth which depends on how much um how big the instance is right you get like a fair share of the bandwidth so these instances they were costing like less about two dollars per month um just in um compute costs we went to performance 1x which is 31 so that is more than a 10x jump maybe if we went to a 4x sorry shared cpu 4x which is about eight that would have been a 4x yeah 4x and also like a more realistic more realistic upgrade

But I wanted to make sure that we get, like, we get the higher tier ones. Performance 1X is, like, the lowest high tier one, which means that you get to full core. It's not getting throttled. And my assumption is you'll also get more bandwidth. And that's what we're testing here. If we go to, like, the next tier of instance, which is, like, compute optimized in a way, it's, like, a huge jump. But does that translate to bandwidth performance? So we're still going through that. I mean, we can try benchmarking it again to see how it behaves. And the reason why you could benchmark it again is because the pop, the one in Heathrow, has already scaled. So let's see, you know, how this one compares. We are pushing 480, 470.

480 okay so i think we'll get a similar result i think i need a 10 million again 480 i did uh i think 100 000 requests 100 000 requests in total yeah just to throw some node its way and see how that behaves and we're 4 000 so apparently scaling scaling up the instance did not increase the bandwidth interesting So the question would be, is this as much as we can get? And should we, could we go higher? I don't know. Is the limitation the network, is that what we just resolve to them? Because CPU and other things didn't really influence it? RAM didn't influence it? Yeah. So I would ask, for example, Fly, like how do they allocate network bandwidth based on instance size? Like how do those limits work? Yeah, that's not clear. And so that was the one question. And what I'm wondering, is 4,000 enough? Because we're looking at the graph, we're seeing the spikes, and apparently...

we never even hit 4 000 requests per second on our existing cdn it means that the ceiling is lower but since we're never hitting that ceiling maybe that's okay not to mention that we've seen bunny for example this is a perspective which i haven't seen in bunny before where we can see the throttling kicking in right we can't even benchmark it properly because it throttles you much earlier and i looked through the config i went through the settings like cdns apparently they're not all configure the same which is why i was looking varnish to see what can varnish do like where exactly is this bottleneck coming from and are we okay with the ceiling is it necessary to have this um this throttling in place For who? For, I guess, just the system, the uptime of the system. The vendors. Yeah, it would make sense for me, too. I mean, for us, like, I mean, we would be at least temporarily not, Pipely would not have a lot of users, I would say. Like, we would deploy Pipely on-prem, basically. Like, it would not be a service we're consuming. Pipely would be software we deploy for us to use. And so do we really need throttling if we're our own user and we control our systems? Oh, I see what you mean. You see what I'm saying? Like, because Bunny has it probably as a safeguard because they're public.

Whereas Pipely would be deployed for us in our use case, right? So we don't need throttling. No. It's deployed for us, but it'd be hit by randos around the world. True. So we can get dossed. Yeah, we could. That is a real possibility. I'm going to send us 5,000 requests a second and we're dossed. To one pop at least. To one, yeah. Yeah, exactly. So I think some form of rudimentary throttling makes a lot of sense. I don't think it would add very much in terms of software on our side to say.

you know you can make it very rudimentary this ip can only have so many requests a second none i think you're at least avoiding that low-hanging fruit i haven't looked into that but having this discussion is valuable i mean this is why you know like i don't think we can have the toy because we're still debating what the toy should be and how it should behave but it's exactly we're building our toy and i think this just makes it more real because these are the steps that we would go through before we take this toy into production and i think this is the perspective which is valuable to have right like the level of care and attention and detail that we go through to make sure that what we put out there will behave correctly and the comparison that we have right now is fastly which you know from some perspectives it behaves really well when we're seeing performance is amazing caching not so good but again it makes sense why they don't keep content in memory as for example we would because we would optimize for that which means that because we optimize for that we want to store as much of it in memory as possible memory that we pay for or right no discs that we pay for wherever they may be and then i think this will also um we'll have questions about like how should we size those instances we just heard that maybe the performance 1x maybe it's a bit too expensive because we need to run a bunch of them and how many remember first time we were running 16 maybe that's a bit too many

Maybe 10 is a better number, but even that might be too much. Now, if we're looking at the cost, right, we're paying $30 per instance. And if we have 10 of those, we'd be paying $300 per month for the compute. I think that's okay. I think that's not crazy in terms of cost. Let me ask you a question. Maybe this is a stupid question, but let's ask it anyways. We want to store, are we designing the system to be memory heavy where we have terabytes of memory available to the system so we can store all of our data in memory?

i don't think so on disk you have lots of memory available if we need it so i think that we need both i think that the data which is hot should reside in memory and think about how zfs works the file system and when you have an arc so this would be exactly that yeah we would want to store the most often accessed data in memory and the one which is least accessed on disk so i think we need both because the memory we can scale it i mean if we go back to two gigabytes of memory right for ten dollars let's say we go we keep the 1x we can get eight gigabytes of memory that doesn't seem a lot of memory for example i would know and this is like where the cache statistics would come in handy how much data do we frequently serve And I know that we have the peaks, right? When we release something, there's like a bulk of content that we serve often. How much is the bulk of, how much is the hot content? I don't have an answer to that. But all these things are getting us closer to those concerns, should I say, that the system will need to take care of. Honestly, I don't think that we should give it more than, for example, 16 gigs per instance. And even that might be a bit big. And I'm wondering whether all regions should have the same configuration. And I'm thinking no, because maybe in South America, and I know this for a fact, there's less traffic than, for example, in North America.

And maybe Oceania, I'm sorry, Asia, let's go with Asia. Again, it's less traffic than we have in North America and even Europe. So then I think the same configuration across all regions doesn't make sense. But knowing how much data is hot, I think that's something important. How would we know that? Just based on like stats to the direct path itself? Yeah, stats from the cache to see how much of those, like how much cache is being used. And are there any configurations? And I haven't even looked into this. Are there any configurations in terms of evictions? Like how frequently should we automatically drop content? I think this is where our cache hit ratio will come into play.

So if you don't store enough of it in memory, you will have a lower cache hit ratio. While if you store too much, maybe you're being wasteful. I mean, having a high cache ratio, cache hit ratio, while a lot of the data is not infrequently used, you're paying for memory that you don't need. The other question which I have is, are the NVMe disks fast enough? And if we think about Netflix, Netflix does the same thing, right? They put those big servers in ISPs. They cache the content on those big servers so that they can deliver them really quickly to customers, wherever they may be. We're not going to go there. So this is not that. But that's one pattern that they apply because they realize the importance of having lots of content close to users. Memory is not big enough. You need disks. Again, we're not there. We don't have that problem. Well, we're getting there. I think we have...

some decisions to make as we go. I think, roughly speaking, the dog hunts, I think 4,000 requests per second, well-managed, will be fine. And I think we'll find out otherwise and be able to scale one way or the other around such issues. What else is left in Pipely's roadmap as we look towards the future now? Because let's do next steps.

I think that now we are finding the place where we can add feeds backend. And we also need the static assets one. So I would add both. When we add them, we need to figure out, do we store all that in memory? And I think the answer is no, because especially static assets will use a lot of memory. But maybe disk.

and i think we should we should look into that can we configure different backends like how does that work we're basically getting like to the hard part of configuring varnish for our various backends and each back end needs to have a different behavior i think so that's something to look into logs and logs to honeycomb i think that is a much easier problem to solve because we would be using vector and now we have the building blocks which we have the first sidecar process if you want to think about that if you want to think about it like that which means that we have there's varnish and there's a couple of other smaller processes that support it we have tls exterminator that terminates tls to origins, to backends. The second one would be this, in my mind, it will be vector.dev, which is what we'd use for these logs. So vector.dev would get the logs from Varnish and send them to Honeycomb. So it's an integration which I've used before. I know how it works. It's very performant. It's very easy to configure. And then we'll have another helping process that would work in combination with Varnish to accomplish a certain task.

and honeycomb and s3 like all those it supports multiple syncs so collecting the logs on one side and just sending them like in multiple syncs that is very straightforward because it just handles all that itself And then really the last hard bit is purge across all application instances. And I think that one is like maybe a step too far to think about it now. But I think the way we, so first of all, now we are automatically, so we have an image to publish. We are deploying the application automatically through CI. That's like some plumbing that you want to have in place. We have support for TLS backends. And that was an important one, especially when it comes to other origins, right? Because let's say if we are running in Fly, we can connect, we can use the private network to connect to the changelog instance. But for external origins like feeds, it would need to go to HTTP.

because we didn't have hdps now we have hdps i think that's that's also like an important building block and now we're hitting these like benchmarking i won't say we got sidetracked by it but i think it's like something worth considering because you may end up building something that won't work we won't be able to use this to replace our current cdn and the goal is to be able to say with confidence that pipely is able to do the work that currently our cdn is doing and what does that mean from a configuration perspective from resources perspective i think everything adds up and it feels like it feels like we're more than halfway there for real i don't mean like will this work no no we're more than halfway there to replace fastly with pipely for us all right take us to the promised land give us that toy it won't be christmas it'll be before christmas adam when is your birthday march march okay that's too soon jared yours is july okay all i want for my birthday is pipely and a samsung that's 95d all right well uh i behaved very well i think and my wife must love me very much because it was yeah it was a present from her to me so um nice yeah so she um loves the nerd in me

She has no choice. If he doesn't love the nerd, I mean, what's left? Well, no. That's what you said, but this is another show for that. Don't answer that. Well, that's cool. I love this exploration. I like that there's a possibility to run your own thing like this, you know, to configure it to the way we want to. I mean, to zoom out, the challenge has been that it's been hard to configure Fastly, not as a CDM, but as a CDM we need. Our particular use case is not that Fastly is not good as a CDM. It's just it has not been highly configurable by us. It's been challenging over the years, mainly just because they're...

I think designed for different customer types. We're a different customer type. And we've been holding it not so much wrong, but it's just been a square peg around the whole kind of thing where it's not perfect for our particular CDN needs. I think we've had lots of cache misses over the years. Like, why is our stuff not cached? It seems like it should be. You know, we're not prioritized as a, you know, as a thing to serve because the way the system works, you know, and that's just it. And we're designing something that serves that kind of system where it serves the data.

Holds more memory has more available to it. It's not a mess. That's right. That's cool. Very cool man. That's when you build a circle man. I can't believe how How cool this stuff is you built man. That's awesome. Thank you. It's coming together It's I love the TV too. Yeah, it is like a well-rounded experience, right? So the idea is to be able to have the TV on now It's a bit right and it's it's running a little bit hot I can feel it. It's not it's not winter yet. I don't need heating in the office Well, it's like the end of the winter but

i'm able to see a lot of metrics i think that's something that um i always loved to be able to see how things behave and when they misbehave to be able to see and understand at a glance what is wrong are getting ddos am i running out of memory like which instances is problematic and i think this is just like a starting point i literally threw the two dashboards that we have over there but i haven't optimized them in any way and i think having something like this just makes it in a more breathing living system well yeah you can see in real time like what's happening the metrics is the life you know it's each you know organs that's your own knock yeah Yeah. It is super cool. Super cool. I'm jealous. I want one. I inspired you.

the christmas is far away right people can save uh i know that we did and uh it's been in the making for a long time so before before i could get this i had to get a system that is able to power it that's what pop os does one of the things which it does it has a gpu right which is powerful enough to be able to to power it i have another monitor here which does like screaming rings i can change things here and set things up a system that's able to be on and that's not too loud that was like another consideration a black wall so it blends nicely it was like so many things like years in the making as likely as years in the making that's right and it'll be just as beautiful as that i would love a tour a home lab tour i would love that coming soon to well i'm working towards that uh but i can i can show you one more thing which was not planned i know we're a bit over time but if you want to see one more thing i'll show you my m25 every year i basically uh take one of these machines online and i have like a 24 25 so this is this year this is the machine that came online it's running through nas as you can see it's an i9 990k it has 128 gigs of ram so it's like fully maxed out basically and this is how i'm just moving the screen a little bit uh this is um

You can see all the storage has two pools. It has an SSD pool and an HDD pool. So spinning disks and some slower SSDs. They're the Evo, I think, 870. And it's something that you need to have in place to be able to have decent storage between Linux and Windows and Mac and everything to just work. So that was one of the projects and I didn't have time to talk about it. Maybe next time. I know that you are a true nice user, Adam. ZFS and all that stuff. Yeah, several pools, several things similar to this. Slightly beefier machine. It's a...

uh a xeon processor and it's a 4210 i want to say silver yeah i think there's 100 some gigs of ram i want to say 192 maybe 128 okay it's something like that it's not 256 i know that for sure right i don't have a need for it i mean it's nice it's a tinkerer's dream to have lots of ram in the zfs system but i just don't need it it's just you know i thought myself just want to have it just to have it and i'm like that doesn't make any sense you know like you spend the money on the ram just spend the discs instead yeah you know because discs are expensive yeah for video storage you know something that that you will you would need be sure to do editing and especially if you edit it from multiple machines you need like a fast network a couple of other things but yeah my home lab's suffering right now i don't have 10 gigabit everywhere i do have it in the network i just don't have it everywhere um so i'm in the process of fixing that there's some some slight life updates for me that will make it more important i should just say i had a flood in the studio and i don't think i can stay here anymore let's just say i gotta go home so i'm turning my home office into a true home lab and work lab and That's in the making. So it's a bummer. More techno team. Yeah, you know, I mean, I'll be close to the things I play with more frequently. I feel like I've always been like two location and it's been challenging because like right now I can't access TrueNAS. It's at home. I can't access that Windows PC. It's at home. It's in the home lab, you know? And I've just sort of like stripped away more and more here to the point that it's like, it doesn't make sense to stay any longer. Well, your background will change. It's a pretty cool background that you have. Yeah, that's the thing. I got to make sure it's, you know, video ready. And that's, you know, I got a month to do that basically. Well, I know we went a bit long. I know we covered a lot of stuff.

I dig this. I love it. I'm glad you showed this. I would love on Make It Work. Do you mind if I promote that? No, go for it. Go for it. Yeah, go for it. I would love to see a tour of whatever you can share. It could just be iPhone. It could be low produced. I don't really care. I just want to see what you're doing because, you know, what I love talking to Tim about in particular, well, I like him as a human being. He's so cool. And I truly think we're not just friends on the podcast, but I think, you know, if he wasn't 2,000 miles away, I would hang out with him and spend time. Same with you. And you're a lot more than 2,000 miles away. Yeah. But I love, there's not many geeks I can meet that like nerd out on hardware.

Like you do and like he does and a couple others do out there that are friendly in the world like Tom Lawrence We met him years ago at a Microsoft something or other I think in New York I haven't reached out since then he's become more and more famous since then so now I just watch him on YouTube, you know, and I appreciate his takes and stuff like that But there's not a lot of geeky nerds who nerd out in hardware like for no reason like we do like we build things we we want to need and so we make a need for it you know what i mean yeah maybe there's some true need but you're like you justify it like this tv behind you because like why why not have a knock like jared said when i have this you know big thing behind you and not let it be a green screen let it be a real thing yeah just because you know so make it work dot fm to not hide the url dot tv that's what i'll say dot tv dot tv that's all new by the way that that oh gosh um yeah And that is, jeez, I keep fat fingering it. I put a comma there instead. I haven't been there in a bit. To .fm. So this is still running from your home lab, right? No, actually, this is running on fly, and it has a CD in the front. Yeah. Because last time it was on your home lab stuff. It was, yeah. Was it Jellyfin? Well, Jellyfin is still on my home lab. Jellyfin, like the media still served from the air because of the iGPU. But...

Actually, like this one, if I would just click on this one. So there's a couple of things here and I'm logged in. So it's obviously the episode, like the audio, which is coming from Transistor. So there's like an embed. And there's obviously the embed video. And as a member, when you sign in, you get the whole thing. This is served from the CDN directly. So this is like the CDN content. And there's also Jellyfin. So once you log in, I see the quality for this one wasn't very good. That's something that I'm still working on. And that's why I mentioned that I have to record my screen locally, which is what I did for this, because Riverside is not great with screen recording. It improved it, but it's not there yet. The quality is not as high.

Distributed podcasting is so hard. It really is. Because you want to share that screen with us, but then counting on Riverside to record it in a resolution that is good for long-term uses. Yeah. Makeitwork.tv and .fm if you want to go the audio route only, but .tv is where you said to go, so go there instead. Yeah, man, I want a studio tour. I want something. Don't take six months. Do the simple version, Gerhard. Or just, hey, listen, we can just Zoom and just show me the real, you know? Okay. I mean, FaceTime. You can just show it to me. Yeah, we can definitely do that. That's much easier. It's a whole backlog that I have to go through.

yes so that's i'm still working on that i it took me such a long time to find like a good editor and i think i finally have him it took me at least four months five months of proper searching yeah to get someone that i'm also able to afford because this is still like everything self-funded uh but it works and uh my first i need to make make it work work before i can you know but um even like make it work on tv uh now there are subscribers and there's like holding like uh members like people can pay for it um so that's up and coming that was something we did talk about cpu for you so i'm hoping you're still excited i am i am yeah we're making steps so maybe I'm keen to be part of that. I just did not have time between everything. Same. I've been focused on getting the agreement solid. I want to make a solid promise and have it be clear to folks. I think that's a simple thing, but understanding your terms between the people you're going to serve, I feel like you've got to examine that and have clarity there.

Cool. Man, this has been a fun Kaizen, a deep Kaizen. If you've stuck around to now, holy moly, I'm not sure what's getting cut, but wow. You are a trooper, you're a super fan, and you should be a Plus Plus member. I'm not going to force you, but changelaw.com slash Plus Plus. It is better. Bye, friends. See you in the next one. See you in the next one. Kaizen. Kaizen!

Alright, that is ChangeLog for this week. Thanks for Kaizen-ing with us. For the entire saga, head to changelog.com slash topic slash Kaizen. There you'll find all 18 Kaizen episodes for your listening and now watching. Pleasure. Thanks again to our sponsors of this episode. Please support them because they support us. Also, because they have awesome products and services. Thanks to Retool, to Sentry, and to Temporal. Links in the show notes. You know what to do. And thanks as always to Breakmaster Cylinder, the best beat freak in the entire universe. I think so. Do you? I'm sure you do.

Next week on Changelog, news on Monday, anti-res, yes, the creator of Redis, on Wednesday. And on Friday, our first ever game of Friendly Feud. Have a great weekend. Like and subscribe on YouTube if you dig it. And let's talk again real soon.

We'll be right back.