Sam Newman is the author who wrote probably the most widely read book on microservices, and yet, as you heard, he said that microservices are an architecture of a last resort. But why? Sam was in the room when the term microservices was coined. I find it to be one of the most thoughtful engineers on this topic, as well as on resilient systems, and his latest book, Building Resilience Distributed Systems, is out now. Today we cover specs versus code as a source of truth, and why Sam says that production is truth.
I'd uphold into keys versus fingerprints, and why I should avoid idobodistic keys when possible. The four dimensions of resilience, robustness, rebound, graceful extensibility, and sustained adaptability. Cognitive, depth, and cognitive surrender, and Sam's advice on how to better use AI so avoid these, and many more. If you want to understand what it takes to build resilient systems, from an expert who has been needed to spend this topic for templates years, this episode is for you. This episode is presented by Turbo Buffer, the AI retrieval layer built on object storage.
It's fast, cheap, and extremely scalable. And drop a cognition notion, and many others are happy customers. Before we start, let me mention Tire, our seasoned sponsor, and our product and team, I'm very bullish on. You probably already know that Git hosting has a problem, AI agents. Git is increasingly becoming a bottleneck for modern, and agent-heavy software development.
And cheers like you or me are creating more code with agents, build agents much more code, and most of us are running parallel agents, which push way more code. GitHub is clearly starting to keep up and has frequent downages. So what's the solution? Entire was founded by GitHub's last CEO, Thomas Dumpka, who by the way was a guest on a podcast when he was still at GitHub, and Thomas revealed Git hosting for the agenda gara from scratch. Entire was built to be very fast and to have your repos regionally close to you, to a loose latency, and this allows for fleets of agents to push in parallel.
Entire's performance is next level. Their repo can handle 418 pushes per second, which is up to eight and nine times faster than every competitor on the market. When GitHub is down, you can still keep working, and you don't even need to migrate away from GitHub. You just sign up to Entire, and the platform mirrors your repo. And here's anything.
Have you ever wondered what problem resulted in this specific code being generated? I find that the prompt and conversation with the agent carries more information than the PR itself oftentimes. Entire captures all the prompt history with your agent right inside of the repo, easy to check back. And it's got a pretty innovative UI to show all of this. If you're looking for Git hosting that works, even when GitHub is down, head to Entire.i will slash pragmatic, install the CLI, and mirror your repo with a click.
I've already done it, and I love it. Oh, and did I mention that it works with any agent and it's open source? Sam, it's great to have you in the podcast. Thank you so much for having me. I'm happy to be here.
Really happy to be here. Big fan of the show. As I say, I'm not gonna lie, I've become a bit of a millennial, and I think I've watched a lot of shorts rather than watching the whole things now and becoming my son. But I was happy enough in general, right now, I'm wherever I am now. Well, so let's go back to the beginning.
You've been known for a lot of your books, your takes on a resilient system, microservices, before that you worked at ThoughtWorks, but let's go into the beginning. How did you get interested in tech and computers? I'm quite fortunate living in UK, we had places and access to computers quite early on. We had an Acorn, an electron at home, which was sort of one of the generations of the BBC computers that came off the BBCB. And then I got a Spectrum 128.3, which is the Spectrum that had the built-in tape drive.
This is a way before the Commodore's. I think some of this was the people I think I'm pretty sure that the Acorn electron was pre-commodal, but the Spectrum would have been around the same time. UK was a bit more of a Spectrum and BBC market, whereas the US, the Commodore's and the Atari's and things are a bit bigger. So that sort of got me a bit interested. And I was sort of stayed interested in it, down at a computer.
I did a very boring thing. I went to uni. I did a software engineering degree at a place called the Montford University, which the running joke in my family is it's the in the top five universities in the Leicester area. There are only two universities in Leicester. I mean, it was, I loved it actually, the great class, it's basically no way I'd see you sit at this.
And I just did work after that. I got a great back then we did these things called sandwich courses. Yeah, and so basically the idea was to do two years of study, you do a year in industry, it was paid, but you effectively going in as a student and then you did your final year. Oh, no. Which is actually fantastic because I kind of was drifting a bit and rather I did my year in industry.
So many things clicked. It meant as much focus going in and I got to do work for a company called the Nucal GCL, and then it was called Alstom. And we did software for the European Space Agency and gas reasoning. But yeah, it was really cool. I joined about the first time at that and then I went back after.
And when I went back, it was just around the first dot-com boom. And so everybody left. So I went from being the person that was doing like nothing to a person did everything. But I remember my first week there, I was given a full-trans 77 code base. So full-trans 77 was released in the year I was born.
And every single for loop had a break condition because we're basically doing analysis on these unique machines. If the models and the time on missions are expensive and we were worried about acting to infinite loops, this is us trying to solve the halting problem basically. So every for loop across these eight thousand files had a break condition of 99,999 loops. And if you went over that, the program basically broken up. All to the models have got bigger.
The computer got more powerful. So it's going to change the world by hand back then. By hand? Well, this is one of the nicest things that ends up done to me in a professional career. Because for trans 77 space, the old punch card limitations to an extent.
So your lines can't go very long. So it's all that fit. You can't go for nine. And I reckon we'll take about two weeks to do it by hand or and instead of book, of course, you can read this and you get me a copy of the O'Reilly said an odd book. And so I spent two days learning how to do regular expressions.
And I wrote a refactoring using regular expressions accounting for line. So I think the closest I've ever come to think like a god, I did it. And I thought, that's the thrill. That's the absolute thrill. I'm not sure if it's a joke or a story, but you know, the two, the two builders need to build a wall.
And one of them starts to build a wall one by one. And the other one's kind of drawing making plans and like, I don't know like take us to me days to build a wall. And for two days, this guy was nothing in the builder building. It was really good. And then on day three, this builder like build a machine that builds, or like the machine that builds a wall for them.
And then sometimes, sometimes it cool, folks down the moment, you said everything. I think I love that story. That story is true. It's all true. It's one of the earliest times I can remember learning something and having such a visceral reaction And I remember just towards when it does in footwork So some grads in and they asked tell some stories about like what happened in previous career I told the story I said I had to deny made the models bigger and then one of the one of the grads Why didn't you just replace it with a constant?
So you could just change the number once anytime in the future and I was like I thought I'm learning once again. Yeah, so that's possible. I love it It's like you you get humbled all the time. It's industry you get to learn stuff Yes, there's reinvention that we were going on, but you were inventing better wheels a whole time So I think it's just nice to remember this every now and then I'm sure everyone who's built code and programs like That's somebody have the realization that you do feel like a god that you can do things with these things Which is just so difficult to do by hand, right? Like there's I think we forget about that one That's kind of day today and we're just like doing our stuff or even like right now using LMS which also incredible magical stuff That would have been impossible, but like it's kind of cool that we are in charge We can do amazing stuff you get that little fizzy dopamine hit and I think that's like which rubbing with a bit now is that People are getting further away from the code. They're trying to work out like how do I still get that kind of dopamine in the same way?
It's not as much an issue for me because I've Later on you know I've been doing this almost 30 years now and I've already been mostly working at the area of architecture from So architecture infrastructure for a while, so I feel a little bit more removed But it's yeah talking to my son is a programmer. He's got into it around an interesting time right now And then you eventually made your words of thought works where you spent a bunch of time What kinds of things did you do at thought works and what are what I think that you learned there? I did anything I mean it was I joined when they were probably just under 400 people left with about 4,000 and I worked in a good workplace I joined a good time because a lot of the I mean we were very much an XP shop and I think that the calls were in program Yeah, I think the call the business still kind of find every year to speak to people there I can't speak like what was that influenced by I can can't back speak. Yeah, no that they should program is very much It's it's this Ken's extreme programming. There was a bit of a London flavor So the kind of London school of texture development and extreme extreme things. I'm actually see around here
So what's the London flavor of? Sort of like if you want to know you're pretty to re growing objects oriented software guided by tests one of the worst title to books In existence or in by a couple of friends of mine Sleeveer even in that price and I think that was a lot of people in the community around here They knew that as a rich meet-up scene And so I kind of got introduced to people through there and came in and I was Same time was offered job by somebody else was a bank the bank could tell me exactly what I was gonna be doing and thought works Basically the recruiter's I'm now with my very good friends and my oldest friends now She said to me, I don't know what you're doing and I like that sounds like a job for me I went for that one and this is the time when like you know we talk back into integration But the tooling wasn't there and talks about CDD and the tooling wasn't there So it was in you know It was around that sort of time that we started work I started building a bill pipeline stuff in about 2004 and then I worked day by day five My first tech lead Worked on something that they did for other. Yeah, yeah It's my first second of thought works. I was on the people that interviewed Jez Humboldt, a backfill for me and a project No way!
I think we used a backfill because I was like I'm sort of half this is happening half that for you so I see myself So we needed someone to take over my job and so I went to do Jez And so I was around a lot of interesting things were happening So I was fortunate enough to be there while we were sort of kicking those ideas around them Obviously people like Dave and and Jez put those ideas together really really well He was also rounded by everyone talking thinking about these ideas I think you've got that idea very early on that this stuff's all up for grabs these things aren't set in stone that we can Push and when you're saying these things like you mentioned it gets a new delivery you mentioned what what I don't say I'm just automation bill pipeline optimization infrastructure automation I'm an over to the US in 2007 actually to work in a big I actually worked at Google for like a year and a half For for us the thoughts employee Wow in where is we were doing we're there doing projects This all been talked about publicly is a great write-up by Mike Land all about our experiences It was me if you mean if you might always colleagues and people with a lot of you guys and people from a pivotal the old pivotal pre There they were not for their really excellent I drew a call back when it was just rub these company
And we were there to teach Google is primarily how to write automated tests and how to write code those testable And so I got out there and I was talking to people out of the value stuff It's like we were going into these enterprise companies doing test automation and stuff like Selenium And you go around talking to people and it's like people didn't even conversations with people like Yahoo And even big chunks of Google who did not think you could do functional automated testing of websites and stuff I guess because they just never did it right. Yeah, absolutely I hate it whenever I haven't pushed myself or like found a better way of doing something I was lucky enough that early days and thought I was just a constant Chirr to my dears and reinvention. I mean goodness knows how many marketing frameworks and inversion control things I was I built two different replacements of cruise control myself, right? And so But like you know It was at that time I created a new deploy my friends Nick Ashley and Grand Tackley which was like the first day to be fracturing tool and all these sorts of things It didn't exist and needed them and it was like if you want to do it I was in mindset And so I think I was always this idea working at the works was always like what's next and I was quite fortunate that
Because I was growing so much there's always things needed to be done and my default is always just a yes day everything So I got trouble over the world is trying to think about my wife He thought works most of my best friends will come from the works now still today So although it's not been there for a long time It was kind of really formative I think in terms of that and it is really interesting because Thoughtworks has a Alumni pipeline people used to work there who are now really well in the industry for their ideas for for think differently Some of the educate people others have created businesses to build software in better ways in certain dimensions It's just interesting because like usually I wouldn't think of I'll just like put it in the box I can be a product companies like Google like meta like they have they build on products And you have like different tiers of these companies some kind of for the best people and they have next time they should Googled with don't for this meta to smacks and there's some companies maybe who are like Lawrence here And then you have the so-called consultancies who are you know They don't build their own stuff or they don't do too much of it They come in and they help companies build or you have a problem you hire a group of experts and thought work technically is in this group And usually you would not think that there's too much innovation going on in these places
But with all works pushed me yourself and other people actually you can have like when you are the best in what you do Lovely can I have a lot of innovation that happening there as well right? Yeah, and it was innovation the different set of constraints I mean I would say there aren't that many people that have come from thought works that were really great start-up product people I mean a lot of you know a lot of thought workers make exit scale up people want you They're actually very few real entrepreneurs that came out the thought machine I know a few of them there aren't many of them But I think there was always the opportunity to innovate even within a constrained enterprise environment So much while working conditions as a consultancy who mostly did coastal delivery at that time We were doing kind of coastal delivery bit of enablement through delivery Your working conditions are often based on what the client does And unless you're there explicitly to change how the client built software So what you were what you got but that was self-petuating because we were already used to sort of challenging and pushing in trying to do new things That's very carried on and so I think we grew aggressively built that aggressively rotating people through that spirit coming carried on I mean that we had to start most engagements with convincing people by testing was a good idea alone TDD Yeah, I imagine how the conversation went like a manager going like well
What would you pair then I have like you know like eight engineers and now you're making it for which sounds like a terrible thing to do Which and then this is I mean I good fortune because the people because I got to know Marty and found quite well You know he supported me a lot when I first spoke I still remember when he said well programming is not typing I said fundamentally make it click for me was an hour's conversations clients if the client thinks that programming is typing and That's why they don't want you to pay a program Well, then you've got a fundamental problem. These we tell with it's not a conversation about hearing the conversations What do you think programming actually is well? But let's bring this a little bit forward because right now we're having the same conversation right for a long for a lot of CEOs Even founders are thinking who have been building tech companies You know they all look at what what is the most time consuming part of what we do in building software 50% of time has So I've been writing the code so a lot of people think you know like programming is cold and lums are really good at training out codes So what is programming? Okay? There's the there's programming I think it's Peter Noah's article the guy who came up with Marcus North for me He's nor in Marcus North for he writes paper years ago Which I'm the trueing of all four and he sort describes
Like the the program is being a theory that lives in the mind of the developers that it's actually us together working together collectively Who builds a program the fact that we happen to make that program work through code The code is really a side effects of how we work together And I think product development is just that extended right what we're building is collectively We are building a system together and the code is one of the mechanisms we use to deliver system It's not the only thing so I think politically that's what it's all about it is that we focus on the code because that's Historically being where you sort of spend the time but I mean This is also classic theory of constraint stuff though right if you speed one bit up you get constraints elsewhere and it exposes Fiction points elsewhere I would also argue that a lot of the time coding was not the problem of the bottle of the necklace how the coding was done The life the back cycles and things I do come back to that Peter or piece again because it's like my brand story reference to reason You know to pay for capital to bet but it's like building a software is a collective It's a collective endeavor is people working together who build software Kent put it in his type first book that software design isn't exercise and human relationships right AI doesn't change that and I think all
It's really happened over say the last 15 years and my experience is that we've got a bit better understanding that we need better working relationships I'll put it for example But that's what the programing is the hour of creating is the collective operating software and it's not just developers do that Never has me and so after thought works you you've run independent right? I did three startups and I worked out for different reasons So don't ask me for advice and how to start up and then I went independent Yeah, I was looking at straight at the time and move back to Australia and yeah I've been independent ever since doing kind of a mix of stuff really ever since I've written the first book of that micro services I thought my spare time at all works and then that book came out and I've been at the long time I realized that I was a day becoming institutionalized you get to a point with a company It's like I think this thing is companies do to change and I've tried changing those things and they don't want me to change them So either I'm bad at changing them true all that I think it needs to be changed It's like I could hang around and get me to go now. I was going now So I went off and I thought I went independent It's been really interesting ever since because it's all the one level the shackles are off The problem of working for yourself though is that the boss is frequently in our soul
So yeah current Sam frequently is cursing past them for the commitments he's made That's one that's not good. Yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah I mean as I say I'm almost if the I'm still not loading the lessons of that right and I've made those mistakes with off years now So I want to talk about microservices You have written the book building microservices in in 2015 you start to write in 2014 and as reference Uber started to talk about how they have thousands of microservices in 2015 So you were kind of writing this book right as the concept of microservices was starting to go mainstream across startups Do you remember how and when microservices took off and where was it even coin was it? Mark? Martin Fowler is doing this going back in a day. So you got to remember again. It's like something in the water
I thought you have a bushfire that was not built up for the bush market off and said this would have been 2011 to 2012 James was talking to a bunch of Different companies who were building would have classed back then as service or into architectures and earlier there being this kind of What we refer to as the gorilla SOA movement which is a name to simplify service or architecture You know she gave us a rest worked on by people like yeah, there's original service or architecture had like this the sex and all format Wasn't a wisdom and simple object access protocol never the word simple be less. Oh, yeah, so yeah, so yeah So with simple and is all complex thing ever exactly so yet It's whole heavyweight and then people up to you in Weber and In Robinson and Jim Weber had like being pushing centralized at Manchester like make the endpoints mark of the pipes dumb to keep things simple Smarts it's all background. We were also is timing a lot of work around like I was running to continue to live Perhaps it's Europe at time and so a lot of my engagements were like optimizing the final model So you've written the code and it's a how I get it here into production as quick and effective as possible So basically taking the idea to continue to delivery so I was around with Jez and days I work in 2006
I've had the conversation with them. That was what happening James was talking to companies that were Creating so such architectures where they could replace the service quite quickly and so basically you look at the sets of finer grain Form of a so that was all happening to James when we had this architectural symposium up in the late districts And James pitched this idea. He was called the micro apps at the time in this room and somebody else in the room said I'm all like services don't remove it. There's me that ten of us in the room Most was one over and in James and Ian say James and Martin wrote the first post so the first paper on Martin's website was written by It was a James's original kind of discovery of this pattern legal micro services at the same time I was doing a lot of stuff and infrastructure automation and so I was coming in it from a different lens I think James was coming in more from an architectural lens and I was coming into it from a Contingent delivery lens I spent at this point almost ten years without being my focus And so I was starting to see the Architectural impediment so if I want to change things frequently, but the things I change are integrated other things like a model Now that's I would describe it exactly right. I mean I did a talk called designing for every release in 2012
Which is so touching it so basically James had to sort of put a name to this stuff and put some Apparently and I thought hang on it like stuff I was talking about over here seems these things in quite sensible and I always wanted to write a book I mean honestly I went to write an a writing book because that's a notebook. I was giving up. Yep And so I just started writing I've always found writing quite I'm quite lucky that I find writing enjoyable and quite easy So I just yeah sat down and wrote the book and that's it But for me it was an extension of continuous delivery and it was a style of architecture That was only possible because of continuing because of DevOps we had these ideas before I mean Erlang as isn't a representation These ideas but making it real practical and attractable thing But I mean really fundamentally it's just the whole thing is been an exercise in smuggling through Concepts of 1970s, which is fine by me right if I get people to pay attention to information hiding and And cutting cohesion that's all right But then as you said like this was due to the spark of bushfire and then the bushfire that starts like suddenly again I think uber was a very notable company starting to talk in public about having thousands of microservices on
Conference like in full queue and I got people's attention and then it kind of seemed overnight a lot of stars started to say like You know what let's just do microservices There was a bit of re-badging going on I mean I was looking at the first book like met Ben Christensen failure on through the our conference Ben was and Ben was looking at looking after performance and sort of reliability and Netflix and also in people are yeah He needed to earn a really state of calm. Are you in Australia? And these are companies that were very much pushing envelope in different ways And so I had two very good real-world case that they should drop off my book But most other people weren't talking as publicly around that and also before Netflix referred to their architecture So they just referred to its fine-grained SOA and to an extent You know other than my son is what happened at uber it elsewhere We didn't have a different word for it then of course what happened was that everyone just called their s my architecture's microservices Which is sort of you know semantic diffusion is thing is to be expected So I just kept fighting the good fight for like it should depend upon the ability and you know I was positioned it as it's an opinionated style of SOA you may not agree with the opinions That's fine to something else but and they're also opinion so then what is the difference between microservices and what was it our service?
So I wanted architecture. Sorry. Sorry. What are the differences? In your very I have my definition you might have my definition but I'm the one that looks on it's like my definition wins That's how I always be my argument right right your own down book I always put into it as it's there are two kind of very strong opinions I think we hold with Microsoft it's the first and it's most important is the Microsoft needs to be independently deployed So you know deployment you know changing is to be a single Microsoft's boundary And you know rather than undiplying everything together Right And the second one which is which I think there are exceptions to is that you're kind of default way for the front-end services around your business domain It is taking the concepts of you couldn't use them. I don't have to but rather than a lot of SOA architectures were dominated to an extent by
tenability competition the track transfer entire architectures layered architectures where you've picked your service decomposition based on nature tech Rather thinking about the business functionality. That's a bit softer Independ deploy but it is a main thing though. Okay. Well, I'm not thinking back at uber I was at uber at the time when your book was starting to become really popular I was there from 2016 and we had infinitely deployable services We didn't really think too much about the businesses or domain they had obviously their interfaces They had contracts that they offered increasingly strongly typed and we use The other way to do it, but it was just more like I mean as you can imagine there was just devs often time not fair experience That's where like alright like let's build a new service here's the interface here's and then anyone can use it Which actually like there were these things were a devil even a not very experienced stuff The boy the microservice and other start to use it and suddenly it became years later They would be these exercises to figure out with the whole path and the labels services and then this like loan service was on the path of like Somebody I need to be re-recordous. It was a really high-tier service because a lot of others were using it I mean there was always I mean also nice little environment so much something but sound clouds in about that So I mean and this is also like about corporate culture and like what was acceptable like in that kind of environment
Just gonna do stuff. It's absolutely fine And I think that's why uber for example I went to I just talked with Tom Tom the seat at the time about like what was a real deal about microservice And he said uber like he never really wanted that many services But it was just more of a constraint because we were had the monolith the story was like which he told it off As well link it up show us below that uber had this big monolith and it was really slow It was slowing delivery down and they needed they needed to move fast because all the competitors were out there So they said all right like they wanted to get rid of the model to put together small services But they knew unless they had a strong mandate that you cannot add new stuff to it This will never happen So they said the mandate was everything you need to be your own service you take care of it We don't care and suddenly so many independent services were born Which was not the goal and years after the fact that when we were decomposed the monolith they're now looking at Maybe sensitive emerging based on business the main etc But this already wasn't really told by the time it was just oh look like uber has thousands of micro services almost one or two per engineer Right and this is what the world saw I mean it's also though But this is the reality of Microsoft adoption is like it's just a total toolbox I mean I describe micro services as being an architecture of last resort really Yeah, absolutely because it pushes you quite naturally to a more aggressive distribution of your your state and everything else
And so you have to have good reason one of the best reasons to use micro services is if what you actually want is an organization with a High degree of autonomy if what you want is teams who are able to act Independently from each other the micro service architecture is a fantastic be good fit But then really it's a means to an end right so that's often what a lot of people get wrong I think it's their dot micro services and they think it's have a magic doesn't have to know if you want organizational autonomy That's what you're working towards and this is just one of the mechanisms to get you there and most of the enterprise organizations The vast majority ones that work within this space. That's what they're after they've got huge numbers of developers lots of middle management They're not getting much work done. They think it was we do organizational tonnity becomes you know The solution to all problems again, I don't know what to have to say with not a good idea And I think that's where micro services can be especially useful when there are other exceptions as well And so an organization like you know uber Netflix as well things like dimensions I'm gonna talk about Netflix But the teams are already fairly well scoped around there was a price already now And so you have those natural high level boundaries around the business domain So those things flow naturally and so that for me was a logical kind of stepping stone
You also have the difference and this is a lot of people are quite early on when you go and do the work in the dot coms Versus the enterprise organization music dot coms you mean the digital companies Yeah, I'm old enough I still talk about dot coms, but the tech companies I mean fundamentally algorithm You know domain with the main concept of credit be simple the domains are very very simple compared to enterprise organizations by knowledge domains And they're much much more complicated because they're based on business processes that have existed for 10 20 30 50 years So what you mean in a lot of companies a product is pretty much a here's our product here's how it makes money and that's kind of And we have another product uber and that uber like a simple thing is there's rides which it makes money like this and then there's uber eats Which is a separate business over time by the way does get more complex But when you start it's like really really simple and it's really separate and it's clear like where one ends versus where the other begins And then you do some hacks when you need to launch uber Really quickly and if you build it on the right infrastructure and uber is order has like two trips attached to it because anyway But but but as you say like early on it starts very simple But then also in those environments your business processes and the digital implementation as business processes evolving together a lot of enterprise organizations Didn't have that I mean their first generation of digital products were
Codifications of paper-based processes. Oh, so you're talking about these companies that are before digital let's say I'm a lead us or a let's issue company virtually all banks virtual bank And now they're like we have all these processes and now we need to build a software that mimics what we do which of course is That's a start off. I mean that's what the whole dimensions on book came out two thousand It was a turn to design to man two thousand four and it was a reflection of the fact the vast majority of software development was the process of us Trying to model existing non-digital processes and how you do that? I wonder if we will have a different but similar problem of the same thing now that you know We know AI tools are here. We know that for example They're great at generating code and we have two companies we have a company a digital company that has this pre AI They're used to you know writing product we're having document deploying developers product managers designers QA They all have their roles and it works like a really well-oiled machine And now you have these so-called AI native startups who just again they have a few people they're just from the start there using agents for everything They have a very different process and some way one this whole thing will mature and we will have best practices We'll have the existing companies who like the previous one they will try to retrofit You know this tooling to how they're working today. It's already happening
I mean, I mean anything is enterprise companies will really be successful doing those version things because ultimately the corporate DNA Will just crush that stuff within right and it'll take a while But I think the whole industry is getting is grappling with the idea of like are we now moving to a new level of attraction? Right, so it's all about attractions, right? But you know we went from doing machine codes to assembly went from semi-channel purpose programming languages and you're coming back to idea of the program being a mental model We would evolve that mental model at the code point of view. That was a most effective place was us working at Java, you know C-sharp whatever it is rather working at assembly level, right? And it's like okay Well, it's actually what we're talking about really we're just typing that level up We're still actually building a product together. We're still thinking about what the software is
We're still building that mental model maybe now more collaboratively involving you know We'll put up people for example in that conversation We're writing our specs and so the question for me is are we ready to make that step and the answer is no not everywhere This is a really good point because if you think about it like you know like one of the the mantras inside of Facebook He's through that I think something along the line the code is code is truth or something like that But in general in a lot of tech companies if you want to know like how things work like where the truth lives Is it in documents or is it in the code of business logic? It's in the code in the business logic and if it's well maintained code often well comments are structured You can actually see the business logic and the rules for I don't know for example I feel like I've got to channel my friend charity at this point and say the truth is production Everything else is is the lies we tell us a production is truth But I think in terms of like a definition of what the system should be doing There is this ongoing debate about where is that definition is it the code with the spec?
We get a rough is a nice article on Marti and Fatty website recently about spectrum development And she talks about the idea of some companies using being spec first or spec anchored And a lot of it comes down to where they see the source of truth being So a lot of companies are using things like Kiro or spec kit to define a feature and And then the LLM executes that spec and you can iterate on that process Right and so a lot of people start off they use the spec to write the code and then they throw it spec away And they get other companies that keep the spec and when the feature changes they just update spec And so there's sort of seeing a bit the code and a bit the spec is the source of truth in terms of what the system should do And then you start now starting to get companies where they didn't even look at the code And they just look at the spec and so we're kind of going on this spectrum She had some great terms for these and I think for me it's like is that the right path I don't know I think there's a lot of questions about what we would need to do to get that We have a degree of confidence about what happens when I execute line of Java code Beyond the safety of the compiler and the runtime and you know that stuff There's a lot more questions to answer. I think before the whole world is ready to say
The spec is the source of truth of our system is supposed to do Production's always going to be the real truth so just to be clear But I think it's a good question to just ask yourself what however you're building and how we're using agents of where is the truth of your system Right now is more in the code. Do you have a spec? Is it scattered between the minds of people Which goes back to the collaboration because again a lot of the reality of a lot of startups are is a few people hold these things in their head Right, which is okay in the beginning But of course as you scale one of the reasons we used to love code review Which is not going out of fashion But we used to share information and when someone left the company or got ill or you know the famous bus factor It was okay because other people understood because they reviewed they were taking part in that part of it But now we're with the AI code review is some some companies are like well We're not gonna do it because of it's a coach that was a super power pairing Right pairing you if you do you do promiscuous pairing you rotate every day when someone leaves everyone's got the same shared context That was something people missed around the benefits of pairing I think there's a lot to be said around this I was trying to Chris forties of head of software head of technology for ThoughtWorks in in Spain And he said the biggest problem we've got around our around AI driven software delivery is tacit knowledge
There's a lot of knowledge in our heads that we've built up that we don't necessarily know that we have We use like a code review you've already had looks down It doesn't really feel like I wouldn't do it like that and sometimes it's subjectivity But sometimes there's something about that you can't quite pinpoint maybe you can't unpack it Maybe you can't but it's also like you have a bunch of the things like that in your head about how software delivery should be done about how the business works Everything else if we want to step up that level of abstraction to work at a spare level We actually need as human beings to be able to unpack that tacit knowledge that we have and make sure it's in the spec If we really want to be be hands off I also think there's a fundamental problem we've got in the current way a lot of people do stuff with livers You've got a misconstant cognitive debt stuff Which is you've got a lot of people running off in different directions I've seen a few people talking about pairing with their AI system not a human Stop pretending it is and so you're all going off you're doing more in a different direction that shared mental model of the program is breaking down And there's gaps in between AI was supposed to free us from drudgery Right, it isn't the most of the developers could be doing more work We've got more context which we lose in that big picture
So I think if we can work collectively on the specs we can rebuild that model then that absolutely gives us a source of truth I still think it's gonna be parts our brains and part of the spec right but for that's a work with there is does have to be the collaboration piece Whether or not that's our dev teams go from being 15 people all into that five people or six people who don't know But collaboration is really important. Yeah, I'm going to a proper company right they do proper stuff They're ex infirm they recently announced they're getting rid of any human based code reviews And they said we can spend more time working on things like the fancy working design dot level and actually I'm okay with that Because firstly I trust that they know what they're doing but also there's something about I mean a lot of companies make a mistake of not doing the thinking where it's trying to go saying We're gonna spend that time doing more important thinking Right, and I think that's sort of how we should view this I'm okay You know not looking at the code as long as you're doing other things to make sure the system is operating correctly And you're thinking critically about how the system needs to change and evolve If you're just not looking at the code and you're not caring about those other things Well, then you deserve everything that's gonna happen to you Which also goes back a little bit what Cherry was talking about on how you want that systems at the valley that works
About having empathy for example for the word that qas and and sre folks have had to own code of production that they Did not write they don't necessarily read or even understand but they know how to operate You know what good looks like in production They know what what it looks like when it goes poorly I mean that's also a bigger thing that now the more developers are having to struggle with because if you want to do spectrum development You have no good looks like I did a I run these likes of half-day conferences for a rally online I'm recently on software delivery and a load of speakers and about sick about that six speakers and four of them were talking specifically around different different varieties of A spectrum development effectively and all of them came back says you can't do this unless you can define what good looks like And if you can't don't bother and you know some of those people were much more bullish like Addie I'm somebody's very good at all this stuff But one of the presenters who was doing they took they got like a small software fraction of three people But like they're actively pulling in their observability data their hotel stuff right into Into their agent to understand how it's changing. I think that's exposing I think more developers into understanding It comes back to the exceptions criteria stuff people don't stories
They're always very narrow and functional requirement stuff So now it's like I think we have to be more grown up about those things the discipline is absolutely important and the expertise to make it It's just needed at different place Sam made a really good point that you cannot hand your work over to agents unless it does these two things one define what good looks like Two prove that your system does what this good a definition looks like You already know who I need to mention what comes to proving that your system works as expected and tisis our season sponsor Agents are writing more of the code So how can we trust that our system works correctly without bugs code review is increasingly naive and ineffective approach to do so Which is why this is goes well beyond this practice and this is runs your whole system in a hostile simulation This simulation includes both targeted testing and fast testing by running the simulation It finds every bug before your users do and because the simulation is fully deterministic It doesn't only find bugs it gives you a perfect reproduction of every issue which makes it a lot easier to fix issues The first thing I thought when I heard about it is this and that is automated bug discovery and fully determined testing that it sounds like science fiction But actually it's hardcore engineering when you look under the hood Jane Street fly that I owe and the et cd community shift agent version code with full confidence because they know it's been verified by intersis
See more case studies and details head to ntisisis.com slash pragmatic I'd also like to mention representing sponsor of trouble buffer We know what's a database that's ridiculously fast and extremely cheap And they are doing something amazing They are completely redesigning their storage architecture from first principles to make search faster cheaper and more reliable at scale If you follow turbo buffer, you know that their storage architecture was a massive part of their early success redesigning a winning architecture is a big deal It's one thing for query plans to pass unit test which is already happening It's another to bring each query plan to performance parity or better while maintaining correctness and reliability and production And it's email sharing the mester reality of the rewrite and how the v3 version is currently a lot slower than the current production one For example, we can see how cold full-text search is 11 times slower than prod and hot full-text search is 126 times slower But we can already see some wins like hot and cold vector search is that parity or better I really appreciate this working progress for you and the real-time logs and what it takes to do a successful storage architecture rewrite The turbo buffer team is keeping on the work log of their journey and you can follow along at turbo buffer.com slash v3 I definitely will do and with this let's get back to Sam and old qa and s3 folks already know about running code that they didn't write
So I want to talk to you about your latest book that is just coming out building resilience distributed systems Yeah, what is the book about and why did you feel this is a good time to write it years and years and years ago I went to the loss of tik which was quite a well-known conference conference is still on No, no, it was the right conference and it was like the conference where a lot of the early.com people went to talk about how they made this It's a scale of other stuff. I got to go. It was being like one of those career ambitions to go to go to go Because after a bit of the first book and I got changed to general sport there who I still chat to him now then he now works Since engineering space pointed me I don't know if it was him or one of some else but the point of it was paper by David Wood So it was all about resilience engineering and the papers utterly baffling to me absolutely no sensitive But it was all because it's not yeah, because it's just it's not written for computer scientists to read and I'm like computer scientists So I was really thrilled But it was really fascinating seeing this whole other world about how we make how people viewed resiliency and then what we're doing software And people like John which I take some of those ideas a little bit by very difficult to take some of those ideas from Sort of safety critical systems and that's how to thinking and make in any way shape or form palatable for normal people like me
And so I kind of this is in back what 2016 maybe 2015 And so I go back and try and read that paper every six months or so and I read a bit surrounding the agency other smarter people that mean that Summarize it data better and then I was thinking okay These ideas now starting to make sense and then also I kind of realized that despite my best efforts Lots of people seem to be building microservice architectures even though if you read my books most of my book is about telling you not to do it You have to read the book which is microservice I'm going to buy the most important thing whether they read it or not But and so then I realized that there are more people exposed to the the horror pain and suffering of a dis And so the goal of the book is to start you off with really easy foundational Simple kind of engineering concepts about how you can start making dealing with the problems that do your system But also take you on a journey into kind of the whole concept of as engineering
And so it starts off with really easy ideas and you study it's more and more complicated and you know So I introduced kind of concepts like what social system is the difference between safety management and resilience engineering is all like big concepts But the sort of second chapter after introduction is like what's the ability to talk about hotel the second or third chapter is off the time Right and these are all I think quite tractable things talk about building blocks right So they're going to permit the permits for yes Just another capital there as well. Yeah, absolutely And to give you know to go through and sort of distill some that knowledge down and also to be a bit I'm quite happy being a bit opinionated on these things But I'm also lucky now because partly because my only work I get to chat to more people So I can actually talk to people who this is their full-time job and trying to steal some of that knowledge a bit And so I wanted that book partly is about being evergreen But it was also about saying there are lots of people building into this and it's put what type of systems you've got off On side for a moment There are lots of people building them whether they should or not and they're really important to us now So how do I how do I help people start and grow and that's that's what I wanted to do I mean, I enjoy the process of writing anyway, but this is a book I wanted to be able to give to people
That's always been why I write that book Every now and I've gone to that answer me to the right book and I found something else I've done I don't know when it's like all that is now so this for me was a book I wanted to exist But that's kind of why I write it. Yeah, and I think it's a book I certainly could have used like I got dropped into a place like Uber where we did have what I realized where distributed systems You know, it's just our systems handling a bunch of stuff lots of services Depending on one another because even a simple thing of like knowing like oh you should pay some flexibility that I mean you figure it out After a while yeah, but it takes a while. Yeah, I mean again, I mean, maybe this is my attempt to unpack some of that tacit I've made so many mistakes and I keep making mistakes I think we're making mistakes as useful But I think also crazy after people can learn other people's mistakes versus a good idea too And so in the book I also try and put in real world examples and sometimes big companies sometimes my mistakes to kind of make this stuff See real and I think the other thing I think I really try and get across is that things go wrong Right, I mean engineering is all about making sure as many things go right as possible Not about making sure everything goes right. No, and there's also you can build it in systems in general
is also by being honest about mistakes that have been made and Talking about them and sharing them because that's how everyone gets better I don't do that you can't build it was an assistant in your company And so this again is my attempt to sort of highlight all the awesome works on before the fantastic posting some reports of people that some companies share Like when there's a system that we all use or many of us use has an outage I mean, it's it's annoying Especially when you're using it But I always greatly appreciate when the companies because they don't have to do this But when they share a public postmortem or incidents of recap or whatever that is they go through what happened for example GitHub had an outage and they share a really interesting one on what all the things that went wrong and in one order And then you also companies where some of this happens and they don't do anything So of course you're not obliged to and anyway, but I always have a bit of a They are obliged to now a lot of them regulations are now making them that have which I think it's great Yeah, I think I door at stuff in the UK and you're a lot of I think there's also we should also separate around that posting some report stuff That there's marketing and PR versus having done a proper post-ins review Now I do think you've got you T of K We said you just know it's part of proper posting to review but those companies may be saying that they might do fantastic process internally
There's also the fact that some companies that put those things out you read them you're like this does feel more marketing Yeah, I think a great if you want a good company that puts out some of the best self-flagellating Post-insiderate review write-ups go for cloudless stuff I was about to say go for themselves in those it's like I thought guys Go easy on yourselves right for make some mistakes, but I quite like and do it rapidly was a 24 hours and they're publicly traded company Yeah, which I think just goes back to the meant like some other like Republican traded We don't want to do it because it would give away information for shareholders Yeah, no like clover does it end and they're fine. Exactly and also it's and the trust it is I mean a lot of how an organization is perceived as down to how you deal with mistakes when they occur, right?
I mean nothing ever goes perfectly for every company So in your book you talk about the three rules of distributed systems What are these three rules? So I really tried to steal this down because I could never remember the eight the rule of eight that came out of some The fallacy is different to always forget what it was. There's loads of complicated stuff But I mean when you strip it all down It's really only three things that really cause all the problems Right the first is that you can't send information instantaneously from point eight to point B It's physics. I mean now it turns out when you get into it starts talking to people about quantum physics I say aha and they go there's a thing called quantum entanglement Which does allow for the theoretical instantaneous read of electron spin states across arbitrary distances Which point I point out that a that only works for reads as far as I'm aware and be not a networking protocol So like but it takes time right so I only have a to be and like that I can't control all that I control bits of that as a developer I can control how much data I send I can control the payload size technology I used to see like DCLs I'm unlikely to be specifying the network cabling and the equipment we use or distance between them So I'm not re-under my control and the moment you go to public internet
BGP has a bad day and your packet has gone down from trous and they get the internet right? Yeah, so that's the rule one it takes time And that's important to recognize because sometimes it looks instantaneous it's not second thing Sometimes I want the thing you want to talk to isn't there sometimes that's under and we do things make it under our control, right? Like we would run say multiple copies of service incidents behind a load balancer What happens in load balancer fails? But I mean like you can't control everything I've had services taken out by a fire the data center a sand caught fire and specifically one bit of the sand the sand had a cashing Storage layer with SSDs these two pairs of SSDs one of the pairs of SSDs caught fire the other pair Yes, it is working fine And so we had these weird latency graphs coming out the sand like perfectly fine if you're loading one through the ones that weren't on fire But the failovers in the other one were rooting to be fair it was rooting around the bits on fire I has it where we had ducting between office buildings and rabbits moved into the darks and ate the renequering cable I can't control for rabbits right and sure these things are like they're the real it's the true things right?
Yes, you can control that and you can reduce the chances you can't do chances. Sorry So sometimes sometimes you think you want to talk to isn't there and the third rule I'm very grateful to my Call for telling the same I've missing is that resource pools are not infinite meaning you can run out computing resources CPU memory I owe whatever Your little balancer could be overloaded and then just not respond even though it's there and we know the latency but it maybe it's you fills up I know the vast majority of the time when you have a system outage you down to that Some resource has been totally saturated somewhere and it might have been that saturation might in turn have been caused by the fact that something is down And you're in a retry storm which is in turn causing, you know, connections starvation which is in cause of resource saturation which is And yeah, yeah, yeah, but that that's the killer and and again I think I'm saying it you know sort of intellectually all of these things But you can go to all the concept complicated concepts in different sense, okay Why is bison time consensus protocol thing? Well, let's just go over down to this bit What about cap and pack hell? Can I just let you just put us up on side remember these three balls first and then most problems end up becoming a little bit
A little bit easier to deal with and then the book after you introduce these rules You start as mentioned second chapter is observability to me that kind of suggests that this is really really important part for this Data systems why is it so important? I mean the way to think about observability is a property of a system, right? So the more observable something is the easier is to understand how that thing is behaving based on X and signals When you want to probe a system which is a single program You actually get some pretty high quality signals coming out of that program therefore give you a high degree of observability I can get a lot of information out of that. I could attach a new bucket to a process I could generate a chord dumb and run that in I could just thread dumb from JVM and work out with my semaphore's a block If this is my whole system like knowing the CPU stock at 100% is actually really useful information because that's probably a bad That's the problem, right?
So this basic information we pretty much get for free with a single program gives us a high degree of a single program running on your machine That's exactly right, but now we go to a distribution system You don't have access to the computer the same way you've got multiple computers And so those basic signals are harder to get you can't correlate them unless you actually work and those individual signals by themselves Don't really give you a proper sense of how the system is behaving So you have to do more work to make the system observable and as annoying with these sorts of things much easier to do that for the beginning It's a retrofit it and then you know lucky We've got things like hotel now open the open so much protocol And specification because that at least gives us an open framework there because a vendor agnostic Oh, I'm on the best charity pacing stickers. Just be really clear. I always think I work independently I have to be very clear charity pacing stickers, but like we get a vendor agnostic open standard for now So it makes it so much easier for me to have a conversation about this But if I don't get the signals coming out of the system If I don't get the stream structure events telling me what the hell's going on with my system I'm flying blind like how do I know appropriate time out is for my client?
We'll be talking to a service I might even know that if I have a sense of what is its normal response time distribution Some of any latencies know the success or failure those latency it's probably about a problem in history I may be taking a look at it If I don't have that information coming out I'm not gonna have a set by time out correctly And then we're talking about a low level thing like latency and things that we can measure at the machine level But of course is it was really when I'm talking about like doesn't work for the user or the customer does it do what they do when they click the button to submit? Does it submit? Do you look at that because we know observability is usually we talk about these kind of latencies that the status is 500 responses that's gonna bad That's means it's down But there's always a player that I always wondered and I found it hard even when we're working in teams too Okay, we have the easy observability like this is kind of text not text book but it's a metric It's a metric and it looks but then there's a stuff of does it work does it do what people care about like again in the case of like an Uber like when people hit a button to request to ride like does it actually request within But then usually five seconds or something like that or is it stuck in an infinite spinner?
Which is bad even your loss of yes healthiest can be but people are upset right You know you you treat the raw data of making some observables as a stream instruction events And from that you build up the higher level of fractions and a few to write Is it a primitive? So a trace right where we get sent solve for a given click of a button what happened in the production system And this is a big blind spot like most organizations had to build on tracing systems This is where things like open tracing stuff came from things that zipkin things like that were built an attempt to recreate Like most people just trying to recreate what Google already built there and this is an attempt to say okay Can we at least look at a flow of calls what happens during that flow of calls? Where did these things go? Well that requires me to get a bunch of spans of all these different machines and a spam meaning a spam meaning an activity based on something I did And then a span can be get more spans so I can say my first band might be a call is hit my server Right, so I've now got a span that starts within that span I might talk to my database and then do collaborations and I can publish an information on those spans and I can work on the service And then you get a trace ID that shared across those spans and you've got timings and now you get effectively a view of what's happened across those calls
But that's all just based on individual events that we instilled together And then you also start getting in the system which is situation which is my system behaving correctly And you know this is going away from again I think half of them just saying what you're actually saying You know again going away from the vanity dashboards is moving into sort of the world we call the SLOs Right, so we have objectives and SLIs and SLIs so an SLI so it indicates up That's our way of saying at this given point in time for this thing Is this thing good or this thing bad and we use a collection of those so one example of that might be a successful percentage of one's time right? So that would be it's a good one or a bad one. It could be a bad one because it's an error It could be a bad one if it's slow good or bad and I collect those together and that helped me know if I'm meeting my SLO So I might have an O-Ratching SLO which is I should be able to request a ride with 99 percent I'll successful ride request within 300 minutes And that would be your statement and that would be a commitment. Yes. I suppose we commitment that's our team is working towards You then get this raw data you can't create your SLIs on the individual things and then you use that to work out what you're looking like If it's green everything good time for thread to speak on happy
You get a bit of it. There's way more nuance to it that I'm doing a very bad introduction to this other thing But without those building blocks because I talk about the first chapter of the book Nobody says I don't want my system to be received. No, no one but at the same time Resiliency is not a binding property. It's not like off or on it is a bit like observability It's a degree so then the question is how much do you need because am I having a conversation with you about multi-cloud and you know doing complete rewrites of systems for critical Standards like Monzo do for example. Yeah, they have a standing bank So it's sitting there is costing the money for the most part But then just a few weeks ago their main application went down for a short period of time And they actually that bank which I'm assuming it caused a millions of pounds or dollars per year just to keep it there It's not even that is beside barber. So firstly there's their standby bank has a real ship down is to features
It's much more bare bones. It's all the core ones like you freeze cards. You can take money out and check bounds It's a completely right so it's no share code because I wait about bug different cloud vendor It's also by design significantly more cheap to run so they've got thousands of episodes in the full stack This is about a tenth of the cost of the cloud cost. So they also run it constantly. Yeah, so it's always there I'm but like Monzo know them on so that works But like it's no point you starting there so actually we thought about the observability stuff in it So you need to roll data, but also you need to make decisions about comes back to what good looks like Then you need to know, okay Do I need to worry about multi cloud vendor the our strategies because that's where I need to be going or am I somewhere else?
And so I think without that kind of sense of your North Star of how is it that we need to be without that defined You don't know how deep you need to get this is what I like to say how what how secure is my system to be having a threat model? You know threat model I don't know and I guess answers question of how resilient do I need to be it greatly helps and maybe it's actually a requirement to do that Foreign decision is how much will it cost me in terms of operating costs in terms of time to build it in terms of for example latency Just a simple example is if you say like, okay, you know the business like oh, would you like multi cloud and this is like? Yeah, we love multi cloud like Amazon goes down because we heard they had an outage last week or a few months ago I actually want to be able to run on Google cloud, but then you'll be like, okay Well, first of all this will mean we're paying two cloud bills and if we want full replication It might increase our latency because we need to wait for to write for both of them Which might mean slower your customers might see things so anyway There's all of these trade-offs and in my experience like oftentimes like you might retrofit that but if you can have a discussion upfront That's great, but to have this discussion you're going like you need to know what the trade-offs are you need to know what's happening in all these dimensions that You know you care about like how long will it take to build how complex will it be how much money will it cost to operate?
But also as technologists we can't just say oh, we have also got to get good at communicating with our business stakeholders without reducing What percent are latency to you like some developers know what I say when I say that all but some How many you know experts in their field who are great product owners or subject matter experts but are not software developers would know what I meant by that And so we also have to understand the business domain the product domain well enough to do the interpretation and help and to bridge that gap And it is a dynamic thing as well I mean, this is the whole thing about resiliency in general Like a resilient system is one that changes it changes and it evolves if you don't change it evolves your business It's not a resilient system And so even those targets we said that conversation you have to have about what is good enough does change But like I think that's like in the case of say the us east one outage towards it last year Um, there were lots of money. Yes, I said you AWS east one region went down is the older region Big region lots of people there first because that's often where new surface got more historically when that went down It caused a lot of people to say do we need to be multi-region and some of these companies would have had the conversation before and as I know we don't
And then it's even the fallout and they may have changed their minds and that's fine Because this is what you mean by resilient system evolves. Yeah, you know, that's the real world But also like it is about what problems you're not trying to solve there was some big name companies that got taken out when us east one went down And I got severely has a fear degradation something that could offer yeah companies like zoom for example It won't take up in time but there was a really affected do we think that people working at z- medius? No, do we think they knew that if the cloud region went down they'd be impacted? Yes, they could be looked at that and said well for us to come out with a multi-region approach It's gonna have this impact. It's gonna cost us this much money. It's not worth it
Let alone talking multi-cloud because multi-cloud versus multi-region is a whole different conversation And that's okay I think as long as you've had the conversation and as long as this is the other secret bit of course It's as long as what you think you need to do is in tune with what your users thinking to doing Yeah, so I make the point in the book when talking about the difference with SLAs and SLAs So the SLAs what you're gonna get sued for if you breach if you only have a deliver to the SLA your customers are ready ever happy You've got to do more than that right? So it's the social contract there as well. Yeah, now with building distributed system building resilience systems It really helps to understand concepts that are not as straightforward that I'm not gonna go through all the things you explain a lot of things in the book But I do want to touch on concept that I found bit tricky to understand and then appreciate and you really appreciate this is Which is item potency. Yeah, what is item potency?
So there's a mathematical concept behind item potency that I don't understand But basically an item Poen operation is one that you could you know, you can apply that operation multiple times without it actually having effect multiple times So it has an effect once first time a really good example of an item Poen operation would be pressing a lift button So you want to go down in the lift you press the down button it lights up But press that down button again. Nothing's changed. Don't it's just coming what we hope it just a coming right? Yeah, that's an item Poen operation some lights which is actually toggles I told it like goes on target. I goes off. That's not an item Poen operation in computing
We have some operations which don't have to write in Poen reads are a delete is typically I'd Impotent right but the example I think I use in an extruders concept is a one of payment I was about to say that I have a lot better item port and example for you. Yeah. Yeah, payment is a good one So I'm gonna pay you money. So I'm a service the ones can issue a payment and I send you please pay Sam 100 pounds Yeah, this is I'm an LP in people back at home Please say pay Sam 100 pounds and I don't get a confirmation back I'm gonna get back a response. I promise from my point of view that two possibilities The first possibility is that you receive that payment rest and you've done the payment It's been completed and the result has got lost to me for some reason I didn't get this box That's one possibility the other possibility is that you never got that you never carried out the operation Maybe didn't get to you. Maybe you crash I think like a good example that I always
I've seen it in the past and maybe it's over but airlines you're about to buy this airline ticket for $1000 $2000 You've checked all of your details and it says like play or reserve and then often some airlines would take you to an intermediate page Which says like please do not refresh this page or submit it again because you might be double charge And what you really want is you pay you know like pay and if you you get the spinner and It's not idempotent if you do it again and you get charged twice Yeah, but it is idempotent if you refresh you you copy the same thing you submit the exact same form Although that's interesting. So the question then becomes right So I think that payment but my end ticket whatever it is and I haven't had that response back But I really want to make sure that the end I'm taking it going through because you were the browser or the people are home Sending Sam 100 pounds. You really want to make sure it's 100 pounds, right? Now when things would work In the tissue system because of rule number two rule number two meaning the thing I might not be there rule number one sometimes I can't just yeah, um and accepting this thing it's not used try to Can you retry because you really want it to happen?
Yeah, and retrying there's the danger that if the first operation had been processed successfully You didn't know about it Well the first operation of the applies actually and you do it again and I can pay twice in two tickets So that's the problem with an idempotency You want to make it safe to retry important operations So to do that means that my own point There's two ways to do it really one that's good and that is one you should do but you kind of have to retro switch your feelings pain and one Which is one you to explain which is useful but has problems right? So first one is idempotency keys. Yep, so you generate your unique ID. You don't need to do your ID for this And so when you do the unique ID which you generate and when you generate two of them They should be different. There's like this tiny tiny tiny tiny tiny tiny likelihood of clash which again what happened?
I think with you ID seven. I think it's like a 59 seconds chance But it's something you can generate locally without an additional ground trip. Yes, you're clicking chances extremely low There's ways to modify you can work with that And this is your Amazon probably well this but this is I mean the AWS API's you see strike adio PayPal they all use the same key so you send it to you ID So then if you send that to me say hey sam and here's the idempotency key I don't send you back a response for whatever reason You retry it you pass on a pony key if I had carried out the processing I could look at it So I have a process this is I have okay great. We have processed it even I get second or third or fourth Have you processed the process process? I think I'm a process process is fine That's a nice simple thing to do the issue with making things idempotency keys It requires both client and service support tip so it has to be a required field To go through and it has to set the support so if you're trying to retrofit this in it's a pain But if you're building it use an idempotency key use it exactly for anything easy idempotency that's the beginning The one you sort of mentioned was I go back. I hit refresh
I feel all the same feels again And I submit the L I'm booking again each of those a duplicate That's another one you can do it disforces good if you're in a retrofit idempotency in So if I send the same fields so the same person Sam the same amount under pounds to my manager and I'm sending this through again what you can do on the service side is Generate with a fingerprint so I take these fields Yeah, I then generate a hash that represents those fields So if you do a second booking it looks the same and you're using that as an important as a retrofit idempotency key Yeah, there's a couple downsides with that you do get false negatives with that I was like a little meaning that it gets we did might get legitimate second requests and get rejected We've got it So a real odd example this happens in and notice happens in Denmark it might happen elsewhere speaking So you can say Denmark you work on a payment system You go in you buy something for two crone with a particular payment system and you give the car The car for to crane whatever if within the next or five minutes you buy something else for two crones and say shot The till the hands every exit if it was forty one crone it'd be fine It's such an uncroneed be fine But that's because the chances of you buying something for the exact same money Yeah, it's so remote So doing some basic idempotency to top double taps
Yeah, and so when you get situations where operations kind of legitimately look the same but actually be different and carry out at the same time So a really good example would be the AWS APIs It is completely legitimate for me to in a for loop to provision say 20 different easy to into instances Now there's better ways of doing it. I'm thinking of falling back good So I can iterate with the fall and spin up 20 identical easy to instances If I just rely on their big fingerprint at all of the same So that's where this operates now And so if you've got situations where legitimately you have operations that look the same but they're actually different Close to each other you have an issue if you've got a bit of space around them What you can do is hold on to those fingerprints for a period of time five minutes ten minutes an hour So there's still some potentials you could stick to this is not ideal Idempotency keys the way to go fingerprints when you have to register fit if you use them both together That's can be really useful So you can pick up situations where people change the parameters between the retry attempts Which leads to confusion a bit more complex topic But a lot of public facing APIs will still generate a fingerprint And so if you send carry out this operation and you give the same idempotency you gave before but you've changed parameters
The server can reject it as being effectively and valid request so that's all those things What's here? There are actually ideas are really simple I think but it's it's And you want to about implementing them there's some interesting stuff that crops up in terms of how you store them on the server side and things But the fundamental idea is a pretty straightforward thing another interesting topic that comes up oftentimes These days it feels more often is thundering guards Yes, it's it's when a system restarts and suddenly it faces a big load Specifically get up at an outage but now a few weeks ago where it was networking related and recovery took a lot longer time because There were retries going on and it created the thundering what is a thundering heard? I talk about it in any situation we've got mismatch capacities You've got too much too many requests or too few resources right resource calls not infinite coming back to one of the three Yeah, so in the chat so I talk about a whole bunch of different ones from different types of scenarios, right?
So you've got everything from the malicious thundering heard like a denial of service attack, right? It's usually denial of service attack malicious attack often short notice and there's not that much you can do about them The answer is you're used fast your cloud flare but I go into details about how that happens You get things like a kind of internal self-inflicted ones So a good example would be something like A cache collapse, right? A system relies on the cache not only for speed of operation but also for juicing load on other components Yeah, the cache gets restarted You then have a lot of cache misses you're hitting the origin and the origin goes down. Yeah, origin was not expecting such a load Yeah, so a common thing there would be you first you flush those cache snapshots the disk reduces the chance of you being out having high cache misses over restart But also a lot of companies I know will just turn their servers off into the cache refills And so that's kind of another example of one retry storms another one back in 2017 square had a big outage They'd restarted multi-pass which was their multi-factral authentication system multi-pass in turn port is data from reddice Those database because you can do that apparently I don't think so Are they using as a cache basically right?
Well, no, I think they're using as a database There are some people that think this is a different podcast I think it's a stupid technology podcast we talk about using technical data So they were putting this data back from reddice and they had a problem that the clients in multi-pass were going Basically if the reddice calls failed they try again But without any delay between the retries and they're hard-coded a 500 retries So they were basically smashed reddice and they kept restarting reddice and then restart multi-pass And it's getting this a long time ago and they fixed it right they fixed it by having a more sensible retry policy and externalizing as values That's not a thundering herd The last kind of sections I talked about were ones caused by like products being too successful Like twitter or in healthcare.gov Right, this is where you have an external real legitimate load that takes you out And I think if you look at all of these as solutions although what's happened is the same The kind of solutions are a bit different So I mean I've got a thundering herd that's coming to me externally that's legitimate Well, I've got different things I'll look at like I will have more I'll have a more constrained signup I'll have your invite codes So in healthcare.gov one of the issues there was many others but was that they went live for like 36 states at once
And they were massively overloaded So in hindsight, maybe I'll have one state would have been a good idea first For example with twitter they were going down all the time So they could have constrained signups or put tweet limits on you and things like that So I'm constrained those are things and they did those things as well And so although they look the same the mitigations The two was a little bit different So I kind of quite like talking about that because they were all great stories I found And of course I think we're talking about like solving it with When you have same resources but then we get into things like scaling like can you scale horizontally add more resources Because there's two sides of this right like whenever you have and so what point can you do that are you going to face a bottleneck Because you sell the central whatever But like that's a great example of where some times mitigations work and sometimes they don't So an example of something like your products are successful if you start up or whatever And you suddenly got successful that success comes like a wave right the wave of success hits you And in a lot of receipts you'll get a thousand customers in and maybe you'll keep a hundred maybe right That's just how things work So you need to be up during the peak But then you probably can shrink down again to keep a cost of like so if you build your application to have some dynamic scale a bit of it
And if you're on the public cloud or similar place where you can spin up spin down That's a viable mechanism right okay We can scale up to handle this and that's fine Would I use that as defense against a denied distribution on a service attack? No, because then your disputed on a service attack becomes a distributed denial of cash attack, right? You go from to not only having no service but also now no money because you keep spinning up You know knows on your puban etis cluster and Amazon just keep taking the money It's great. Yeah, and you still can't keep up So that's a perfectly valid mechanism for some things But you kind of have to know the context that we should say you can't just scale yourself out of trouble But I think we're going back to a lot of times as like as an engineer You want to know about the business you want to know what creates a value what creates revenue What is important for the company versus what is not and then where we want to maybe invest to Yeah with our infrastructure with our reliability versus where it doesn't matter and it's not worth doing To the trade-offs and I think it's about you don't know unless you get information out of your production system You don't know how close your eyes to the red line So that's why they've served you to skip it is all about trade-offs and it and that's why I do a whole chat on like formal methods
I talk about cap and pack out and I say look these things are complicated but actually really all comes down to a conversation without Understanding business context. So example I give there is like if an e-commerce company If I want to buy an item online right if it's gonna like online and if at that moment on a purchase if you're not sure if you've got it or not Most e-commerce companies will say yeah sure you have it because they know there's a good chance They do actually have it in stock and if they don't have it in stock they can get it in back order And the worst case scenario they can email say sorry we funded money right so that's not great It's not ideal now if they said we don't know if it's not going to sell it to you They're missing out on sale and I could buy that somewhere else probably and they might have had it So e-commerce companies in that situation because that's that sort of understanding the relationship the customer the impacted they get it wrong They're gonna try to make the item keep that problem exactly the same but now change the business context now with some of the tickets So you're getting tickets to go see take take right you know if you tell us it And we're not sure if we've got a ticket on it because our link to that system is down But let's take it so you're taking your kids taking your wife is all going to be awesome We can't support you all we can take it so it's gonna be amazing and then we make sorry we don't have tickets
Now the problem that is firstly the devastation is a bit different they can't back order a ticket Yep, and so in that situation it's like the impact the car is so much greater than the map of the flight And you might but the flight so what they do is to make a different decision so in that situation if they don't know if they got the ticket They don't tell it again. I'm not sure we cover in the book But this is the fail open versus fail close right when when you have an error or you're unsure Are you going to go through or not and as you say this can be the decision the same problem two companies can take a decision a company Years down the road could take a decision uber at this classic example of when payment We weren't sure if it went through do we allow the right to continue or not and early in the company's years We always fail open. Yeah, just have the right will we might have the money and if not we'll deal with it As the company starts to go for profit. We feel closed like we're not sure we have the money if the charge went through You're not gonna get the right and I guess in company different different by their different focus And if you're a developer you needed to pay attention to like what is important right on the business it was growth initially It's okay if there's money and then now it's I don't you know the conics profitability But also that open closed those open close those open open discussion isn't one size itself the whole system exactly
It's a feature by feature basis and even the con even then it was a system You could say this taxi rides 10 euros, but it's a thousand euros you can make different decisions Now if you're approached from a mathematical and from a computer science viewpoint, we just talked about cap theory That was a conversation about cap theory right so in the chat I talk about his cap his pack out Which extension of cap I explain what those are because when you get deep into two things you'll hear about them But fundamentally that's what it's about right when you have a partition That's what he and cap really defines. I can't talk to one part of my system Remember it's rule number two, but I can't talk to the part of my system What do I do and all you need to know is a developer I can't talk to that what do I do? Sometimes we can do make technical decision. I'll try again I hope I don't bring my system down for the retry storm, but I'll try again if that doesn't work What do I do? Well actually often the answer is yeah, should we look at cap and pack out where else but go how much conversation with the product Don't know what we do the resolution of that often ends up being a decision about are we trading off consistency versus variability in pack out Harvest and yield if you're getting into that solution of production, but don't worry about that That's in the book. That's why I said later. Chaps are like 12. I think well, Evan don't don't start there start with having the conversation rule number two
I can't talk to this part of the system right now. What should we do? And that starts with having a chat that starts having a chat when we're talking about resilience you talk about the dimensions to think about resilience What are these dimensions? I start in the first chapter So like coming back that paper by David Wood's I mentioned that was impenetrable to me at start and he comes up with Four concepts for resident engineering and I like the four concepts he comes up except as things broadly right? So it starts up with robustness. This is where we can basically I think in his words absorb known perturbation
So we model a system and something goes wrong and we can kind of just deal with it in a way that none cares about So a very simple example would be Incubinities are pod fails and and could get some replacement now the pod failure that's the perturbation is going to be wrong But we dealt with it great. So that's the start and the thing is most engineers I speak to they focus in time on this Don't look the other three folks and time is well I mean nasty things about robustness is the things we often do to make our systems more robust in turn increases the complexity of our system Which in turn creates increases the surface of new types of failure Mm-hmm I bring Kubernetes in to deal with failure of a container now got Kubernetes and that becomes Yeah, and that's not so why the way that's going to be where off I'm so why sometimes people don't go multi region or multi cloud because it's increased instead of other failure types Yeah, so that's not why it's robustness, right?
Can't we do anything and for things be robust? We have to know about me ahead of time. This is predictable We do it then there's rebound which is when it does happen it goes wrong. How quickly can we cover? So this is the this is typically the time to recovery while it's time to detection and time to recovery He got yeah, I'm measuring that and I talk about the dr chat So I talk about you know the other Exactly, yeah, but that's like how quickly we get back up and running so we've got into a degree to functionality How quickly up on running so that's thinking about rebound and again for that you have to be very humblest about because of Robostages there's just a little never fail. It's like but what if we quite wrong, okay?
We have to admit things go outside and then I guess we're recovery on how you can recover This is where I mean if you want to do it well teams that are serious about this they do Simulations of this isn't going down This is where Netflix is very famous chaos monkey comes and wears goals and shuts off parts of the system to like test this slide And there's drills that are being in done how we get how our on-call process works How are decision-making etc etc this is actually part of all and then you either once that's not enough because you need to do regular System systems changing I like to shout out to uptime labs who are basically partly another they do excellent instant training But you can do online or browse fantastic People so you've got that recovery process How do you get up and running and I think kind of techies about with me and then we go into next to which the third one is graceful Extensibility and so graceful extensibility again It's a slightly weird name is basically dealing with surprise when we make us it's a robust we're dealing with the known No problems. Yeah, what about those? What about things? We don't expect?
I don't expect it. This is now not about computers and about humans is why Dishesives of fundamental social tendencies people tend to work in together. We couldn't expect rabbits How do we deal with rabbits? How do we deal with the following dates? It turns out badly is the answer But if you're in a corporate environment, which is very command and control quite hierarchical We've got a very narrow job description you're often constrained in what you're able to do those organizations tend to deal with surprise badly Because the corporate structure is structured around the known and so often structurally you'll find that difficult you mentioned drills earlier It's also a great way of helping a surprise coming in and having people often from outside saying this has happened how you deal with it Google had a thing and I feel like I'll do it sre support round they do a thing with the wheel of misfortune that If it's a wheel today, you'll see some of the speed in the wheel Do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do do Do you know something attack by a state in a name in certain name with rogues date here?
Here's what's happening. How do you deal with it? Um, that is all about like people and experiences and trying new things and constantly pushing yourselves because learning also from others I remember this like very famous outage happened a few years ago at meta where their DNS system through BGP it just all went down What's up meta everything to fix this because they pushed out an update to their BGP table, which is in its IDNS to their servers now to fix this they needed to get access to the computer that physically to get access for whatever we Because they couldn't reach it remotely so they need to go into that building now in going to the building They have to go to the door that protected this System which was running on their own domain, which was down so they had to break physically into but again This is the thing where they share the details which was great because now everyone else could think about like do we have any other physical security For example types or own domain because if you have it you want to change that or learning from what I've learned about and this Learn takes us from like grace. It's a pretty much a deal with a price That's like sort of almost in the moment We really do it by into the fourth one which is sustained that ability which is how can we learn from what's happened not just the learning
So do we have the environment in which we can learn and share and grow but then also adapt the system based on that learning and it's called Sustainability because it is that is a completely continual process. That's not just today That's not just when you have an incident that's you don't have an incident What are we doing to learn and I think your point you raise that was excellent, which is I like reading other people's instant reports There's a little bit showing for you with companies like but it's also a little bit of it is like they've learned something Can I learn from them and do something that does my system? So when you don't have an instant read other people instant sports says the place called the void It hasn't been updated for a bit. There's like hundreds and hundreds of instant reports That's worth going through so that idea of sustainability is all about us adapting and changing over a long time scales for those last two things So staying that's missing great sensibility this all comes down to people and culture like having proper environment cycles safely is just Absolutely quiet if you can't raise issues if you can't challenge in a safe way You do what's a price bad because things don't get raised and you won't learn because things will be hidden Yeah, and so that all comes down to behavior stuff and so the last three chapters of the book we focus on that is looking at some of the people
Social aspects of that sort of 14 13 chapters 12 chapters. Sorry about the other technical and then lastly chapter We'll run those side things So like what can we do to make sure we have that environment of psychological safety because they aren't you can't actually be resilient? That's pretty amazing because so far throughout this whole conversation We talked about building Brazilian systems We talked a lot about people about about housing but resiliency one thing that we didn't really talk about just passing We as AI even though like this book is coming out now And I know it has a section on AI as well But it's just interesting for like that even though like AI is changing how we're building software seems like some of these ground truth How to operate reliable software are a little bit just outside of its realm, right?
Well, I mean, I mean, it's not I mean First thing is some stuff in there that can help I detail quite a few examples where around sort of correlation analysis and you help support around Post-ins and some ins and dot I would do some amazing stuff in this area This is what we can use tools to do a lot of it's a really there's more fundamental kind of things here The first thing is the macro and the micro the micro level is new stuff We can do here and I talk about the macro level is that none of it stacks up financially Right, so if you're like taking a longer view of our resilience if I'm building my product based on a market sector that is fundamentally flawed These companies don't have a viable path to sustainability What we say which one we're talking about open AI and anthropic, yeah, and the high-vives is right Conservative analysis by the economists are showing that they need to be generating 2.6 trillion in revenue to justify the CapEx Especially 2030 in tax software markets on point four trillion so macro level. I don't know how that gets sorted I'm not smart enough. I also don't have any shares of its companies, but you know, that's a separate conversation I can't solve that what I can start thinking about is okay, that's the one I'm operating in How do I deal with that? I can't deal with all that what I can do if I'm taking a few two billion years being very careful about how I use AI
For example, you know, if I'm thinking taking a longer view being more flexible about model choices That's gonna help me because I've got the opportunity to open away models and different vendors We think it's basic things like open AI and anthropic are pretty terrible when it comes to availability of their core models from their services Which is why everyone I know also would use like they'll use Google or Amazon or Microsoft as either their primary place Like I said, there's models that fall back so I have better uptime and things so there's all that stuff We've got to deal with that level. There's also a conversation about non-determinism and charity called me up on this a bit I said the thing is that you know, we're going from deterministic software to non-determinism You know, that's a challenge for us when things work and then she said to me our distribution system is never deterministic And now they're really not deterministic and of course She's right because when we take our disability system and we add the people in beyond the rabbits, right and the fires and the people We always have the degree of non-determinism. We just amping that up a bit. Yeah, so I think that's a continuation So for me, I think it's a confusing landscape. It's not good or bad I think the micro level is really great things that we can do was the micro level that's the tactical things that we can do, right?
I think really I mean by that is I'm taking the macro economic policy situation out of it's better But as we can make you stuff, right? There's some good stuff in there that we can make you stuff But I also think as an hedging we need to do longer term if you're building AI into your product There's some simple hedging things so being multi vendor being multi model for example breaking rather than having a single agent Breaking that down to more workflow models therefore you can take bits of your workflow and realize actually it's gonna Now be swapped out for a bit to terms that I write the problem efficiently I think these all sensible things that you can do we can't fix this Mm-hmm. We can at least deal with our world and I think there's some sensible things that we can do in that space What are the things that we know that we do not know about AI?
I think the tech world in general is quite naive about AI and fundamentally misunderstands what an LLM is. Let's be real. Why did the LLM delete my database? Well because I have no concept of causality. That's what it comes down to. There's no concept of course there's a glass here
I push that glass onto the floor. You'll be working as a copy down there If I ask you what's gonna happen instead of last week and you find if I push it onto that floor You're looking at it as a hard floor that might break you're carrying out a bunch of Calculations in your head you've got an idea about the physical properties of the universe You've got to since I push the glass this is gonna happen. LMs don't do any of that There's no concept that if I do AB happens. Well, there's a push for world models, but these elements are not world models Well, well, well, it's been around for a long time. They arguably predate LLMs They can do stuff like that so they could make decisions like if you have the appropriate model and appropriate inputs They can at least get support of if I do X I then why happens I've been told not to do why this is well the guardrails around them so really not long term They'd like solution and so I think we expect LLM to do more than they can actually do because they seem so smart Yeah, and so I think that's the problem and that leads us to Over estimate their capabilities a bit too much. I think or maybe over rely on them without knowing and we might have some incidents later
Absolutely absolutely and I think we're knowing where and when to kind of to to use those things And I think what we don't know right now is where should I be having and again is wider? Where should I have the LM do it? And where should I just write some deterministic code like there's a lot to be said but just having your agent call a script and you know That's great works rather than having it implement scripts in its magical brain for you Well, there's something to be said that it seems both labs both leading labs and dropping and opening I have realized that coding is so important So oftentimes when you ask you on to do something it will start to generate code because it writes a deterministic program Like because that part will be deterministic And this very interesting when when you ask it if it will often just pump out a Python or types of whatever Especially when it has to do with involved compare this or that and they have trained these models to like default to Decoding I'm not sure how that this came but it seems to be working to some extent. It's also like a straight tractable Yeah, right. You know, they are there to generate plausible code So they're just like plausible words, right?
Yeah, and the context of code they generated some plausible code that we can now validate does execute Yeah, and so that's a really nice tractable aspect And there are some places where it works really really well, but there are other areas Terrible like I run my book I use very little a title Well, you don't use it for the writing users for The writing is always if it's all is a very research, but in the context of the research is if I use no But in that context, I plausible but then I don't click on the links. Excuse me and I say well that they're wrong I am checking the work It's done way more work than I could have done going through all those papers and gardeners I mean this paper should read through right but that in that context I feel completely fine and able to do that in other settings you're getting now this problem called cognitive surrender Which is where you're using a item right documents that you're never going to read that's not really around a problem with the LLM technology That's the problem with us. Okay, can we talk about this cognitive surrender and cognitive depth?
Yeah cognitive death so kind of terms been around for a while and that sort of explains the erosion in our ability to Think critically into reason that can occur when we're using AI tools There's been a bunch of research into this show There's been some studies that showing the people's ability to think critically is impacted the more they use chat box and things I think we've sort of seen other examples Margaret has always got a great article about this and she cites some sort of research around it as well And that's sort of explaining that I mean her thing is she's talking about technical debt building up in system We can be careful about you so I have a cognitive debt We need to have a shared mental model of what this system is supposed to do thinking about it narrowly for the lens of a program building And if our user they are An example we all go off in different directions that shared review breaks down that becomes a debt Cognitively cognitive surrender is a bit of a different thing which is like oh, yeah, it's fine It's basically LGTF right yeah, but for whatever the I tells me yeah LG same what's typical looks good to me approving large pull requests where you couldn't be bothered or you try to look at it for whatever reason you Didn't want to go through or I came from someone who you
I was the I wasn't able to talk to him on all stuff right and so LGTM was literally a sandwich pattern match on and takes out an approval Yeah, so you saw LGTM that person's approved right and it was that was I think that may be without me Which you came from not sure I think you can't remember yeah, but it is this idea that it's become shorthand for And I think that's that's that kind of surrender thing is coming out lots of ways You know we're getting AI to write documents and now it's gonna read and we've got AI That's reading documents that weren't written by other people then so the concept is really that the humans It's come back to the human being in the loop. There's very subtle differences in those workflows into the kinds of differences it changes like there's There's a big difference a co-product talks about a big difference like if you were say a cancer on oncologist or I can special There's a big difference to use oncologist working and an LLM doing sort of detection of some scans you're looking at going up Should look over there by the way, you might be That's actually quite useful right versus the LLM power AI going here's what scans I've looked through here's the one I need to sign off right the last one the second one right takes the expert out of the loop and makes them an approver and often in that context
They've got rid of a bunch of oncologists. You've got one person in the funnel but it's gonna lead into issues right the other model is we're making people better I think that's come back to original idea of AI or something to pitch right was it was free of some drudgery? No, just like the Dutch Revolution is going to free some drudgery But like it would elevate us we'd have less toil we can be this we can focus more on critical thinking and smarts and so much Our current use of AI is not freeing us up in critical thinking is causing more context switching It's causing less time for thinking people are working longer hours. We've got to show this now That's not necessarily AI's fault. It's just how we're using it. It's like we address that a little bit
Can we talk about complexity and AI maybe adding it? I'm gonna quote you from about it from a few months back You wrote I worry AI could just become an exercise in doubling down on current complexity rather than stripping things back to simplify at least in the platform space Yeah, I can't think of that context in which you had that conversation But the platform space was all I mean the platform space is kind of interesting right because that was an attempt to make things simpler developers because Kubernetes became popular and it's not developed a friendly. That's not a sign for my business. Most of us are rather such a No, I mean well if you have a platform Yeah, yeah, there's a platform things problem, right? And that's what they get paid that will they get paid but and so I think there was this idea that I was sort of hoping that they'll be a bit Complexity claps around that space. I had his hope for things like WSTI for example simplifying down those programming models a little bit
And I think I was starting to be a bit of a bit of a conversation It's not to be heard about some of that stuff going on what then happens of course is AI has come out And everyone's like I've got to AI all the things and so rather than thinking about what we need interest to get that complexity back We've now just got another piece that requires it still have all the old complexity adding new complexity on top and Somebody got a couple of that stuff. Yeah, but I think it goes that's a little bit of like we seems have more work with AI and probably doesn't help that They can't have more complexity, especially when we're not being clear of what we want We're not being critical of the output when we have cognitive depth or cognitive surrender, especially all cognitive center You know just right so it these things might be amplifying the fact that it's just more more complexity, right?
Yeah And I think there's also that connection through to what the AI is doing So it comes full circle a little bit Which is the idea of potentially not looking at the code isn't say that you don't care about what the system is that you're building Is to say that you've got a better way of understanding what your building so all the concerns is if I'm using AI to create So I'm gonna create awful those dare and the complicated system I was gonna understand or we can't ever happen we still need to remain connected to the system that we're building Which is what I'm a big fan of like The modularizing system whether not you put those modules and different computers I don't actually care getting a high to work within the modules having the humans think carefully about what those modules are on the connections between them Let the AI go hog wild inside those boundaries that are easy to change the gaps between them That's what human beings spend to think about the mental mold housing spits and slops together That was a very sane way for us to balance our time our energy So the answer correctly that you're saying that's probably a sensible approach is have the architecture in your head know how your system works Know how the boxes work know where there's AI inside of maybe it's generating code Maybe it's running your incidents response in isolated ways and then spend your time talking with other people in the business on your team
With your customers because you're as you said systems like software is people's ideas connected and you have things in your head Or you write it down But now using it as a tool to enable you to maybe take on larger chunks of the business and maybe make better decisions Because maybe you have more time to as an engineer understand more on the business on what is working what is important those kind of Yeah, and also to make the structure on the model is a bit more explicit as well I mean That's one thing a good module structure does when this is why shopping for I've been so successful keeping a thing is a monolith right Because they spend a lot of time building them on the application as a modular domain given architecture, right? Because it makes the structure easier to understand you can zoom out and see how those things fit together But also in the short term while we're having these debates about are we going full software?
Full software factory or not we don't have to make a one size fits all approach We can say look this is our architecture be a modular monolith that microsarcasm really matter and over here this area Here's where we try to look at the code anymore, but we specify the interface we have verification around that We know what good looks like for that module, right? We've got functional tests to look at say some of the operational requirements as well These areas over here we still rapidly these are really key and really important to us We're not really to give up on those yet because I don't have the trust level yet We don't take that risk just yet And so over here we're gonna work with the AI more collaborative way while we work together We're still gonna insulate involved because we're not ready to make that step here But if we approach our whole system with one big black box, then we've fought into making one on either decision And I think that's healthy Do I understand because you're saying that you think that the software actually dark factory you give a good at this spec or and The AI building that are operating at that that you see that is viable There are some companies I know that already already on this not everyone should be doing it whatsoever I think if that is our future if that is our future I think it's a lot of this there the question is about how you got that adopting and trying So what I'm trying to do is find ways in which we can lower the barrier to entry to try these ideas out in a safe way
And like more risk environments and it may be simpler domains those kind of things right people also like a lot of dark factories Don't create finished products like real world be on the fact that real world dark factories don't mean it for most of them correct components They don't create the finished product. Yeah, so in the same way to think about a module being a piece of the overarching system well, there's a factory on these bits here and And actually looking at it like if we had to rewrite that from scratch it might take us like with the collaborative way But they might take us a few weeks after we write that we had to So actually we're quite relaxed actually that's a purpose is it's working final It's a module around which you've got great test coverage away for example where it's not on the principal path from a latency point If you or it's a service where we've got really good observability data and really well defined SLOs and we've got good state state Well, we can start running as a factory now and seeing what's happening in production We can have a look at the SLOs we may be adding some special SLOs around it I'm here you're saying that as a professional someone who wants to be up to date with the new techniques that we can use it's probably if you have the opportunity to start to build a Software factory is somewhat autonomous place in a component in something lowers
It's probably we're trying it out to see how it works to see what you learn to see if it even works or where it breaks down But do it in a way where you're minimizing the last rate is a little bit like when you know what microservices came out the concept You didn't start by moving your biggest thing you started to build a small service that can really matter all that much And you were okay to learn from it when it goes down and it wasn't on the on the health path for the business Yeah, it's in start with is getting head around spec So looking in spectrum development with that's Kiro or Tesla or spec it whatever that is I mean they're all quite different How to get to the article on that is a good on-campus get ahead of it start off by playing around with that because that's the stepping It's like you have to find what good looks like as an input your factory. Yeah start with the spec Good look input and also pretty output Yes And then it's like how you verify how you're gonna go about very funny out I'm a factory have a think about that and you know like not just in terms of the kind of normal sort of behavioral aspects of it But I'm with Kevin You see so he hates non-functional requirements because it's a functional if something is non-functional It means it doesn't work. So he breaks the world down. It's like development requirements operational requirements and business requirements I think it's nice by doing that so long as reacts these How do I verify that the software being created by this module is has matched my developer requirements?
So for sustainability my operational requirements latency uptime and my business requirement is it working features working? So long as you've got a way Validating your little software around that and you don't know what the good looks like for those three things guess what isn't for you But you can do this but you can't be factory. No, you do specs and that's your edging and you wait down I think it might not work is there may be some there may well be some kind of this could be Model driven development all over again, which we do not have time to get into very angry about it But it could be that but I have a hope it might actually work this time unlike MDM and then as it's closing We I think a lot of what we talked about is it's important to hold the art of turn your heads to think about the art Is to think how to modulize now for engineers who want to get better at this kind of high-level thinking strategic thing architectural thinking outside of reading your book Which I think brings a lot of really good ideas on architecture as well I'm not just really interested in some outside of that what would your recommendation be for a tech professional friend engineers like I'd like to get better at Architecting system to know what good looks like I think there's a few books that can point people out
I think the self architecture fundamentals by Neil Ford and Mark Richards is good read just to be clear their friends of mine So I'm not I'm biased in recommending it. I would say around the modularity I'm like on offs book on balanced coupling is fantastic And he's book on DDD is also very good, but that that balance coupling book is really great He's also direction skills and clothes it takes those ideas to definitely definitely take a look at those I think you're looking at things architecting you think in terms of modules for me I don't agree with having your marketing is but there's lots of great stuff in there But for me actually will start I sort of into that writing the technician ability microservices was really all about smuggling in Getting people thinking about single information. I think it's very More than 1970s still comes back to me modules still the whole thing for me So I would read that's book and if you want to go read your partner's papers from 1971 1972 That's still worth reading but those be good jumping off points I would say you can still I think the nice thing about thinking terms of modules is even within an existing code base You can start to modularize it in some sort of ways You can start to think about bound reading things about clear separation about using cancellations of information And I guess it's fair to say like look if you have a idea how you can make your existing code is a bit more modular
You have zerosuits not to do it I mean we don't trust agents fully but this is a textbook example of like if you can say here's my goal refactor This like if it would have taken by hand extra time it will be faster So you should be empowered to have these experiments with you right? Especially if you've got a decent test week and if not start with that start with that if you've got a strategy type language You don't even need to do just a good old fashioned ID. It's an amazing job, right? Don't you type language now don't have the excuse especially if it's a modern one that's well understood by the end So you feel using you know Python or Ruby it's so much better now It used to be for refractionless code bases absolutely you need to get a test week in around that But yeah, you can start modularizing and that's about building and that's also a little bit going back again to taking We don't want the architecture only in your head right?
We want the the code To sort of represent that mental model Yeah, because we communicate through the code maybe the future we communicate in through the sex and when you make those things more explicit Then I think that helps everyone's down the same page There's George Fairbanks wrote a forgotten name the book now But in it he has a pattern called an architecturally evident coding style and that we should code in such a way that it clearly communicates The architecture of the system and I think our programming languages are quite bad I have a good concept of modules in general So I think we have to go a bit above beyond ourselves But we could do more to make the architecture systems that a little bit more explicit This is also what I say in time production because especially if your multi structure ends up being a process structure In other words, it's process boundaries. I can go to the networks to find out my architecture I shouldn't have to but at least you know, I know production is true. It's like football is life production is truth So where can people get your book? You can go and order it now. If you can go buy from a local bookshop That's always a great idea It's also going to be available for people on the online platform in digital form You can go and read it as part of your subscriber But you can get it there as well, but a lot of people are having the physical copy and we have these in the show
I'm also checking out so good about directly and also you've been doing some videos, right? Yeah, well, I've been doing some stuff on Dave Farley's one stuff engineering channel It's a kind of a good excuse to do some semi-ranty, you know monologues to video and have some fun chats So I go for the one stuff engineering channel on YouTube And you can find videos from me and Dave Farley and Kevin Henny's there We got stuff from Kent Trisha Chi, Emmy Borsch down north is there as well I'm pretty sure I've forgotten somebody else it's most couple of people as well But we try and put videos out once we can get a bit from place to kind of work with you And if people would want to work with you where can they find you how can they find you what kind of what kind of stuff can they get help from you for? But I do like to tell talks to people like internet events I do like customer training courses and a whole variety of things around the systems and security and microservices I also do a lot of consulting for clients helping the rounds of re-architecting their systems and stuff like that If you know more about that, I actually go to my website Sam Newman.io and you can contact me there You can also find me on LinkedIn, but if you come straight to the website use email I'm a little bit old-fashioned like that and I work with clients all over the world So if I'm able to help you head over to Sam Newman.io my contact details are there. Wonderful. Well Sam, thanks again. This was great
Thank you. Thanks very much to Sam for this conversation I really like Sam's three rules of distributed systems one it takes time two Sometimes I think you want to talk to isn't there three resource pools are not infinite Sam said how most of the edges happened because of this last one resource pools not being infinite and services are getting saturated But I was chuckling when he was talking about rabbits doing networking cables between office buildings Then again a few weeks ago with thousands of flights stranded on the east coast of the US It was because workers accidentally cut a telephone cable in New Jersey used by the FAA Another food for thought from Sam is how failures should be handled differently depending on the type of business An e-commerce shop might decide to show you the item when the inventory service is down worst case after you order and pay They just refund you because we're like oh we don't have the item basically failing open is fine in this case But on a concert ticketing website You don't want to sell a ticket that might or might not be there and so failing clothes is right called to make us again As we discussed in the episode if you sell the ticket people might book flies They might get upset you might lose of us customers finally. I appreciate it how level headed perhaps even cautious Sam is about AI He's got no doubt that AI is very useful at the micro level basic a tactical that things up
We can already approve but he is worried about the economics of the model providers his suggestion is pretty pragmatic H your vendors and your models the multi vendor and multi tool and wherever you can swap non deterministic AI agent work Close to deterministic ones basically have LM generate deterministic code I find very very refreshing thinking check the show most below for the story about scaling uber the early days with hubris for C2 to on palm that we discussed and it's an awesome story and for more the pragmatic interior deep-dives on resilient systems If you enjoyed this show do subscribe your favorite podcast player and thank you very much if you recommend the boss cast to your friends or your colleagues Thanks, and I'll see you in the next one