A Conversation with Amazon CTO Werner Vogels

Warner Vogels is the Chief Technology Officer at Amazon, where he has played a pivotal role in shaping the company's technology vision for over two decades. Before joining Amazon in 2004, Warner was a research scientist at Cornell University, where he focused on distributed systems and scalability, both of which are concepts that would later influence the design of AWS. He holds a PhD in Computer Science and has authored numerous academic papers on the reliability and performance of large-scale systems. As CTO, Warner has been instrumental in guiding Amazon's transition from an online retailer to a global cloud infrastructure provider.

He is one of the key architects behind Amazon's push into cloud computing, helping to define the new model for delivering infrastructure. He is known for his pragmatic customer-focused approach to technology and for championing ideas such as You Build It You Run It, APIs Are Forever, and more recently, Frugal Architecting, which emphasizes cost-effective and sustainable software design. In this episode, Kevin Ball sits down with Warner for a wide-ranging conversation. They discuss the early days of Amazon, the birth of AWS, the principles of the Frugal Architect, aligning cost to the business, engineering business collaboration, technical debt, and much more.

Kevin Ball, or K-Ball, is the Vice President of Engineering at MENTO and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI and Action Discussion Group through Latent Space. Check out the show notes to follow K-Ball on Twitter or LinkedIn or visit his website, kball.LC. Hello, K-Ball here, and I have the absolute honor of getting to introduce today the Chief Technical Officer of Amazon, Werner Vogels.

Thank you, Kevin. Thanks for having me, and it's going to be from there. I'm excited to get to talk to you. I like to start, especially with people who have done a lot of speaking and being introduced places.

How do you introduce yourself? What do you highlight when you get to say, here's who I am. I've been doing this for 20 years, and I'm kind of tired now, but I have a lot of stories to tell. In general, there's always been to your past, and I've been an academic, and I was in the Army, and worked in hospitals, and all these kinds of stories.

Don't really matter. The last 20 years of Amazon have really formed me, especially on Amazon every year, it's a different year. When I was 18 or something like that, he would have asked me if I would work for the same company for 20 years. I would have asked it.

But now, it's 20 years later, and I'm still here. Yep. So we wanted to talk today about cloud architecture, and this is something that I think the rise of the cloud, Amazon, has very much led that way, and it has changed the way that we do software. But you've been very focused on it for the last few years.

So how do you introduce this topic as a field of study for software engineers? Well, maybe we should go back a little bit in time. That makes the whole story a bit more easier to follow. When just started Amazon, it didn't really want to start a bookshop.

Now, it was just fascinated by the internet. What could you do on the internet that you couldn't do anywhere else? And it just picked a bookshop. A good bookshop has 40,000 titles in stock, yet there's millions of books out there, so you could do something on the internet that you couldn't do anywhere else.

Unfortunately, nobody has written a book about e-commerce before, because we're not in the same state. This is 94. And so anything that Amazon tried to do after that, they basically had to invent themselves. Every piece of technology, everything you now know about e-commerce.

Now we probably just used Shopify as a platform, but there's nothing. Amazon engineers have been really, really good in inventing themselves out of the corners that they got themselves in, because there were many things that nobody had done before. And the cool thing is, actually, and this is already before the year 2000, that AI played an important role in all of that. Think about recommendations, similarities, customers bought Xboard Y, things that we now consider normal everywhere.

And we don't call it AI anymore, because it just works. Now, one of the things that definitely, in my earlier days at CTO of Amazon, we get fixed capacity. And we had a really, really good mechanism to sort of predict what would happen the next year. So we bought capacity for that.

Now, if I bought didn't buy enough, my customers will be in helping. If I bought too much, the CFO was so happy. So you always need to be careful in that. And then, of course, Black Friday came four times the traffic from normal and nightmare, really.

But what really happened five, six days before Black Friday is that suddenly the team would show up. They would say, oh, we've got this brilliant idea and it will deliver 50 million to free cash flow. And we need to implement it immediately. And I go, no, we've got.

But for some reason, we always made it work, because this constraints also breed creativity. And you found ways to do it. Now, when the whole cloud thing started, suddenly everything changed, of course. Because suddenly, you didn't have to live within the constraints that the original fixed hardware had.

And I thought that that would certainly also become the point where people started to realize about how much certain architectures cost. And mostly, I mean, I love the way that we built AWS. We built amazing technology that nobody else has done before. And CTO, of course, I'm amazingly proud of that.

But maybe even the most proud of is that we changed the economic model. Before that, I was extremely frustrated as a CTO, because, indeed, in vendors, I always felt that the vendor was in charge. I was never in charge. If I wanted to get the cost down and say, with this database, I needed to make a five to ten year contract.

I had no idea how much you would need at Amazon the five years from now. So you massively overscaled also to avoid sort of the penalties you would get if we check on the number of databases that you were using. And so, and then you would write this check and this check had many zeros on it. And the moment you gave the check to the guy that you were dealing with, you didn't care anymore, because it was paid.

And so at Amazon, we had this principle that we wanted to be the Earth's most customer-centric company. You understand how that works in retail. But we started thinking, how would the Earth's most customer-centric, I'd typify the look-black. Well, the first thing we need to do is change our economic model.

Instead of the people that have to pay your phone, which they had to do with every other IT company, you only had to pay for what you've used. And now that seems normal, but that was revolutionary. It completely revolutionized. You're in electricity at home.

You pay for what you've used. You don't go to the electricity and give them money for the rest of the year and hope it's efficient. So in some sense, it was a normal model. And I think people caught up on that pretty quickly.

Lots of other large IT companies didn't. They were still very much addicted to the 70% margins that they had. So the biggest thing that I mean, we're really proud of was actually changing the economic model. So this pay-as-a-go model, which are for us completely natural, also allows you to think about the choices you make, the goals and costs.

Well, for it, in 2008, the crisis, the financial crisis, we already saw some of that. Let's see if there's always some other concern about the cost of digital infrastructure. But most organizations actually that we worked with used the COVID period also as, oh, let's accelerate our digital transformation. Let's move everything.

And so we're really debt-concerned. And go forward to 2012. I gave my first keynote at the re-invent and I gave a long list of it. How I thought development would change, was changing on the cloud and all the reasons why we're doing that small building blocks.

We deployed it to availability, so on, a minimally production. And I also said, now you can architect with costly mind. Everybody ignored that. Why?

Because moving fast and innovating was way more important. It didn't matter, of course. You could get all the things that you wanted. And you didn't really thinking about your idea and the things that you wanted to achieve and your customers and stuff like that.

And kind of cost of all of that was sort of put on a drug. Until a number of years ago, I think when most of the people, most of the financial people started thinking like, should this really be costing us this much? And that made me think that if our customers started becoming more concerned about cost, maybe we should revisit that topic and give them some solid advice about how I think, after by that time, it was at 15 years at Amazon, the experiences that we had and how we sort of have integrated costs in many of the architecture decisions that we've made. And so that resulted in Frugal Architect.

Yeah, absolutely. Well, and I think we are very much in this phase right now where it's harder to raise money, interest rates are higher. Everybody's looking at how do we cut costs and actually pay attention to this in a way that they weren't when money was going freely. Absolutely.

And especially think now we're about all the efforts in and around AI. There's a lot of models over here, $15 per million tokens, and there's models over here, they cost $0.15 per million tokens. Which ones give you the better results? You know, and that's still a bad way.

When I say Frugal, architect, it don't mean cheap. I mean that you get maximum value for the money that you're spending. Or what you want to do and then work backwards from that, and this is how much is it going to cost me? Do I really want that?

Yeah, and so this is a bit of principle advice in Frugal Architect, but there is also some sort of practical advice. This is what you should be doing. Yeah, well, let's start to dive into that because I think a lot of folks, they do, especially with the ease of scaling up and down with cloud. It's very easy to just say, oh, use what we need.

And it scales and at some point you say, how did we get here? This is so expensive. So you start from the design side, right? How do you think about designing your system?

Well, of course, you know, there's many systems out there that have already been built. So, but definitely with the first laws or lessons or what if you want to call them, a really targeted, the upfront thinking. Yeah, where the things that are often non-negotiable, upfront, think about compliance, think about security, think about accessibility. Those are things that are just given.

And then there's a bunch of other things like reliability, fault tolerance, performance, things that you trade off against each other. And really it would come to that later. I think cost should be part of that. You should be upfront, be aware that whatever architectural decisions you make, there is a dollar amount associated with it.

And you know what, that's fine, because after all, whether you were buying a hardware or whether you have now operational costs in AWS, the money needs to flow in anyway. But here, every little piece that you make has a financial consequence and such that is at least something you should keep in your head upfront. Now, I also wanted to make the case that especially in AWS, until we can give people clearly milligrams CO2 for this particular computation that you've done, cost is a pretty good proxy for sustainability. The more you pay, probably the more resources you've used.

And in the search, you can keep these two things in bells. I mean, we're all, there's enough companies that require sustainability to be reported to the board these days. And I find, and especially at the younger group of developers, they are absolutely passionate about making sure that they don't ruin the planet while using AI, for example. And in the search, having cost as just one of the other non-functional requirements, which we always have, makes you at least aware of it.

Yeah, that makes sense. And in some ways, it makes me ask, like, why do we even need to say that? Of course, cost is a thing that we need to think about. It's true.

No, but even in the earlier days of Amazon, maybe in my own heart, where engineers would never concern about cost, it was there anyway. And of course, at the different level, you were concerned about cost. But as an engineer, you just asked for five more servers and boom, that it were. Yeah, absolutely.

So let's look at what that is, because maybe engineers aren't used to thinking about this. What are the important aspects to trade off in cost? I think your second principle is around how you align cost to different things within your business. So what does that look like?

One of the things that I find extremely important as an engineering organization is that you need to be deeply involved with this business. After all, we make technology for our business. Now, you don't make technology in vacuum, that's because you think it's funny. And also, you have to remember that AWS is a business to business.

We make technology for other people to build applications with. And as such, you need to be closely reliant to your business unit to truly understand what they actually want from you. I mean, I've seen so many old situations where the IT department is somewhere else. Yeah, and they get a list of requirements, go build it, come back to the business, non-sets.

And why? I mean, often requirements change and that's normal. Or you discover new information or the business changes trajectory and things like that. So I've always believed that engineers should be in the same room as business people.

Because then you start building applications or systems or support systems that actually really meet the requirements of the business instead of that the business has to adopt to you. Yeah, and I think that that's really making sure that you align the two. Also, very important is that technology costs money. And so if you have the scale, you just scale up and so you'll start growing.

But your income as a company is coming from a completely different direction. And let's say you've built it as a page-go model, but you actually have a subscription model. Those two things don't really work out well. So thinking upfront about over which direction you think you're going to get revenue, then you need to make sure that if you grow, your costs grow over exactly the same dimension.

Because otherwise, I'm pretty sure you're going to be ready to throttle. At some moment, a long time ago, invested in a young business. And they were making these small MyFi devices. Now everybody has one or your phones have internet connectivity.

And people buy these devices and they buy 10 gig or 20 gig or another. And at some moment, the customers get frustrated. Why can't I just buy an unlimited package? It can be expensive.

Okay, so they made a very expensive unlimited package. What do customers do? Start watching Netflix 24.7. And as such, usage greatly explodes above what the big price was.

So having a financial model and having a technology scaling model that are not aligned is at risk for your business. Yeah, that makes a ton of sense. Now pricing is also hard. A lot of folks won't pay for pay as you go in different parts of the market.

So how do you trade off those things or evaluate this as one of the many trade-offs that you might have? Well, trade-offs first of all. I don't think any, even before we put a cost into the mix, trade-offs are always a part of architecture. How highly available does this component need to be versus that component?

And what does the kind of performance, how much capacity do we need to get to this performance? And in that sense, for example, measurement is extremely important, knowing and truly understanding what is happening there. If you get an average webpage latency of 1.5 seconds, it means nothing. It means that 50% of your customers are getting a worse experience.

You need to know how much work. And you need to know how to control that endpoint, let's say the 99.9 percentile. And how can you bring it in? How much does that cost me to actually do the engineering?

And then do I get a return on that? The reason is that faster web pages give you more conversion. But there is a point of limiting return, of course, after 1.1 seconds, nothing matters anymore. And so you need to think in your engineering about how much is this work going to cost me.

And is there, from the business perspective, a return on it? Of course, we all want to build the fastest possible web pages ever. I mean, as an engineer, I would love nothing else. And doing that is just that for the business.

That's just busy work and useless because there's no return on it. So that definitely, I think, in the trade-offs part is really important. And it comes back to something else. Your application doesn't exist out of one big thing.

Take Amazon.com. You go to the webpage of Amazon.com. And there's a few things that actually always need to work. Because without that, we don't have a business.

Search, browse, shopping cart, checkout, and reviews. Because if reviews are not online, people are not that interested in buying. Then there's a number of things that are quite important to customers. Customers who board the export, why are your recommendations, similarities, things like that.

And then there's a bunch of nice to have. Best or least, now you need to have a discussion with the business about how much money you spend on the things that are tier 1, the really important one, tier 2, tier 3. And tier 1, you might say, oh, I absolutely need to replicate that over three different availability zones. Or I need 99% availability.

Well, that's one question. And tier 2, you may say, you know what, three nights might be fine. If recommendations are offline for five minutes, we can handle that. And then tier 3, now the best of the list, it may go, well, you know, that's offline for an hour.

I don't think anybody misses it. Yeah. And so this is a conversation you need to have with the business. I mean, we're used to as engineers to make those decisions.

And then what we do is we make everything four nights available, which is, from a business perspective, way too costly, because there are parts where from a business perspective, you can kind of live it out, even if it's for five minutes or 10 minutes. And it has a significant impact on the bottom line. So there's all sorts of tools and tricks that you can think about to decompose your application in such a way, but the most important part of it is that you then take it to the business and have a conversation with them. Because they're the ones at some moment that don't have to pony up the money for it.

Yeah. Well, and as you mentioned before, like requirements change as well. It may be that the best seller list is not important when you launch it. And you discover that drives tremendous amounts of purchases.

And so that ROI goes up and we need more reliability. Yeah. Or you build things that once you get it in the hands of the customers, they go like, what? Yeah, that's probably more common, right?

Then it needs to be cheap enough in retail, definitely. You have this massive ABA testing environment. And instead of hiring focus groups and psychologists and whatever before we build things, you might as well build it and put it in full of customers to see what they think about it. Yeah.

At some moment, we build something called your digital soulmate. This is the personal Amazon that is in terms of purchasing, just like you, wouldn't tell you who it was. But maybe we tell you about that person also bought that you didn't buy. If the idea that might be inspirational, customers hated it.

I hated the fact that there was somebody else just like them. Yeah. Now, maybe that's probably 10, 15 years ago, maybe it was the changing social media and things so that people don't care that much anymore. But it was much easier to quickly build, but would you in front of customers go like, no, that's not what you want.

So thinking about that and knowing that things are going to sometimes prove out to be not actually valuable, how much cost alignment, how much architecture went into that before you built it versus building it to be evolvable if it turns out to work well. So if you think about innovation at Amazon, I mean, everyone knows what really built out of all these small teams that all take care of their own real world and things like that. They're all trying to have innovation. And some of these innovations, especially because our A.B.

testing environment is so robust, aren't terribly costly. But there are teams that are working really hard on what may look on the outside world small problems. I don't know if you have ever bought shoes online. People who buy shoes often buy three pairs of the same shoe, different sizes, and then send them back.

I think that's a bad customer experience and it's not great for the business here. So we have a small technical team that sits with the shoes team and tries to understand all of these kind of things and can we build a data set? Can we build data sets such that, you know, if these Nike 11, that particular Nike 11 fits you really well, maybe you should buy these trails for Adidas. Things like that, you know, just those not cost of the money?

No, I mean, but it may make a major difference for our customers if we can really give that good advice. And as such, small teams are all charged with doing that kind of level of innovation. The bigger things go up to the central management team, of course. Now, if it requires significant capital investment, then we have this principle that if we make major capital investment, the result of it needs to have a significant impact on a balance sheet.

I mean, it's not necessarily interested in putting all the money on them than just getting that money back. But you know, if this is really successful, it should be really successful. Probably AWS is a really good example of that. But think also about things like the Kindle.

That wasn't something that you expected Amazon to deliver. And remember, we sell the Kindle still at cost. That means if you never read a book and Kindle, we don't make any money. And also in the early, early days, one of the bigger challenges to customer service was that people would call in and say, ah, and complain about something.

And then the customer service agents would say like, yeah, but that's not by Amazon. That's by a third party. People got no, no, no, no, no, no, I bought this on Amazon. So it's a significant portion of our customer contact.

All that to do with the third parties and mostly we shipping because people love to sell, but they hate to ship. And so starting fulfilling the Amazon so they could put their goods on our warehouse and say we will take care of the guarantees on delivery. Major difference. Did it require a lot of capital investment on our side?

Yes. Or think about something like Prime. Prime is not just a gimmick. We had to completely relay out the complete fulfillment network to be able to afford Prime.

And Prime doesn't pay for itself. Prime is a really interesting example. What if we broke it down from the Frugal Arctec principles, right? So like you said, it doesn't pay for itself.

So how does cost get incorporated into the architecture of what makes up prime? You know, if in around 2000, 2000, well, no, it's something like that, you would have asked people, why don't you buy your loan for your digital Amazon and they go like loan furniture? They sell books, music, video, or your TV or your electronics or things like that. And one of the bigger challenges with Prime, I was like, do not only build a subscription model, but to make sure that people would understand that this would not only apply to books, it would apply to if you buy a big screen TV, it comes free to you in two days.

And as such, you know, it incentivizes our customers to do cross-categically shopping, especially for those items that are, you know, maybe big or costly and transport or things like that. Now, it all comes in the same bucket. Now, of course, one of the things our customers really want is convenience and we're not really talking about two-day delivery. We're not talking about one-day delivery or where I live some part of the year in Dubai.

There's two delivery windows. That is two-day before six or two-day before 11. If you get something the next day, it kind of disappointed. And this is how people change the importance of technology, and that kind of things that we can do.

But believe me, if you want to do same-day delivery in New York, you'll have to make some investments in making that happen. And as such, there's a clear thinking behind it. But most of these also, you know, Prime, their experiments, nobody's done that before. And things are not an experiment if you really know the outcome.

And some of these experiments, just by the nature of being an experiment, need to fail. You have this phone, I don't know if you remember this, the fire phone, 800 million dollar write-off. Not all of these investments work out to be that we plan them to. And that's okay.

And when Jeff explained this to the shareholder meeting, there was nobody that better than I. Because in exchange for this moment, there are 10 or 20 or other big other success stories to make. And sometimes you make these these combos. Well, and as you say, like you can't know how it's going to turn out.

That actually, in some ways, feeds back into your Frugal Architect principles. Beyond design, you have this sort of measure and observe area where you say, okay, as we go, this is going to change. I'm sure Prime today looks very different than Prime when it was originally imagined. So how do you keep track of what's going on and then evolve it as you go?

Thinking about without measuring your flying blinds, and whether there's a lot of cost or reliability or uptime, or how are people changing their behavior and the Prime, but I wanted to make actually one thing clear. In Amazon retail, we have the luxury of being able to experiment. You bring things in front of customers, they don't like it, stop it. Or whatever, you adjust.

In AWS, the world is different. Because as soon as we launch something, people start building their business on top of it. That's not something you can certainly then pull the plug from underneath, because people have been doing this. I mean, we're still winning SimpleDB.

You can't sign up for it anymore. But there are a number of customers that are still using SimpleDB because they've tuned the hell out of it and know exactly how you want to run it. But we launched this and the same is like with APIs. Now, APIs are forever.

It's one of the hardest things to do is API design, because you need to think about sort of how is this going to evolve because this is going to stay around for a while. And so measurement is extremely important, but also making measurement visible to everyone. And I wasn't told the story, and it goes back a long time. Most Americans don't know this story.

But in 1972, there was an oil crisis. In other words, the hijackings and the thing at the Olympics in Munich, with the hostage taking off the Israeli athletes there. And so on Sunday, we couldn't drive a car. But also a lot of research was being done why some houses use more energy than other houses, although they were comparable, same family in it.

It turned out that the family that was using more energy had their energy meter in the basement. The family that was using less energy had their energy meter in the hallway. That meant that every time they walked through the house, they got confronted with their energy users. And that changed behavior.

Every member, one of the first jokes I made, so envy go home at night, you can turn off the lights. Because as engineers, we're usually having some desktop underneath there, and you just let it run, we go home. In the cloud, it's a much better idea to actually shut your development environment down, because you're not going to use it anyway, because money. If you have that on a big monitor somewhere in your engineering environment and see how these things change up and down, it changes behavior.

And so getting good insight, also in sort of making changes between do I want to run this on Intel, do I want to run this on a graphiton? And things like that. So immediately seeing that your cost drops by 30%, it is a big monitor field. Yeah, we are natural optimizers.

Give us a number and we'll try to push it up or down, whichever one it is. To be the same, we talked mostly about sort of if you have total control over how you building your applications from scratch from gaining green fields. Most of our applications aren't like that. They've been around for a few years, people that developed them, probably are not around anymore.

But paying off technical debt is crucial in any organization. No matter how brilliant you were with developing your frustration, I'm pretty sure there are some things to fix, some things to refactor, or some things to look at, where am I cost going? And does that meet the intuition of how much this should be costing? Before cloud, I remember at this moment where I think we had 12 different searches within Amazon.

Don't ask me why we don't centralize those things. But there are 12 different ones. It's a natural outcome of experimentation, right? Like experimentation leads to diversification.

But then what do you do? As a team, you're allowed to move fast. And if you feel you're being hampered by another team that's seen as a complete something, you just go do it yourself. I don't know if you realize that has been forever a button in your order that says digital orders.

That was mostly because integrating digital into the traditional order pipeline was a way to mature, because we allowed that team to build their own pipeline. Not the best customer experience probably. But we were talking about something else. What was that?

Technical debt. Technical debt. And so, you know, there are certain engineers who love to tinker, who love to... So I'm not showing every engineer to be the same.

Some engineers love the babysit and SAP system. And they are conservative and they make sure they just think we'll run to the max. Absolutely. Those kind of people you want to hire for that.

There's also engineers that would go off to think, and do some innovation here, and shape them off to something else. But there's also a group of engineers that really love to look at the minuscule things that make things better. And, you know, if you can find... can build a team out of these people and having this go around the company, go look at things.

Can we see... we have so many optimizers, we have so many deep insights into the execution of these things, all these flame graphs and things like that. And where's all this compute going? It should be going anywhere.

And, you know, they find gold. And so, that's why I think, you know, it's... And it's obviously saying, pre-emeter optimization is not a good plan and things like that. It doesn't mean you shouldn't use your brain up front when you're actually building something and not show this up to a later moment.

Oh, we'll take a distillator. It's a bit like... and then technical debt is like the mortgage on your house. Now, if you don't pay off your mortgage, the bank comes and we possess your house.

If you don't actually, eventually solve all your technical debt, it'll come back to haunt you, whether it's in reliability, in cost, in performance, it will come back to haunt you. So, it is a worthwhile effort to put some engineering against. It's interesting thinking about technical debt in the context of what we've been talking about today in terms of aligning cost. I think as a business goes through these different phases, as you do experimentation, as you...

our product goes through those phases, you assume technical debt because you're optimizing for different things. Right? So, you may not optimize for cost upfront because you think there's a 70% chance this thing gets thrown away. I have a really pretty good example of that.

I mean, you started Amazon Fresh. We had no idea how the interface was going to look like. How did people want to interact differently with the Fresh interface versus, let's say, the normal video interface? Yeah, of course they wanted.

And with subscription mechanisms and things like that, but you wanted the team that actually was building this to be really agile, to be able to move things around fast. So, they start going from Ruby on Rails. Why? Because it's a great prototyping environment, good visuals, you can do this.

But they knew on day one that the moment that they needed to start scaling, it needed to be rewritten. Let's put it like that. Yeah? Because it wouldn't be able to...

or it would be a point maybe able to scale, but it would be at an enormous cost. And bringing it back to the normal Amazon principles, we decided to go with Ruby on Rails first because we didn't know how things were going to look like. It was a great prototyping environment, but then you do need to pay for your technical debt eventually. I'm feeling exactly that right now in my day job.

By the way, we're paying off a Ruby on Rails technical debt issue. Yeah. What did I see yesterday? The most...

I mean, Stack-Op-flow is not that popular anymore, but they do have this survey, I think, which I pay attention to. And the drop-in Ruby developers seem to be something like 75%. So, there's quite a bit of movement in that, which I always like because I think if there's one thing as engineers that we're always forced to do is new learning stuff. And it's the cool thing, I think, but you do need to do that.

Well, I think programming languages is kind of at an interesting moment right now, especially with all of these LLM-assisted coding tools and learning a new programming language is probably easier than it's ever been. Yeah, I do think so. Yeah, no, I think there's learning anything these days. You can get a little assistance on the side that will help you.

It would be great if the assistant would actually really be able to track your program really well and sort of start suggesting which things you should be paying a bit more attention to, but I'm pretty sure that will happen in the future. But one of the things with respect to programming language and the Frugal Architect, let's go back to that one. I ended the Frugal Architect presentation with a quote from Grace Hopper, from Admiral Grace Hopper, our famous first programmer. And she says, the most dangerous word in the English language is, we have always done it this way.

I think that goes to often how we treat it. Oh, I'm a Java programmer. Or, you know, oh, we're really good at Rails. You know what, this looks just like the project we did last year.

Let's just repeat that. And especially when it comes to things sustainability and cost, we should really reevaluate which programming languages we're using. You know, if Python and Ruby are 75 times as inefficient as Rust, that maybe there should be some light bulb going on in your head and go like, well, maybe if cost is important to us, or sustainability is important to us, maybe we should investigate. Now, I know learning REST is not the easiest thing to do.

Here you have a compiler that is so big that you start to hate it. Everything you get in exchange, blindingly fast, superior security properties, really make it really amazon at this moment, the programming language of choice. We are rewriting significant forces in REST. Single principle engineers wrote a great article on my blog about how they're thinking about why they moved from Java to REST when they were building a raw D SQL.

And security plays a very important role in that, but also efficiency and cost. The thing here is that there will be another programming language after REST. That is even more efficient and has even better security principles, or whatever, as engineers will be learning the rest of our lives. And I think it's fun.

We as engineers, we have the most amazing jobs in the world. We can go to work every day and create something new. Who else can do that? Nobody.

And we have the most, we are the artists, we are the creatives. It doesn't look like that on the outside. Yeah, well, and this brings back, I think, a little bit to something we were talking about in terms of paying off technical debt and different languages maybe being appropriate for different phases of a project. How do you set up your project to enable you to, say, move easily from Ruby on Rails over to a REST based project or something like that?

I know you've talked before about Evolvable Architecture. So what does that actually look like? Evolvable to architecture is a little bit different because, well, let's take Amazon S3. We started off with six microservices that was the whole thing.

Now it's very low over 200. Or obviously, you don't know, I forgot to know, then you can probably tell you that. You know, everything that we added, but we never took S3 down. It wasn't that we changing software or things like that and send an email to all of our customers saying, sorry, but Friday evening between six and 10, S3 is down.

Well, probably you got TV when we work anymore and you can't get coffee and I wouldn't be surprised to be at that. I'm working anymore. But always evolving yourself because you know that a few years from now, you will be running a completely different architecture. You think differently.

S3 is a great example. In version one, we were just storing three copies of an object on different servers. And then we started realizing that it is a coding actually could help us here. And actually still get the same level of durability, but you needed to store less data.

And it's a significant cost improvement, of course. So you can introduce that. Doesn't mean you come to recode all the old objects. Maybe once you touch them, you recom them, but you just leave them there.

And so we know that sort of over time, we went to different storage model, where you would only store an in-service or in one server or add all sorts of functionality. Now these days, you can store your factory embeddings in S3. So evolveability is really realizing that this is not going to win forever. But we need to build it in such a way that for our customers, it looks like, it will win forever.

It can't go that. So that in terms of evolveability is crucial for us. And in terms of me thinking about technical depth, when you think about the example that I gave you multiple programming languages, sometimes what you do. So in Amazon, we organize the small teams.

Each team has ownership over the piece that they have control over. They can make decisions. There's agency there. So if you want to do something that may spend some of these teams what you do is you fire up another team.

The other team has a task either coordinating or maybe do some of the work. Because remember, these teams are relatively small. They weren't sitting around waiting for more work. They had their roadmap planned out for them.

But you may actually bring a team on board that whose task it is to start carving up pieces of fresh, rewriting them, making them as a service. So we can actually call them. We don't need to worry about how they're implemented. So you need to play around a little bit with your organizational structure as well to get these things done.

Yeah, that makes sense. When I think the carving off and the balance of the API is forever, the contract to the customers forever, the implementation you can't fall into that we've always done at this way, Trapp. No, because there's always something new. Sometimes you build things that are intuitively, what you find intuitively the right way.

There's a great website called the Amazon Builders Library. And there's a whole bunch of, how to say, non-obvious thinking where there's a great one like Calm, who's about constant work. I mean, you really think that's inefficient. But in terms of stability of a system and other principles around system, it looks great.

And so here you have, that's a great document where things are being traded off against each other. A bit more bytes on the wire, a bit more work to be done, but the system itself is work stable. So there's a lot of work you have to do, but you might play around with your organization to actually get that done. Yeah, no, that makes sense.

Well, we've covered a lot of this. We've kind of gone through all the different pieces of the frugal architect. We're getting close to the end of our time. Is there anything we haven't talked about today that you think would be important to cover before we wrap?

On this big topic, no, I think we've gone well. But as always, I want this to be practical. I don't want this to be just some high level architect that never got his hand dirty and suddenly he started talking to you about cost. I think important is the relationship between the business and tech, because that's where he only met us.

And so if you keep that in mind, a bit of an agile working together with the business to make sure you build the right things. And requirements change. That's not new, by the way. In the 1990s, there were enough reports already about changing requirements as why all these big projects fail.

And actually, that is still the case. I do believe that the involvement is centered around decomposition. So decomposition is small services, but also the decomposition of the application into cells such that you minimize the exposure to failures. And just so many different technologies to think about, really build your system.

I'm still having a great time. Yeah, I think there's no shortage of things to learn and to do in the engineering world. Absolutely. Awesome.