Lee Adjison spent seven years at Amazon working in retail, software distribution, and Amazon web services. He then moved to New Relic, where he spent four years scaling the company's internal architecture. From his decade of experience at fast-growing web technology companies, Lee has written the book Architecting for Scale from O'Reilly. As an application scales, it becomes significantly more complicated, while at the same time receiving more traffic.
The intersection of these two problems, the increase in complication and the increase in traffic. This intersection leads to a variety of discussions around availability, risk management, and microservices. Lee and I didn't have time to get through everything in his book, Architecting for Scale. But if you enjoy this episode, check out the book.
Lee also spoke recently at the O'Reilly Velocity Conference in Santa Clara, so you can check out his talk or check out the upcoming Velocity Conference in New York. Lee Adjison is the principal cloud architect and advocate at New Relic. He's also the author of Architecting for Scale, a book from O'Reilly. Lee, welcome to Software Engineering Daily.
Thank you. It's great to be here. Your book, Architecting for Scale, opens with a discussion of two problematic things that happen when an application, a web application, begins to grow. The application becomes significantly more complicated.
The application also starts to receive significantly more traffic. Both of these things are problematic, but the fact that they often occur at the same time with an application is particularly problematic. Explain why that is. What are some of the canonical problems that happen at the cross-section of these two events?
That's a good question. I think the biggest thing that happens in a nutshell is it happens when the company is getting successful. As the company grows, the company gets more successful. They're naturally getting more traffic, but also their customers are demanding more and more capabilities from them.
The more capabilities typically require more people building more features or more capabilities, including scaling capabilities into the application. The larger number of people working in the application, the larger number of features going in, inherently the more complicated to get. Unfortunately, success breeds these two problems. You get more traffic and you get a more complicated application.
At a high level, what are some of the basic techniques that can be used to build and manage these large-scale applications and ward off some of these problems? I think the biggest, the number one thing that people should keep in mind when they're trying to approach this is they need to understand what risks they really have in their application. Knowledge of the risks that are in their application and understanding what the consequences of those risks are is probably the biggest thing they can do. In the book, I talk about the concept of a risk matrix, which is a way for you to essentially bring visibility to the risk within your application and talk about not only how likely a risk is to occur, but how severe that risk might be to your application.
Absolutely. I want to get into the risk discussion a bit later on, but I want to talk a little bit about your experience. You've seen several real-world applications that have suffered from the scalability issues that you're talking about here. You were at Amazon for many years.
You've also opened at New Relic for a while. Let's talk about Amazon first. What are some of the scalability problems that Amazon encountered while you were there? Certainly.
When I first got the Amazon, they were just in the process of moving from a large monolithic application into a service-oriented architecture. They were running into the problems of success, not as much from a scaling standpoint from the side of the number of requests coming in. They were handling that pretty well, but they were very severely running into the problem of scaling from the number of engineers working in the same codebase. Their whole philosophy there was based on what they call the two pizza teams, which is the small groups that own a particular piece of functionality.
That works great until you have 500 two pizza teams all working in the same codebase. I just made that number up. I'm not sure how many there really were, but it was a very large number of teams. Once that happens, now you're stepping on each other's toes.
It became very, very difficult for them to innovate and to add new features and capabilities to the application. Really, it actually became hard for them to even build the application, have it pass all the test suites and to be deployable simply because so many people touching it in so many different ways made it very sensitive to problems. Their solution was to move to a traditional SLA architecture. This was before microservices.
Their services were quite a bit larger than what typically services are being built today. They moved to that architecture. I was involved in that migration process too, that service-oriented architecture. What that allowed them to do is to have individual teams truly own the codebase and the support responsibility.
You've topped the bottom, the responsibility for a particular area of functionality or capabilities that was important to their team. That really dramatically increased innovation. In fact, I think if you could look back with knowledge at what the Amazon website looked like in 2006 and 2007 and I think suddenly you would notice that around maybe the 2008 time frame, not 100% sure on the date there, I don't remember offhand, but around that time frame, you probably will notice a huge uptake in the number of changes to the site without any features and capabilities, etc. That really happened as a result of the completion of this move to the service-oriented architecture and their ability to innovate and to grow truly took a dramatic step forward.
One of the things I remember about this shift to the service-oriented architecture is there was this blog post, I think it was by Steve Yege where he was talking about this shift and it was not just a technological shift, it was very much a cultural shift and a team mentality shift. I think you talk about this some in your book, but what are the, how, maybe you can talk about how you saw the same as on how can an organization shift the culture to going, or are there cultural shifts that need to be made to go from this model of every two-pizza team downloads this giant codebase to every two-pizza team is managing this somewhat isolated codebase? Yeah, there absolutely is a culture shift that occurs there and I do talk about it in the book, but there are lots of components to that, not only technology-related cultural shifts, which technology you use and how you do your work, but also the really, how teams interact with each other and the need for a team to be able to be self-sufficient, yet dependent on other teams to do their job as well. It really can be a stressor for an organization and so you really do need a good leadership in place that can help an organization move its way through that cultural change.
And honestly, each organization is different. The Amazon move is very different than the New Relic move, for instance, and those are the two main ones that I was heavily involved with personally, but I've seen other organizations go through it as well and every organization is different, but ultimately what it really boils down to is there's a cultural shift that has to occur and it's an ownership shift is really what it is. It's a mindset shift of a development organization moving from being a part of a larger group to being an entity of its own that has responsibilities outside of its group as well as responsibilities within itself. And that shift really has to happen culturally from within.
When you were at Amazon, you worked at both the retail side and the AWS side and I can at least imagine how the retail side works. I worked at Amazon briefly for eight months, and so I got some sense of how the services work and how the management works on that side, but the AWS side is still almost this incomprehensible level of complexity to me. How did these two parts of Amazon compare to one another? Particularly, how did the AWS side of things think about scalability?
Oh, geez, there's a lot of questions and a lot of discussion in there. These are only can spend a bit of time on that if you really want to. So, can you talk some about the AWS side or I mean, obviously, they've stringent confidentiality stuff, but I'm just so curious about how things look on the inside. Yeah, I think it's still Amazon, but I think the difference is in Amazon retail, you remember from being there, there was a huge focus on customer experience, and that was the if nothing else mattered, that mattered, and making sure that the customer experience was paramount, and everything else could suffer if necessary, including people who work there, but the customer experience was paramount.
I think with AWS, well, still customer experience was important. I think they cared more about capability and their ability to scale, and so while retail cared about customer experience, AWS cared about scalability, and that's important because that's really the biggest feature that they provide to the customers. So ultimately, it's still customer experience, I guess you could say, but I don't think AWS could have survived if they focused as much on user experience, and I don't think Amazon retail would have done as well if all they focused on was scalability. You eventually joined New Relic, which makes application monitoring software extremely popular.
When you joined New Relic, it was in hyper growth. How did the scalability and growth issues of New Relic compare to those you experienced at Amazon? Oh, that's a good question. I think ultimately, there did the growth scales were probably very similar, except New Relic was earlier in the process than Amazon was.
When I joined Amazon, they were already a public company, they were already a relatively large company, but they still experienced huge growth. When I joined New Relic, I went to Portland for my interview, and the entire development team had lunch around one table. And now they have four floors in the largest building in downtown Portland. And so it's the same sort of hyper growth between those two organizations, but it was a lot earlier in the process and New Relic than it was at Amazon.
And in fact, I think they've felt the same sort of growing pains as time has gone on. Certainly New Relic has had growing pains, and we know that Amazon has had growing pains. Certainly. So we could talk about war stories for a long time, but I do want to get to some of the topics in your book.
And one of the earliest things that you talk about in the book is that most of the applications that we're building today are software as a service, and it's slightly different than what we were doing 10 or 20 years ago. And you write that SAS applications are particularly sensitive to scaling issues due to their multi-tenant nature. So for those who don't know what that term means, could you define multi-tenancy and explain why that creates difficult scalability problems? Sure.
I think if you think about a traditional application before, let's say before the cloud really took off and before web-based applications really took off, what an application was was something you downloaded to your computer, and you used it on your computer, or your team used it on a shared computer, or your company used it on a shared computer. But it was a, you had a copy of that application that ran on your own hardware and your own infrastructure with your own data source that the people within your control and your area of influence used, and nobody else could even touch that. It was completely independent. If you had, when a company had multiple customers, each customer had their own copy of their application running on their own systems.
That was the old way of doing applications. The multi-tenant nature of the cloud and of SAS offerings in general is basically the idea that you have a single application that's running somewhere in the cloud, and that application is running on multiple computers with multiple data sources and multiple redundancy. It's a large application, but that single instance of that application is used by all or a large subset of your customers, and they're all sharing that same application. The advantages of this, of course, are the larger the infrastructure for your application, the more you can centrally deal with issues like scaling, like availability, even backups, and operational concerns can be centralized in a way that allows you to deal with those issues a lot better.
But also, you can deal with scalability a lot better because you, individual customer is only using a rather insignificant amount of the entire application at any given point in time. So even if they had a large spike in their scaling needs, compared to the application as a whole, that's typically a small spike and can be absorbed within a size of the application. Of course, the larger the application is, and the larger number of customers you have, the more that case is true. And that's kind of the idea behind multi-tenant is you share these resources, you share this this application infrastructure among a larger set of customers, and then that result is everyone gets the benefits of the shared and larger infrastructure.
The first part of your book focuses on the topic of availability. How do you define that term availability? So availability is basically the ability of your application to answer the questions and perform the operations that it's designed to do. So I think the way I talk about it in a book is I give an example comparing availability and reliability.
If you ask something to determine what the answer is of 3 plus 2, a application that returns the answer 6 has poor reliability, but an application that doesn't return an answer at all has poor availability. So the idea is availability is about the responsiveness of your application and the ability of your application to do what it was designed to do. Of course, if you ask a service, what is the answer to 3 plus 2, and it gives you 4.999999, then maybe that's a little bit better. And we did a show recently about the Google site reliability engineering role.
And what I'm getting at with this is like, one of the conversations we had in that show is that defining availability is very important, but it can be a contentious issue because distributed applications can have these really unusual failure scenarios, particularly partial failures. Like 4.999999, because it's kind of a partial failure. It's like really close. It's kind of like that's not not available.
So how should a team come to a conclusion about what availability means when there is such granularity to the meaning of this term? That's actually a great question because you're right. That is a slippery slope. If you're not careful, you know, it is 4.9999 a answer or is it not an answer.
And honestly, that depends on the needs of the requester and the needs and the promises that the service or the application makes to its users. And that gets into another concept that I talked about in the book, which is SLAs, which are service level agreements. The idea is that it's really important that you define for your application what users can expect it to do and what the boundaries and guidelines of that are. And that applies not on an application level, but even at an individual service level within a company.
If you define what your service is expected to do and what it's expected to perform as, then those sorts of questions go away. So for instance, if the 3 plus 2 returning 4.9999 is in fact, it's coming from something that has an SLA that says one digit of precision maximum is all you can expect, then there's probably a decent answer. But if you are having a slide with a higher level of precision expected from your application, that would not be an acceptable answer. So in that case where there's a slippery slope, having a well-defined SLA in that area can draw the line of where on a slippery slope is okay and where it's not okay.
Absolutely. And you know, that Google SRE book that just came out, I think your book and the SRE book are extremely complimentary. And both of them, you know, talked about this SLA stuff. The Google SRE people talk in terms of SLOs, also which are like service level objectives, and I think service level I, or SLI, which are service level indicators.
These are kind of these different layers of abstraction around service level agreement. But in any case, in terms of operationalizing this, I've heard many experts in this space talk about the importance of health checks for your service. So you want your service to have a call that you can make to it that says, is this service okay? To what degree is this service okay?
Like what is the health of this service? How important are health checks to institutionalizing the level of availability of your services? I think for the organization itself, they're insanely critical. So again, if you come back to SLAs, where your application has a set of SLAs that it is guaranteeing, and I'll use that word, yeah, I'll use that word with whatever meaning you want to put into it.
But if your application is guaranteeing a certain level of SLA commitment, it's that application's responsibility to make sure it's delivering on that SLA. And there's various ways you can do that, but certainly health checks is an important part of that. And so if you want to make sure that your application is available from Hong Kong, you should do a health check that's, or you should do a check from Hong Kong to make sure it's available. But in addition, you may want to have your application itself go in and do a self-diagnosis occasionally and return that as a health check response.
And the key with all of these is you want to be alerted when there is a problem before your customers are affected by that problem as much as possible. And even if your customers are impacted, you want to learn about before they tell you, because you want to know about it and give a fixed result that before it becomes an issue for your customers. Another topic that your book covers that I just love is risk management. And this is also another section that is in the SRE book.
It's so funny how parallel these two books are. But why is risk an element of software architecture? Oh, that's really good. I think the big thing that I like to talk about with risk management is visibility.
And that is every application has risk. The question isn't whether it has risk. The question is how well aware of it you are. If your application that you just fell through, anybody's application will go down at some point in time and will have problems at some point in time.
But if you don't know or anticipate or expect or plan for that, then there are random events that affect your business. But if you know about them and can plan and expect and be able to deal with them, they become much more business decisions to be able to deal with. For example, in the book, I mentioned a, it's kind of an offbeat example, but I talk about the big game. And I talk about this person who buys a brand new big screen TV, invites all of his friends over to his house to watch the game.
And right as the game starts, the power goes out. Well, everyone goes home. Everything's over. So the guy called the power company to find out what happened and the power company responses.
I'm sorry, but we're only responsible for power 90% of the time. And the idea there, obviously there's expectations there. And you can talk about, I think I used that as an introduction to the SLA management issue of what is the expectation of your customers? What is the expectation of the service that you're consuming?
But also that feeds into the process of risk management. And so the power company obviously knew there was risk of power going up. There's all sorts of things that can cause power to go out. When can blow down power lines?
It would go into thousands of different things. But they can't remove all of those. But they can remove enough of them so that the ones that are left are at what they consider an acceptable risk level. So you're not going to harden all the power poles with reinforced steel and make all the wires 20 inches in diameter and solid infrastructures that we can never ever break.
That doesn't make either economic or aesthetic for that matter sense. But you are going to figure out what's the likelihood of a power line going down. What's the likelihood of the backup for the power line going down. And what's the impact of that power line going down to your customers and to your service and to your company.
And as long as you know those values and can plan and predict those values, you can make a business decision of what is an acceptable level of risk that weighs not only the cost of risk mitigation, but weigh that against the impact to your business and to your customers. Of course, you're talking here about known unknowns or known knowns. When we have a high throughput system, one issue of risk that is important to talk about is tail risk, which is risk that can occur with several standard deviations from the norm, which is this type of risk definitely gets exposed when you have massive load on your system. You have hundreds of services.
They have these unpredictable, well, I mean, they're predictable paths, but that's like a death star of paths that are so hard to actually visualize or predict. And all these services can compound into these unexpected consequences. And it sounds like these tail risks go, I don't care about this. This just never going to happen.
Three standard deviations away. But this type of stuff compounds into an event like Amazon search went down last month for eight hours or 10 hours or something. And is there a way to architect against tail risk? It's a good question.
And certainly, I love your Star Wars example, but every application has a two meter hole, right? If you just should perfectly, then the whole thing blows up. And knowing where that hole is, is what's important. And you can arguably say that it's impossible to know all risk, and that's a valid point.
But I would argue that in a lot of cases, we talk about things like like three orders of magnitude too small to to deal with risk. But when you're talking about the scale of application we're talking about, even a risk that is three orders of magnitude too small to normally think about means it'll only happen once every day, instead of once every month or once every one, every second or whatever. But the answer to say what I'm trying to say with that is the scale of those tiny risks gets blown up out of proportion with a scale of application and complexity of applications that we're talking about here. A strategy for architecting against this or institutionalizing, as I just say, I think is the chaos monkey paradigm and Adrian Kropf, who is the network's microservices guy or formerly of Netflix, he says that chaos monkey is the first service that you should stand up as your service or architecture or your microservices architecture.
For those who don't know, chaos monkey is basically the service that goes around and just kills servers at random. It's supposed to prepare you for scenarios where essentially any component of your architecture can fail at any time. What do you think of that idea that like, let's just make chaos monkey the first service we stand up? I think that is a fantastic idea.
And anyone who's listening to this, who is standing up a production application for the first time, follow that advice. Run chaos monkey in your production environment and never turn it off and have it run continuously. And that should be the most important service in your system. It is so much easier to put it in as you're building your production application the first time than it is to add it later.
Because when you add it later, if you go into a little bit more detail what chaos monkey is doing is what you're telling you what chaos monkey is doing is it's going in and saying, here's your production environment, I'm going to shut it off here, here, here, and here randomly just cause. And that is a horribly scary thing for anyone who has ever dealt with ops before. And it's a horribly scary thing for anyone who is a business owner for any application. And the reason why is because if you've never turned that service off just to see what happens, the likelihood is pretty high, it's going to cause a problem and cause an outage.
But if you build it into your culture and your infrastructure at the beginning, we are first building an application, then it's part of the environment, it's part of the culture and your applications already sanitized to certain types of problems before you start getting scale of customers. And so the net result is you have a much more reliable system, you have a much more reliable architecture, much more available architecture. And you've got this chaos monkey mechanism built in that will allow you to continually make sure that if you ever violate that trust by doing something that generates problems, you'll know about it on a smaller scale quicker than you will if you don't have it in place. What are some other ways to impose strong testing regimen and strong availability and risk aversion requirements on our system?
Like maybe you want to talk about continuous deployment or whatever you think are the important things we haven't touched on yet that impose this strong strongly tested regimen on a system. I actually had a manager at one time. I won't say where I worked at the time, but I had a manager at one time that says I would much rather focus on my ability to quickly make changes to my application than my ability to strongly test my application because my customers are going to test the application. As long as I can respond to their issues quickly, I would rather them tell me what's wrong than a QA engineer tell me what's wrong.
And that's it sounds like a bad quote when you just hear about it, let your customers do the testing. But in fact, that's really what continuous integration, continuous deployment is all about is the ideas get changes out to your customers quickly and let them tell you what's right and what's wrong. That's not only from a standpoint of quality, but it's a standpoint of feature set and capabilities and user experience. In fact, the matter is if you spend a lot of time doing significant QA yourself ahead of time and slow down your deployments, you're going to slow down your speed of change and still not necessarily catch the things your customers are the most concerned about.
So the idea of spending less time and energy doing QA and doing more rapid and more automated deployments allows you to respond to problems quicker. And in fact, you end up with a more reliable and more available system in the long run by doing that sort of approach than you do by having a huge QA department that tests things before you go out. So yes, speed of deployment, speed of change are really critical for high availability. Now, you have to be careful with that though too, because if you, speed of change can also destabilize your system.
And if you let it destabilize your system, then you can enter a downward spiral of availability. But as long as you don't let it destabilize your system too much and you keep that balance, but still have the ability to respond to problems quickly, you're in good shape. So, but I think one thing you're talking about here is that in order to speed up that that continuous deployment process that customers reporting your problems process is that if you have a large volume of customers, you can make a deployment that only goes out to like 5% of the customers or like 10% of the customers gradually ramp that up, which is great to do when you're at a certain scale. But maybe, I don't know, is that possible to do when you have a smaller system?
Or I guess maybe that doesn't even matter if you're not thinking about scaling quite yet, if you don't have a large population of customers. Yeah, that's a good question. And I think different people have different valid answers to that question. In my personal view, the partial deployments are really good for testing either new features where you're not yet convinced that your system can handle the scale of that feature for your entire customer base.
Or in cases where you want to experimentally show different treatments to different customers, and you don't want to impact your large customer set completely yet until you figure out which treatment you want to actually expose to your customers and then roll it out. I think it's great for those sorts of things. I personally, and I can say there are different views on this that are perfectly valid. But I personally don't like the approach of using the canary and a coal mine approach.
You can show it to 10% of customers and if it breaks, it only affects that 10%. Because a lot of the problems we talk about only appear at scale. And so exposing your change to 10% of your customers doesn't necessarily expose it to the traffic you need in order to truly test it the way you would if you expose it to the entire customer set. The third part of your book is entirely about services and microservices.
We've already talked about this a little bit. Listeners probably already know the advantages of breaking your application up into different services. You can have them cleanly partitioned, you can scale these individual components that have significantly more load than other components. How does the approach to microservices that you've seen at New Relic, does that differ from the approach that you saw at Amazon?
I think so. And I think it mostly has to do with the origin of the services. In Amazon, they took the approach where they decided that we can't survive without services. So we're going to stop and only focus on converting over to service-oriented architecture and then we'll start innovating again.
So they did a concentrated effort to completely change everything over to SLA architecture and their new infrastructure and completely get rid of the old, what they call the Obidos large application model. They made that as a conscious decision to do in a sort of a period of time as possible so that they could get rid of the old and move on with them quickly. New Relic started down that path and decided that they were better off with a more longer-term approach. And so the approach that they've taken is as they develop new capabilities, they build them into services.
As they find that it is easier to deal with a piece of capability that's in the core application by moving it to a service, then moving it to a service. But the core application is still there and they haven't moved everything out of that core application yet. And so that difference in strategy really does impact the types of services you get and how you build them. One of the ways that it impacts that is Amazon was able to focus on core capabilities and core standardized capabilities sooner than New Relic is.
So in Amazon, you have a base user service and a base session service and some of those base capabilities, the low level infrastructure pieces to put a lot of that for New Relic and still within the core application. So I think that has affected, and that is not necessarily good or bad either way. They're different business approaches to the same problem, but it has affected the outcome as time gone on. Your book makes an effort to focus on timeless principles.
So it is not as focused as much on specific technologies, but what are the technologies that you're seeing evolve today that are having a real impact on how people are building microservices? Oh, Docker is a big one. Docker has really done wonders for the evolution of the DevOps model of building applications. And the DevOps model is really a central model for being able to build this service-oriented architecture.
The idea that a single team can build the capabilities to the point where it is a pluggable install item that can be dealt with by a core organization that doesn't have to have any business knowledge about your application at all. That capability is really key to moving from one or two or three applications to hundreds or thousands of services. And so I put Docker is really the number one technology today that has helped make this happen. I mentioned DevOps too, and that's not a technology as much as it is a cultural change.
I think that's a major component as well that's occurring now that has made this more possible and culturally within the company. And the DevOps thing is really an evolving definition. So the, the Docker side, there's also these container orchestration systems like Kubernetes. How do you think these systems, the amount of power that it puts into the hand of the operator, how does that change the narrative about DevOps?
Like does it broaden the scope of what DevOps is? Like what's your forecast for how these container orchestration services are going to shape things in the future? What I like about those orchestration services is it really, it helps clarify the role of the central operations organization more than anything else has previously. But the whole migration to DevOps culture, one of the questions that remains is what's the role of the main ops organization?
And I think Docker's line that it draws in the sand makes it easy to define what is in the Dev organization, a DevOps group. But it also helps define the roles that the operations organization can now perform and can now excel in. It's not the operations group can deal with things like, I have this many containers, I have to scale and do this number of things. How should I distribute them among the hardware that I have available to me?
And that's really what these orchestration systems are all about. But that Docker allows you to place that role completely and totally within an organization that now doesn't have to deal about the business impact of moving things around and can only deal with and can only and only has to worry about the cost benefit of moving something from one location to another or of using one technology for orchestration versus another one. And they can focus on their expertise and can focus on their capabilities without impacting the business itself. So there have been numerous discussions on this show about companies that build a monolith and the fact that they start with a monolith helps them work at a fast velocity early on.
And later on, they break it up into microservices over time by splitting off functionality or they take an approach that's similar to New Relic where you build a new service as a service and interacts with the old stuff and over time, it'll just your architectural trend towards services. But we did one show about Kubernetes recently where the Kubernetes evangelist who came on the show said that today we can just build microservices from day one. If I'm building a new application, I say this application is going to be great. This is going to run for five or 10 years or 20 years or 30 years.
It's going to be this long lasting company. And I'm not going to pay an extra cost to start with microservices from day one. I mean, do you think that's a promise that is going to be lived out? Is that actually going to happen where we maybe have the Ruby on rails of microservices and one click spin up a microservice architecture?
And it's no more difficult than starting with a monolith. Oh, I think we'll certainly have the Ruby on rails and microservices at some point in time. And I think that'll be hugely valuable to us as an industry to have that. But I guess I don't see that debate that's going on right now as much of a technology debate as a business debate.
When you're starting a new company, what's more important to you is getting something in your customer hands quickly, or is it building a foundation that allows you to grow for the long term? And in a lot of cases, in our industry is to former, but in a lot of industries, it's not necessarily the former, it might be the latter. And I think it really is a business question as far as whether or not it is important or not to build something quickly, but not necessarily scalable first versus something that is scalable and more foundational, but take a lot longer to make that happen. And I do think it'll always be harder.
I think we're always out of that statement. I think it is harder to build something that is microservice based than monolithic initially. And I think that's that's not going to change for a while. But at some point in time, as you build your application, the crossover does occur, whereas harder to continue to build in the monolith and it's much easier to build in microservices.
So it really becomes a business decision on where you want to put that investment is more important up front or more important later on. I want to talk a little bit more about the failure cases. So what are some best practices around architecting your services to be able to deal with failures, whether it's a partial failure or a complete failure? And also, if you're building a service that interacts with other services, how do you architect it to deal with upstream and downstream failure cases?
That's a good question. And actually, I have some information in the book that talks about that, but there's a lot of good sources on this as well, too. But basically, what you want to do is first of all, you need to define the SLAs so you know what the expectations are. But then once you have the SLAs defined, you also want to understand how I'll use the word important, how important you are and how important your dependencies are and how important the people that depend on you are.
And one of the things I talk about in a book that helps with that is the concept of service tiers. And this is actually, this isn't something I made up as concept came from Amazon. And I'm sure you've remembered from your Amazon days. But the idea is, I define multiple tiers, but like a tier one is a service that is business critical.
If that service goes down, your business is down. A tier two is important, but not quite business critical. Tier three, you can live without for a short period of time, but you want to get it up and running quickly, tier four, maybe you can live without for a longer period of time. And if you think about it, things like, if you look at an Amazon website, the gateway is a tier one service.
People want to be able to go into type in Amazon.com and get there and see a result and expect it to show something. And if it doesn't, they lose business quickly. Check out is another example of a tier one service at Amazon. Search is a tier two service, right?
You could live without search, but you really don't want to live without service. The site is still up and running if search isn't available, yet it has a huge impact in the business of searches and available. But for tier three, it might very well be a fulfillment center process where you're doing an automated inventory system, where yes, that's important to your business. But if it's down for a couple of hours, your business doesn't lose money and your customers aren't impacted.
And so it's different systems and different services that implement those systems that have different levels of responsibility. And it's important to understand what level your services and all your dependencies as well as people that depend on you. Because if you are calling a service that is more important to the business than your services, you can have a higher level of expectation that it is going to be running and not spend as much time. I'm worrying about dealing with it if that services down.
But if you are calling a service that is less important to the business in general than you, then you really have to worry about what happens that the service is down because you do not want that service going down to bring your service down. Okay, that's a great explanation. So, you know, as we get to close one interesting point you make in your book about the idea of buying managed services such as those from AWS that can get a query, a queuing service or you can imagine there's all these different platforms or service products you can get from AWS. And this build versus buy debate is quite interesting.
I find interesting at least if I'm architecting a system and I'm running on AWS, like how many, what components, how do I decide what to build and what to buy? And unfortunately, I think there's as much of a business decision as well as anything. Again, if you're building an application, if you're a small business that just started up and you're trying to build an application quickly and get something up and running and in your customers hands quickly, you know, use the managed services because you don't have an infrastructure within your company in order to deal with those things yet. Let someone else deal with that.
Just throw money at the issue and don't worry about the problem. If you're, you know, a larger corporation has a solid operations department with strong processes in place that have proven to help you over time, that last point being very important, then maybe you're better off managing yourself and more economical managing yourself. And it really depends on your business needs. I'm personally, I'm a strong proponent to using managed services because I would much rather put the job of managing my core resources like database, et cetera, into the hands of someone who knows how to deal with it and has a lot of experience doing it than the hands of someone who doesn't.
And so, you know, if I would much rather have Amazon manage my database, then my small startup managed my database. But, you know, I thought it was a larger company than something a little bit different. And in fact, if you look at company like Nurele, we manage a lot of our own databases ourselves. But there's some databases that we let Amazon and other providers manage for us.
It varies a lot. But, you know, in our case, our core databases that contain our core data that is critical for applications, we feel we really have the expertise to do that. And we have built an infrastructure for managing that and an ops team that knows how to manage those core databases. So in that respect, we feel we have more knowledge for how to handle our data in those cases than Amazon does.
And so, we consider that a value of our company. And so, you know, that's a business decision in that case. And it really does depend on the business needs is managing this important to you or not. So your book, Architecting for Scale, what did you have?
What were the things that you did not have time to fit into this book? What were the other topics that you wish you could have included? Oh, I wish I could have spent more time talking about cloud and migrating applications of the cloud and how to use the cloud effectively for building higher scale applications. I do have a section of talks about the cloud, but honestly, I would have liked to spend a lot more time talking about that.
That's probably the biggest point. Well, maybe that could be the sequel. Yes. Okay.
Well, Lee, thanks so much for coming on the show. This has been an awesome conversation. This time flew by. I really appreciate you coming on the show.
And thank you for your time. And thank you for inviting me on here. Thanks, Jeff.