Scalability in all its forms has been an active challenge for software projects since the earliest days of computing. Today, cloud computing services facilitate frictionless access to high availability systems. But that success can make the challenges of engineering a high availability system seem easier than they tend to be in practice. Lee Atkinson is the author of Architecting for Scale, How to Maintain High Availability and Manage Risk in the Cloud.
In this interview, we discuss a variety of challenges modern software engineering teams may face and strategies for resolving them. Lee, welcome to Software Engineering Daily. Thank you very much. It's great to be here.
So to kick things off, can you tell me a little bit about where you get your start in technology? Sure. Yeah, I've been in technology since, well, since 1987 is when I left the university and entered the technology market. I worked for Hewlett-Packard in Cupertino, California back when Silicon Valley was a little bit smaller than it is nowadays.
I stayed with HP for about 10 years doing test and measurement software systems. And then I left and went to my first startup, which went the way that startups usually go. And after a few years with them, I accepted the job with Amazon and moved to Seattle. So throughout the whole process, I went for the Bay Area to Philadelphia of all areas to Dallas and then finally ended up in the Seattle area, which is where I live now.
And I was with Amazon for about seven years, both in the retail side of the company as well as in AWS. And I did a couple of podcasts episodes just on War Stories from Amazon, but Amazon was a great company to work for. And then I moved on to New Relic, which was an analytics company that was just a fairly small startup company at the time I joined them. But I worked with them for about seven years in a variety of different roles, ending up being kind of an ambassador to talk to their customers and to within the marketing and sales organizations going around talking to customers and trying to sell as a brand associated with, I'm going to do that over again if you don't mind.
Oh, yeah, go for it. Okay. And finally ended up at New Relic, spent seven years at New Relic and doing a variety of things there all the way from a developer role all the way up to you at the end. When my book first came out, I went around on the road with New Relic talking to customers and doing presentations at conferences and things like that kind of promoting the New Relic brand by my book at the same time.
And is that architecting for scale, high availability for growing applications? Yep, that's right. That's the first edition of that book came out in 2016. And that's when I took on this new role at New Relic and started going around talking to people.
Well, I definitely want to get into the book, but one more sidetracked before we get there. I'm curious at the time you joined Amazon. At that point, where were they at in their cloud journey? Oh, that's a good question.
So, the time I joined Amazon, this was in 2005. So there was no AWS and it was just Amazon retail and Amazon retail was a monolithic application. It's called Oby-Dose. And my first job was I joined the team that was responsible for driving the migration of their website from a monolith to a service-oriented architecture.
So that's like jumping in headlong into a large project, just getting started. There's about 100 engineers involved in that project. But there were a small company at the time. The stock price was $30 instead of $3,000.
There's a lot of smaller organization. But I did that and I worked in the video game group for a while. I built the first software download store for them and created the group that ended up becoming the up story eventually. And then this upstart group within Amazon called AWS, which was just getting started at the time.
They called to me and asked me to move over to that side of the organization. So I moved over there and started working on Elastic Beanstalk, which was their first foray away from infrastructure as a service like EC2 and into platform as a service offerings, which was what Elastic Beanstalk ended up being. And so on. So did that for a number of years and seven years total at Amazon.
So was there until 2012, I believe, before I moved to New Relic. And at that point, is that when scale become a prominent feature of your career? Have that been there for much earlier times? No, absolutely.
That's when scale entered my vocabulary. You can't even when Amazon was smaller back in the 2005 days, everything you did was done at scale. It's an amazing mindset shift that occurs when you move to an organization like Amazon. That scale is such a central critical component of everything they do.
And so that's absolutely when scale entered my mindset when I started to think about scaling and scalability. And I didn't really ever consider myself an expert in that area until I moved to New Relic. And what happened when I moved to New Relic is I moved from this large company with huge scale to a small startup, but still had huge scaling issues. You know, they were at the time, like I said, they were about 100 people total in the company, small company, relatively small company compared to where they are now.
But they still had a big data intake pipeline for big for them anyway. And so they had scaling problems and we're just starting to go through the growing pains that I'd seen so many other times within various projects within Amazon. And I was watching them struggle with some of these growing pains. I said, well, I know how to do this.
I know how to help with this. I know what needs to happen here. And I realized that the things that I took for granted when I was at Amazon, the things that took for granted when I was at Amazon weren't things that everybody just knew. They were just, they were things that were very unique and very specialized skill sets.
And so I kind of led the effort within New Relic to help them improve their availability, which was based on scale issues and improve the scaling. And that's when I really focused a lot on the organizational changes, the strategies for building, scaling into the, the DNA of the company, if you will, and to help make that a mindset shift that occurs, which is really what you need in order to build a highly skilled company and highly skilled application. And that's like I built that mindset in. And that's when I started realizing that this is really unique information that would be valuable for other companies as well.
It's things that took for granted at Amazon, but really benefited New Relic. So that's when I wrote the book. So the lessons that are found in the book, are they for an executive who wants a scalable company, a director of technology who has to get the team on the right track or who's really the right audience? I think the main audience is the range from, you know, depending on the size of the organization, the range from manager, senior manager, all the way up to director, a hands on CTO in that sort of range.
So that range would be the ideal range for this book. It's not designed for the true executive, the CEO or the CTO of a large organization, nor is it designed for the new engineer that's just joining a company and trying to figure out what to do. It's really for that middle area. Well, scale is something that the cloud has been in some ways a solution for is it's going to become to the point where maybe I can just consider scalability a problem.
I'm my outsource that it's Amazon's problem and I just focus on my code. Are there critical lessons I'm going to need to learn? They're absolutely are. I think that's one of the biggest challenges that I face when I talk to companies and both when I was at New Relic as well as my consulting business now is the concept that scaling is a thing that you do and that you since it's something you can do, you can hire someone to do it for you and you don't have to think about it.
But scaling is really about a cultural change within the organization. The scaling and availability go together and those can't be solved with tasks. They're solved with the mental change occurs within an organization to change the mindset of the organization to be thinking about a scalable company or a scalable organization or a scalable application and a highly available application. Now, once you do that, the natural progression is the cloud becomes a tool that helps you accomplish your goals, but you can't just say, oh, I'm using the cloud.
So therefore, I solve my scaling problem. It's the chicken comes before the egg, you know, sort of thing. You need to be thinking about scalability and that leads you to the cloud, but the cloud doesn't solve all your problems. It just enables you to solve your problems.
What are some of the things that initially people tend to bump into as early growing pains? So most early scalability problems show up as availability, how to just brown out, it's black out, things like that. And usually they're caused from not thinking through ramifications of process decisions that they're doing. You know, for instance, you know, a smaller company will know how they deployment process.
It's less formalized and how they process that. That, you know, someone can make a change to a production system and not log the changes in the right systems that make sure other people know what things are going on. They're a lot less formalized and how they do things. And that's okay to a point.
And certainly this, you know, we can talk about agile and how agile fits into all of this. But the problem that happens is by not thinking through the problems, you start making stupid decisions or stupid mistakes. And the stupid mistakes end up being outages that you're not prepared for. You don't have the tools for.
You don't have the processes to solve quickly. And you don't think about them as problems in the same way. I'll say, I can't tell you how many companies I've talked to you that think about downtime as a kind of a throwaway concept. You know, that's we're going to launch a new version of the app and we'll just bring the app down for an hour in order to do that.
They just naturally think that that's an okay thing to do. But those are the sorts of decisions that sort of mindset that leads you into this trap that leads you down the path where as you scale, you start having more and more problems and more availability problems that turn into more and more scalability problems. Well, I'm thinking of a typical startup or the ones I've interacted with. I don't know that any of them have a good handle on what their availability is.
Are there best practices for establishing that or coming up with a true measurement of how many nines I can award myself? Lots of different things that come to mind there. One of the first things you want to do though is measure everything, right? And you know, that's of course I came from the background of a measurement company.
But even before then, I realized the reason why I joined New Relic is because they did this that sorts of things that companies really need to do in order to keep their applications going. But you don't need something like New Relic. You just need a way to monitor your application so that you know what the performance is. You can't solve problems.
You don't even know where your problems are unless you know what is currently available. So measure, measure, measure, measure first. And once you measure, then you start knowing where you're doing well, where you're not doing well, you know the places you need to invest and you know what to do to make that better. The next step beyond measurement is establishing baselines, establishing SLAs and understanding what you, you know, baselines are so you can see where you're improving and how you're improving.
And SLAs to establish what the minimum expectations are from one component in your system to another, which ultimately amounts to your customer expectations. I imagine this could vary a lot by organization, but are there any common metrics or measurements that most people are typically looking at? At the highest level, most measurements having to do with latency, with for spite of availability, those sorts of measurements are really the basic things that most people think about. When I think about measurements, I think a lot about time to resolution, you know, MTTR, MTTD, a mean time to resolution, mean time to detection.
How long does it take you to notice that there's a problem on your site? And then once you notice there's a problem on your site, how long does it take to resolve that issue and fix it and bring your application back up to your normal standard? Those are two measurements I think a lot of companies don't think too much about, especially in the early days, but really are critical ones to building a highly available, highly scalable website. And what's the share of responsibilities between a traditional software engineer and someone in more of a DevOps role?
Yeah, so I don't like separating the two out. I'm a big believer in the development organization is the one ultimately responsible for all aspects of the service. You've divided your application into services, you divided your, each of the services is owned by a particular development team. That team is responsible for all aspects of the service.
That's including, that's building the service, designing the service, architecting the service, and operating the service and up to and including on call for running and keeping the service going. Now, whether or not in order to accomplish that, that team is all engineers, all DevOps people, whatever the combination is, that's less critical. And in fact, I'm not a huge proponent of role division by I'm a developer versus I'm a DevOps person. I really like organizations that play those roles a lot more fluid.
Every developer should understand how to do the things that a DevOps engineer does. And a DevOps engineer really should understand the concerns and problems that a developer is running into. So I like running organizations where those are much more fluid concepts. Yeah, I've seen a lot of success with the do one thing, do it well concept.
Not so much for the professors, as you said, we're many hats, but in terms of the microservice. So having a small team that owns that end-to-end makes a lot of sense. But then people will say, well, now you have a problem of multiple microservices in the coordination of what happens when something fails. Are there extra challenges there as we scale?
Yeah, certainly. One of the problems I see some companies running into is deciding where to put service boundaries and how many services is the right number of services for this application. And sometimes companies, when they move from a monolith to service architecture, they go too far. They make too many services that are too small.
And what happens when you move to a service-based architecture is the individual services themselves, the smaller you make the services, the easier they are to support and the less cognitive load it takes in order to understand how the service works. But the more complex the interactions are between the services and the more cognitive load it takes to understand the overall system architecture and how things work together. So what tends to happen when you move your services smaller and smaller is you need less expertise and smaller teams to actually build and operate the services. But you put a greater load on your architectural team and the team that owns the interactions between the services and you create problems at that layer.
So it's kind of the sweet spot where your services are big enough that you don't create the cognitive load of the service interactions that occur, but small enough that each individual service isn't really just a mini monolith. And finding that sweet spot is kind of a, there's some trial and error to it, but there's some ways you can go over that. I talk about them in the book A Fair Mount about how do you size your services and where do you draw the service boundaries in order to make the load between what it takes to manage the services and what it takes to manage interactions between the services be the right balance for your organization. There's been a growing interest in serverless functionality, I guess, notably Lambda functions.
One of the things that's appealing to me about them is it's an opportunity to maybe focus a little bit more on my code, worry a little bit less about how it's going to be run and scaled up. How do you feel about that? Am I giving up too much in a typical Lambda environment or are things still manageable there in a responsible way? I think anyone who's who's read some of the things that I that I write will find that I'm not a huge fan of serverless.
I'm not a big advocate of serverless in general. And primarily because it forces and drives decisions, in my opinion, for the wrong reasons. A serverless architecture really dictates a style and architecture and a sizing of your services that I think is in general often too small for the services to be. You end up with this problem that I'm talking about where your individual services are a lot less complex and easier to manage.
And so the advantages of serverless work there, but you end up with substantially more services and a more complex interaction between them. And so you end up with taking that complexity and moving it out of the service and moving into an area that's a lot harder to deal with. And that tends to happen more with serverless architectures than they do with servers. You're going to have to deal with the issues around how your infrastructure supports your application, whether you're doing serverless or whether you're running on top of servers directly, dealing with it and deciding to deal with it and planning for it and building it into your processes and your decisions upfront gives you the flexibility where you can then make your code decisions based on where you want your code to be, not where the infrastructure requires it to be.
So that's one of the big things I have against serverless is that forcing a specific style and structure and size of services that isn't necessarily consistent with your application. Another problem with serverless is the inconsistency and how they operate in performance that goes along with it. Serverless functions really you have to deal with a unpredictable performance curve. And a lot of applications have a hard time dealing with that.
A lot of people don't think about that and a lot of people don't deal with that problem and that ends up showing up in performance issues later on in other aspects. And finally, I think people don't understand that serverless, even though it's less infrastructure presumably, that doesn't mean it's cheaper infrastructure. And I've seen many companies just go so overboard with serverless in areas where they really weren't designed to be used so that their cloud bill really shows the cost of that value in ways that weren't consistent with how they use that value. You see what I'm saying.
Yeah. Well, in contrast, could I get your thoughts on Kubernetes? Yes, I'm a huge Kubernetes fan. I've always been a big fan of containers as a way of delivering service technology into an operational environment.
And Kubernetes has been the godsend of how do you run containers in an operational environment in a positive way. That's my view. Now, there certainly were other technologies before Kubernetes kind of became the golden standard. And certainly, there's nothing wrong with things like Amazon Container Service.
But Kubernetes is kind of the is the gold standard, if you will, and Container Management. It does a great job with it. It's got a learning curve that's higher than you would want it to be. But it's well worth it.
And you end up with a solid infrastructure that's easy to deploy new capabilities into. Well, if I were personally responsible for scaling up and maintaining high availability on some system, I would definitely be concerned with how good my source code is, that I'm deploying the right services, doing everything in-house correctly. I also, it seems, might have to worry about my cloud provider. We've seen some big outages in that regard.
What's a good way to split my time between worrying about my house and my provider? That's a good question. I'd still let's look at the cloud providers and the outages as I've occurred recently. And certainly, I think one of the problems with the cloud provider outages that occurs is that they get a high level of visibility.
And certainly, if you let's take out this last round of outages that have occurred recently, I think it's fair to say that the outages tend to be outweighed by publicity than by real impact. Now, the last rounds were pretty bad and they may change the scales a little bit here. But I think a application running on a cloud provider versus an application running on-premise environment and how often the infrastructure causes problems that cause the application to have a problem in those two environments, I think you would find that cloud providers are very, very, very stable and very high quality and very, very performant, et cetera. And that the vast majority of outages, I mean, from a percentage standpoint, you're going to get a lot more outages when you're running on your own data center in your own environments.
And I think even the question is what this last set of outages does that change that ratio any? I don't think it does, but I haven't thought far enough along yet to know whether that's really changed things or not. Bottom line is, the when a cloud provider goes down, it affects a larger group of people. And so you hear about it in the news, but it's not necessarily for an individual company and individual application, any worse of an outage than they're currently used to if they're not using a cloud provider and running in their own data centers.
So I don't really see the outages that have been occurring as problematic, just more of a reality of running a highly scaled application. I still trust AWS to run a high scale website. Well, with if you're someone who has the opportunity to work on something that could have unpredictable spikes and traffic and need to be available and grow, there's just a certain degree of uncertainty. And the only way I know to manage that is to have smart planning and risk assessment.
So no surprise that risk management is a major section of the book. It's not something that appears in a lot of computer science curricula, however, so I think it's something a lot of software engineers can pay lip service to, but don't really know where to begin from a quantitative sort of perspective. Can you share some ideas about how we frame and measure risk? Yeah, yeah.
So I strongly recommend that each individual service team builds and develops their own risk matrix for the operation of their service. And what a risk matrix is, is it's a list of known vulnerabilities, known problems, known things that you think might be a problem or you know are a problem, and grade them by using two criteria. One is the likelihood that this risk will happen and the severity of this risk if it does happen. So the likelihood is how, you know, what's the percent chance of this thing happening on some sort of scale?
And then the criticalness is if this does happen, how major of a problem does this occur for you? What you find is there's a lot of risks that have high likelihood and they're very likely to occur, but the seriousness of the risk is not very high. And there's other ones where the likelihood or it was very, very serious that this problem occurs, but the likelihood of them occurring is very low. But then you'll find there's other risks that are not very likely to happen and aren't very serious if they do happen.
Those are the ones you can simply ignore. But then there's the ones that are likely to happen and are very severe when they do happen. And those are the ones you end up paying the most attention to. So anyway, you brainstorm this list, you keep the list up to date as new vulnerabilities become known to you, you add them to your list, this becomes your gold standard for the risk that's built into your system.
And you associate these two variables, likelihood and severity, to each item in that risk. And then you track everything using that. You focus on the ones that are the high highs, the high severity and high likelihoods, you ignore the lower ones, and you use this as a tool to help prioritize the work that you do to improve the availability and quality of your service. Then individual service owners should share these matrices with their dependencies as well as up the management chain so they can combine them and create application level or service or component levels or company levels risk matrices that contain all the information about everything that's going on within the organization.
So the risk matrix becomes a way of communicating technical debt up and down the management chain as well as with your dependencies. So they become a communication tool that allows you to share where you are in your organization. And what's important is when you come to trying to schedule a project now and you run into issues where management wants to done this day and you say it's going to take longer and those sorts of problems, you can use the risk matrix as a tool to try and show you. Look, we have these risk items that we need to resolve before we can fix this or before we can do this or that sort of thing.
It becomes a tool that can help you with your communications to understand and what management make decisions like, well, it's okay for us to have these risks. We understand what these risks are. So we are willing to trade these risks off for these other things that we want to have done. So it becomes a tool that allows you to talk technical language and management language and have every business language and have everyone understand what they mean by these risk items that we're talking about.
Well, earlier on, we've been following the threat of your career. And I think we left it off while you were at New Relic. The book has come out. You're going around touring a bit about it, then decide to go off on your own.
That's a risk to be taking and some risk to be managed. What made the timing right for that to be going on a new venture at that point? Well, a number of personal things were going on as well as some changes within the company that were going on within New Relic that were going on. And ultimately what finally decided is, so the timing was between some personal items, things that were going on, some changes at the company that were going on.
And then the start of the pandemic, which basically meant it was very hard for me to go and talk to customers now and do a lot of that made me realize it was a good time to be thinking about doing something different. And so New Relic and I came to an agreement, I left New Relic and started my own consulting company. And that was now, let's see, it was May of 2020, 2019, I'm sorry. So this is pre-pandemic?
Well, 2020, so May of 2020, sorry. So it's going on two years now, it's been about a year and a half. And I've been doing going like inkbusters ever since and really enjoyed, it is a risk and it's a risk that you wonder if you can ever do that risk anytime in your life. And it was probably a time in my life when I could afford to take a little bit more risk, you know, the kids were out of school, you know, they're living on their own, it was just my wife and I at home.
So you know, we didn't have the family issues to worry about. So it's a good timing from that standpoint. So decided just to go off and make the change. And it's been a great positive for me.
I've been very happy with what's been going on. I've got some good clients right at the very beginning, which gave me the motivation and I've been growing and going like inkbusters. I think recently I've started to make a change where I've been focusing less on individual clients, you know, where someone will bring me in and use my expertise to help them. And I've been doing more educational outreach.
I've been, you know, I did the second edition of the book, I'm working on another book, I did a book for Redis Labs on on caching and working on another book for them. I did a series of courses for LinkedIn Learning and a bunch of other content output like that. I've been focusing a lot more on doing that and, you know, and selling that content. So I'm not sure, you know, in the future how much of my time will be traditional consulting versus, you know, push education.
But some ratio between, you know, all or nothing on one of the others of those, I'll be doing some combination of both of those, I'm sure. Well, the book's now in its second edition. Are there any notable changes from edition one to two? Yeah, it was actually a fair amount of change I went on there.
So so what happened was the first edition of the book came out in 2016. And that's when I started my new role at New Relic, where I went out and I, you know, would be the promotional engine going talking to customers and doing trade shows and things like that. So I ended up talking to an awful lot of customers, you know, awful lot of situations and trying to help with the problems they were running into. And so I've learned a lot after the book came out about how other customers were going through the same problems that I saw New Relic and Amazon and other places.
And so I used that material that had developed over the course of now several years of doing that from 2016 to 2019. And that's when I decided to do a second edition of the book. So the second edition of the book came out in March of 2020. And basically, it's a complete reorganization.
It also has about 30 to 40% more information, as far as by page found and that sort of thing. And so there's a lot more information in it. And it's radically reorganized into topics that I think are will make it easier to read and easier to understand by individuals. So there's basically dividing the five tenets that I talk about.
I talk about availability, talk about service or any architectures. I talk about organizations, you know, people management, building organizations that are designed for scalability. I talk about risk management. And then I talk about the cloud.
Actually, the cloud was a brand new section that was added to the book as well. And it's a bigger focus on how technology in the last several years. So a lot of changes that occurred. And response has been very, very positive.
I've gotten lots of people who've bought the first edition and bought the second edition and said it's like night and day comparison between them. So it's a I think it was a good upgrade and it has a lot of good material for people who really are needing to solve these problems. Well, before we wrap up and send listeners on some links or whatever we want to do there, are there any particular trends or technologies you're following and excited to see going on here in 2022? Yeah, I'm very excited about citizen developers and about the trends that are going on in that area.
I'm also rather intrigued, but not quite sure where it's going in the idea of AI assistant development and some of the technologies I'm, you know, of course, AI in general is a big area. I have T is a huge interest of mine. But I think the general areas of citizen development and AI assisted development are two areas that I'm in particularly intrigued about. Very cool.
And we touched on a number of educational solutions you're coming up with. Where's the best place for listeners to learn more? Best way is to go to leaterson.com, L-E-A-T-C-H-I-S-O-N.com. You'll see my books there and links to where to get them, as well as the courses I've done.
Right now, there's four LinkedIn courses plus a couple other courses. And then my O'Reilly book, my Redis book is there and a couple other, I've been contributors and a couple other O'Reilly publications as well and they're all listed there as well. Well, Lisa, thanks so much for coming on Software Engineering Daily. Thank you very much for having me.