There is a quote from Jeff Bezos. 70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.
That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had no choice but to manage your own database, your own data warehouse, your own search cluster. If your server crashed in the middle of the night, you had to wake up and fix it. And you had to deal with those engineering problems in addition to building your business.
Technology today evolves much faster than in 2006. And that's partly because managed cloud services make operating a software company so much smoother. You can build faster, you can iterate faster, and there are fewer outages. If you are an insurance company, or a t-shirt manufacturing company, or an online education platform, software engineering is undifferentiated heavy lifting.
Your customers are not paying you for your expertise in databases, or your ability to configure load balancers. As a business, you should be focused on what the customers are paying you for, and spending the minimal amount of time on rebuilding software that is available as a commodity cloud service. Rich Archibald is the director of engineering at Intercom, a rapidly growing software company that allows for communication between customers and businesses. At Intercom, the engineering teams have adopted a philosophy called run less software.
Running less software means reducing choices among engineering teams, and standardizing on technologies wherever possible. When Intercom was in its early days, the systems were more heterogeneous. Different teams could choose whatever relational database they wanted, MySQL or Postgres. They could choose whatever key value store they were most comfortable with.
The downside of all this choice was that engineers who moved from one team to another might not know how to use the tools at the new team that they were moving to. If your previous team used MySQL and the new one uses Postgres, well, you're going to have to figure out how to onboard with Postgres. And that onboarding process is time that's not spent impacting the business directly. By reducing the number of different choices that engineering teams have, and opting for managed services wherever possible, Intercom ships code at an extremely fast pace, with very few outages.
In our conversation, Rich contrasts his experience at Intercom with his experiences working at Amazon Web Services and Facebook. Amazon and Facebook were built any time when there was not a wealth of managed services to choose from. And this discussion was a reminder of how much software engineering has changed because of cloud computing and the managed services built on top of raw cloud computing. To learn more about Intercom, you can check out the fantastic Inside Intercom podcast.
It's a great window into how Intercom works and how they think about building products. Rich Archibald is the Director of Engineering at Intercom. Rich, welcome to Software Engineering Daily. Thanks very much.
It's great to be here. Today we're going to talk about a talk that you have given at several conferences, which is called Run Less Software. And we're going to talk about it in the context of Intercom, which is a very popular software product, and also some of the experiences that you've had in prior roles at tech companies. Since the focus is Intercom, and Intercom is a really detailed product, let's start off by giving people a perspective for what Intercom is.
Could you just describe the Intercom product and how it manifests for users? Sure. Intercom is a messaging platform that helps businesses connect with their customers. So you can think of Intercom kind of like Slack or WhatsApp, only for businesses to use to help them have conversations with their customers.
Intercom's mission, like Intercom's very much a mission-driven company, and our mission is to make business personal. We think that a lot of the ways that businesses actually communicate with their customers today is kind of like old, janky, stiff, kind of spammy status quo, and we would love to give customers the ability to, or we would love to give companies the ability to have really meaningful conversations with their customers and help them build real kind of relationships with them. For people who have not seen Intercom or are not sure if they've seen Intercom, sometimes if you go to a web page and you see a little chat icon in the lower right that is on the landing page interface, you can click on that and you can interface with people directly at the company. Oftentimes that is Intercom.
There are different companies that provide that, but Intercom provides, I think, the most popular one or, in my opinion, the one with the best UI. So how would you describe the engineering culture at Intercom? That's a really interesting question. I would describe it as a very product-first engineering culture, if that makes sense.
So it's not engineering for engineering's sake. It's not fond of a lot of buzzwords or, like, it's not fond of things like, we're going to do service-oriented architecture because that is what everybody says you should do. Everything is very much grounded in what do customers need, what do we need to do in order to build the best product for our customers, how do we think about identifying all of the most important problems, how do we make sure that we understand all of those problems from first principles before starting to write code. So I think we spend an awful lot more time understanding the why and the what before actually trying to build code or solutions.
One of the other things that I think is really interesting about Intercom's engineering culture, again, in that kind of product-first, product-thinking kind of a way, is we have a very high amount of collaboration with all of the other kind of dependent disciplines involved in software development. So we work very, very hand-in-glove with our product management teams, with our research teams, with our design teams, analytics teams. Everybody sits together in the one pod. Everybody goes to the same stand-up.
And I guess that product-first, product-thinking is probably the thing I think that shines through the most. Probably the next most interesting element of the culture is the pace at which we move. And I know a lot of startups move quickly. At Facebook, we used to deploy code, new versions of Facebook, twice a day.
I thought this was really fast. I thought it was fantastic. I came to Intercom and found we deployed new code 70 to 100 times a day. At Facebook, a code push took an hour to two hours.
At Intercom, it takes 10 to 15 minutes, fully automated, push on green. Almost every single merge to master results in its own deployed production. And so that kind of pace of development, that kind of philosophy of shipping is your company's heartbeat, that is really interesting to me. And it is probably one of the things I'm most proud of about Intercom's culture, which has stayed very strong and ever-present over the six and a half years or so that we've actually been around.
And as you can imagine, as you get bigger and bigger and bigger, and as you add more and more people, and as you add more and more lines of code, the obvious thing to happen is that your software development process will slow down, you will deploy less often, you will become more cautious. And I'm really proud of the fact that we have been able to save fast and save fast and save fast and also still achieve really good stability and have really good availability, really good scalability, really good efficiency while actually doing that. Yeah. Well, I think there's a direct connection between the fact that you're a product-focused company, but you also emphasize choosing standard technologies.
You even go to all as far as sometimes referred to as boring technologies. And that's part of it because the boring technologies these days are really, really good. If you look at the buffet of options you have available on AWS, you've got a ton of stuff that you can pick from, and you don't really need to reinvent the wheel most of the time. You're already introducing enough complexity, building your own product for your customers.
You don't really need to introduce more complexity by getting fancy if all the ingredients that you need for your recipe are already there. There's actually a Jeff Bezos quote that you like to reference. 70% of building a business is undifferentiated heavy lifting, and 30% is creative work. And things will be more exciting when those numbers are inverted.
So we obviously like to get to a place where 70% of building a business is fun, creative work, and only 30% is the building, you know, piecing the building blocks together. So how does that quote reflect your thinking at Intercom? I think it comes very much back to this product-first, customer-first thinking. And any time we sit down to think about a problem we are going to try and solve for customers, or a feature we are going to try and build, and any time we use kind of customer language, we are never using MySQL or Postgres or MongoDB or DynamoDB.
We are talking about things like reliability, trust, privacy, and functionality, and solving problems. And so that actually language kind of really leads you to realize that database technology is not your competitive advantage. Whether you choose Postgres or MySQL or MongoDB or DynamoDB is really irrelevant to your customers. What is actually relevant to them is how fast you can get a really good product to them, and how quickly you can evolve it as you learn how to make it better once it's in their hands.
And that, you know, that has kind of really led us down that path of choosing these standard technologies and moving towards this philosophy of run less software, which is really all about saving time, moving faster, and enabling us to build better software, build and maintain better software more quickly, cheaply, and easily. You gave the specific example of being able to ship something like 70 times a day. How does picking standard, quote-unquote, boring software, how does that lead to a faster release schedule? I guess what it has allowed us to do is any time somebody is trying to build a feature or figure out how to evolve a feature, a bunch of decisions have almost been made for them more easily.
So if this is a feature which is going to require a database, it is absolutely going to be built on top of AWS, Aurora, MySQL. If it's a feature which requires a key value store, it's absolutely going to be built on top of AWS, DynamoDB. So a lot of the standard building blocks, the decisions are already made for you, and because they are already made for you, the odds are you have been using these technologies over and over again. They are probably really well supported with libraries, monitoring, documentation, everything inside of Intercom, so that they are really easy to use, really easy to understand, and therefore easy to make new software with quickly.
I guess one of the other things about it is we talk a lot in the industry these days about how talent is scarce. I read this fantastic, I think it was a US government study that showed that there were three times more software jobs advertised than there were hires made over the past year. And engineering resources, like engineers, it is just a war for talent out there. It's really hard to hire people.
It's really hard to retain people. Everybody has open positions, and everybody's actually trying to make the most out of the software engineers they have. And we feel by using all of these standard technologies that are primarily outsourced, primarily outsourced to a wonderfully trustable company like AWS, who is innovating so fast, we actually get to run this really big, scaling, very, very fast back-end infrastructure with a relatively few amount of engineers, which means we can have more and more engineers focused on our customer-facing challenges building new products and features. And that also allows us to ship more quickly, I guess, relative to if we have to spend more of our precious engineering time doing database upgrades or figuring out how to monitor or instrument loads of different back-end technologies.
Let's take a specific example. You referred to this just now. There was a time when you were running your own Mongo instance, and you eventually moved Mongo to entirely AWS Aurora and DynamoDB to scale the user storage. What were the maintenance issues of running your own Mongo, and what were the use cases within Mongo where some of them went to AWS Aurora, some of them went to DynamoDB?
Great question. One of the, I guess, USP features of Intercom is that we allow you to store almost an infinite amount of schemeless data about your users on our platform. So anybody who starts using Intercom, whether you are a free or a paid user, can start sending us thousands of things that we call custom data attributes, and you can store integers, strings, arrays, whatever you want. You can change schema types seamlessly without doing anything, without actually telling us anything.
And this is a really, really powerful feature for customers to use, but it's also a very hard feature for us to maintain and scale on the back-end. And when we started building Intercom, we had this feature running on a single MongoDB replica set. As we scaled, we started to do manual sharding of our MongoDB replica sets. MongoDB has a max index size.
You can index a max of, I think it was like 64 columns or such within their indexes. We found TCP connection pooling and scaling difficult to deal with, so we started to have tens of thousands of TCP connections opening and closing to our MongoDB replica sets. Every time we deployed new code, we were seeing lots of CPU churn with this, and we were seeing disk space issues. We were seeing expensive queries threatening our database.
It was expensive to run. We were just having lots and lots of problems with it. We also started to use Devara as a TCP proxy. So Devara is a technology that was developed by Parse.
It's written in Go. And so this was like this extra complexity we now had to deploy in front of MongoDB. And all of this was probably leading to some of our most difficult scaling challenges. Our teams were getting paged quite a lot at night, and it was just a pretty kind of tough situation to be in probably about 18 months ago, I'd say.
And so what about the selection of technologies to replace Mongo? How did you come to that conclusion? How we came to that one was, I think it was really great. I think it was Intercom taking a product-first approach and really trying to understand our problem from first principles.
Our CTO kind of stepped back from the chaos of all the firefighting and took a long look at the stack and the features and everything that had evolved and the system that we had today and decided, do we need to keep it the way it is? Are all of these features still relevant to the customers first? Is there a way that we can simplify the problem by maybe cutting features or cutting scope? And we did a tiny little bit of that, but really decided that the core of everything that we were providing was important and we needed to find a way to scale it appropriately and find the right cost and the right type of operability for the solution.
And we looked at a bunch of different ways in which we might solve this problem. I guess first we looked at MongoDB because that was the one we were on. We tried to figure out if we went to a natively sharded MongoDB, whether or not this would solve our problems. And we reckoned it might, but we didn't think MongoDB was ever going to run natively on AWS as an AWS service.
And by now we had a really, really strong reference to using AWS, native AWS infrastructure as a service or platform as a service. So while we thought it was possible to solve it with MongoDB, we would much rather use an AWS service. And our next favorite AWS service is probably AWS Aurora. And so we tried modeling all of our user service on AWS Aurora.
Aurora, like any MySQL database, still only has a single host scaling point. So we reckoned while we could do it all on MySQL, we would eventually run into scaling bottlenecks and need to shard our MySQL cluster. And sharding MySQL clusters wasn't something we really, really wanted to do so early in the lifecycle of this new iteration of a solution. And so then we tried on our next favorite AWS solution, which is AWS DynamoDB, and we tried modeling all the data there.
And that had nice seamless properties that allowed us to add and remove new custom data after we call it really easily. It was natively scalable. It was cheap. But the problem with DynamoDB was it didn't support multi-value primary keys.
And each element of customer's customer data has a three-part... primary key which is the app id customer id and email address and so we would have to do some crazy concatenated string primary key and so we decided that wouldn't work but then but then one of these one of the ways we think about choosing standard technologies and we think part of the art and magic to it is being able to take a complex problem and break it down into pieces which look like they would start to fit into some of your standard building blocks and so we were able to separate the problem of the multi-part primary keys and model that nicely in aurora and we were able to take out the part which is the schema list data and put that in dynamo db and write a little bit of ruby that would help us knit both of those two things together in a higher level library and we got this lovely solution which allows us to use the best of both worlds and allows us to use aurora for the thing it does best and dynamo for the thing it does best and that has been fantastic for us we've seen way way better availability we are still in the process of fully rolling it out we're out for a significant portion of our apps but so far that we are expecting somewhere in the order of an overall 90 percent cost production we've had no availability issues we now we now have a solution which is natively scalable and it is basically allowing us to move from something which was about three to four engineers worth of a constant operation and constant worry and for data that was sitting outside of our vpc to data that is now 100 inside of our vpc 100 on native aws services where all of the like main bulky undifferentiated operations tasks are now handled by aws for us and no more of the mongo maintenance issues thankfully hopefully and like i want to give out about mongo db i think it's an absolutely fantastic piece of software and i think there's actually there's absolutely no way intercom would be where it was today without mongo db i think this thing about choosing standard technologies is really interesting and opinionated and when i give the talk one of the things i talk about is how intercom standard technologies are intercoms and they fit us and they fit our experience and fit our engineers but mongo db could be perfect standard technology for a company who has a slightly different use case to us right makes sense another example you've talked about consolidating all the relational databases at intercom to amazon rds is another instance of standardizing on amazon technology but before that your effort was actually divided between administration of two flavors of aws rds you had mysql and postgres these are both relational databases seems standardized enough why did you need to move everything to mysql why did you have to get rid of the postgres aws rds that is an interesting question i think it's even more interesting to ask why did we end up with mysql and postgres to begin with okay sure and i don't think there's a good answer to this other than we didn't have we didn't have a standard we were a fast-moving startup who had hired a bunch of people who had preferences and experience with a bunch of technologies and people were like building new features and using whatever they were comfortable with and we ended up with a bunch of with most of our features built on top of mysql and a bunch of other features built on built on top of postgres we ended up with probably 90 of our engineers really comfortable with mysql and a very small number of our engineers comfortable administering post postgres databases we had a couple of like annoying outages short outages where one of our postgres databases had failed and maybe we hadn't noticed it because the majority of our operational tooling was built around mysql and while it wasn't a huge pain for us it was this constant little niggle of outages or expensive operations where you are trying to figure out how to cleanly change a setting or deal with some performance tuning on postgres and so we had this off site probably about six months after i joined the team where we were trying to figure out how can we save time how could how can we make things run a little bit better a little bit easier remove a bunch of cognitive load and this is where this kind of runless software idea originated from and i guess the most immediate low-hanging fruit that was uncontroversial was let's take the time now and invest and move away from postgres to the thing that we know and love mysql and that further and that further incentivized us then to let's actually start bringing in some mysql training for everybody else in the company so maybe anybody who wasn't as familiar with mysql we brought in a bunch of mysql consultants and got them to do a bunch of training with us in order to help upscale anybody who was maybe maybe would have preferred postgres and now feels a little bit more forced into using mysql that's great there was an example you gave where you could not outsource the technology and that was the example of elastic search you have to manage your own elastic search on bare metal why is that so why is that so we we started using elastic search when it was still a relatively young technology there wasn't anybody who was doing outsourced elastic search at the time and that and that is why we started doing it and as we started to do it we invested in a bunch of training we got some elastic search consultancy we sent some people over to amsterdam to do some awr to do some elastic search training courses and we decided if this is something we are going to have to run ourselves we are going to do it the right way and learn about it and become as close to experts as we can in it so i guess that was the reason why we started doing it over time a bunch of different companies have brought out hosted elastic search products elastico have their own um hosted elastic search product aws has one as well i think there's probably some other companies who do it now as well but you were just too early yes we were too early but about every six months we revisit this and decide hey is now the time for us to try and outsource elastic search and the honest truth is the aws product i believe is nowhere near good enough and nowhere near stable enough for somebody who runs an elastic search cluster of our size and some of the other elastic search providers in order to use them it would require us sending customer data outside of our vpc which is something i'm super uncomfortable doing with from a security and privacy and gpr perspective don't you have to do that if you're storing customer data in aws databases you do to a certain extent but aws has so much certification and so much security that it is still a very easy and tight and safe security story that we can tell ourselves and tell our customers having worked inside of aws myself for nearly eight years i know the lengths to which the lengths and maturity to which aws goes in order to safeguard all this customer data and i guess i'm a little bit of an aws fan fan boy having worked there for so long i kind of really really believe in how well the company thinks about security and thinks and thinks about privacy so i'm just i'm just willing to put an awful lot of trust in them to be honest yeah trust is hard won and easily lost as bezos likes to say you know we're talking about all these managed services and it makes me wonder is cost ever a concern here when you're even when you're just going whole hog into all these aws services it depends on what you mean by cost and what you mean by expense and what you think are your commodities and what your constraints are i can tell you that from my perspective the most things i worry about spending most of the things like the things that i think are the most expensive to get and you need to be the most careful how you spend them is engineering time because i think that because i think that is your most precious resource i think money is relatively easy to get and interest rates are at an all-time low i think the stock market is high i think people i think investors are incentivized to put their money anywhere other than a bank so i think if you are a fast-growing credible successful startup it's relatively easy to get money i think it's easier to get money than it is to hire a bunch of really great software developers and so that's actually kind of the biggest expense i think about but also i mean aws over time consistently their prices go down rather than up and so by betting on aws over time we automatically know things are bound to get cheaper other things by using more of aws things also get cheaper the more you're willing to reserve or whatever front the more comfortable if you are the more comfortable you are with them the bigger discounts they have everybody knows amazon runs on like very very small margins relative to bigger companies so it's hard to believe somebody else is going to be able to do it sustainably for the long term more cheaply than than aws so i guess there's a bunch of opinions in there i'm not going to talk too much about company financials but we're still healthy so like honestly i think engineer time is the biggest cost you need to worry about and engineer efficiency rather than raw dollar costs but even even if you are wondering about and worrying about raw dollar costs i still think investing in aws for your undifferentiated happy lifting is still the right way to go and the pace to keep the pace of the company as fast as possible now it's engineering time cutting down engineering uh time spent on managing infrastructure is going to make the company go at a faster pace but you really emphasize this in at least in the instance of the talk that i saw your sense of paranoia and how you think about competition you actually let off your talk with this discussion competition and how you could imagine a world in which slack or facebook come out with things that are competitive pressures against intercom and i can imagine that affects the pace at which you feel you need to sprint as as a company the messaging communications uh company how does that affect your engineering management and your choice of software it affects it a lot and one of the things i like to do is compare and contrast my time at amazon facebook and intercom into into a couple of phrases and i think of my time at amazon like when i was at amazon i think of i describe amazon as like high quality low cost and kind of a frugal environment and i describe facebook as kind of innovation focused fast moving and generous towards their employees i think of then intercom as like product focused dizzyingly fast like crazily fast moving and kind towards its employees and that kind of a dizzily fast moving the feeling that life is short time time is precious opportunities are fleeting now is the time you need to make the most of this opportunity it is kind of really this feeling that kind of permeates right right away throughout the company and leads to this feeling of let's actually move as fast as we can let's be aware that we are we are on the wave of something we are riding the quest or we are riding the crest of this like consumer communications revolution almost where companies want to talk to their customers like real people and build real relationships with them intercom was the first to come out with an in-app messenger we were the first ones to have a single integrated platform for sales marketing product management customer support to all talk to their customers in the same way using the same platform we have this advantage let's keep going let's not actually get caught by any of our competitors and so if you think like that you just really want to move as fast as you possibly can you have all these great ideas and you are a little bit afraid that your r&d capacity your own ability to build software and build great software and learn from it and build some more that is really your own competitor you're actually competing against yourself you have the opportunity to win if you can only just keep moving fast enough and so like that feeling is like definitely part of it i think part of the other part of it for me is i didn't start out my life as a director of engineering i started out my career as a sysadmin and like back 15 years ago when i was a sysadmin uh sysadmin meant racking servers cabling them imaging them recompiling kernels uh a bunch of really really low bunch of really really low level work and over time i've seen that work be abstracted commoditized um eradicated i guess a little bit by aws and infrastructure service and this feeling of the industry is moving really really fast around me and a lot of things that were once valuable have now been commoditized and abstracted away and we need to keep all of us need to be aware of that and keep moving up and up and up the stack and saying focus on what customers need rather than what is technology of the day and we need to keep moving with that and rolling with that i guess it's a you might know i worked on amazon briefly as well so i saw that frugality firsthand where you know the company just really keeps costs low why is amazon so unique there why why haven't other people instituted the frugality as a virtue did it just have to do with the fact that amazon started when there were fewer options for companies to work at and so they were able to say look we're not gonna give you like massive free lunch we're not gonna give you tons of perks we're gonna give a decent salary we're gonna be inspiring vision like how how do they do that i'm sure you've reflected on that i certainly have so i think it's unique in one way i think it's like really unique and opinionated and i think people are always drawn to things that are different and opinionated and anti-status quo so i think that is kind of one thing it has going for it i think the other thing it has going for it is i think people follow leaders and people want to work for people who have a really great vision i think people want to work for mission-driven companies i think and i think amazon has all of that in spades i think jeff bezos is one of the visionaries of our time amazon is a very values-driven company i think it has a it has a very clear mission so i think that helps i think there's a lot of ridiculously smart people there as well and the absolute best perk that you can ever have is working with world-class people i think there's and i think there's still a lot of amazingly world-class people at amazon i think the last thing if i'm maybe a little bit more practical is the stock price certainly i know over the last four or five years amazon has had an average of i think 41 stock price growth over the last four or five years it's like fast it's fast approaching being one of the world's first trillion dollar businesses and so i think that the fact that a large part of amazon's of amazon's employees compensation is stock-based or or su-based and you can see this compounding 41 growth year after year i think all of those things together provide like a really compelling reason to work there or stay working there when you were working at aws did you did you have any ideas around this run less software idea that i mean when i think about working amazon that doesn't really seem like uh i can't recall anything similar to a principle like run less software but amazon's a very different company yeah i i actually don't think we had it at amazon or certainly where or certainly where i was inside of aws we were we were kind of operating below the hypervisor so us using services like aws in order to build our own services would have been creating circular dependencies and that was certainly very much frowned upon we were all about considering ourselves like a utility understanding all of our single points of failure and making sure that we had absolutely full control over them so we as a team had full control over them i think that started to change over time though and i think that because when i see things like dynamo gb having an outage it seems like half of aws goes down now or when s or when s3 goes down half of aws goes down so i'm i'm thinking that maybe they started to have more of a change in their philosophy internally there where they are now starting to run less software for one of the better word and start to use more of their own technologies and just accept that when dynamo db goes down or when s3 goes down it's effectively going to be an aws region-wide outage but the benefit of that is that each of the individual services who depend on dynamo or s3 are going to become more available because they don't have to run their own key value stores. They don't have to run their own MySQL databases or things like that. That would be my guess.
Does it worry you that so many people are on AWS at this point? And it is, it's like a utility, it's like a multinational utility, it's like a global utility. I mean, I sometimes am a little worried about it, especially when you have these outages, and I know how trustworthy Amazon is also, but these things just happen. You have long tail events and they happen, and it's kind of scary.
I guess it's kind of like the banking system, though. Yeah, I think it is and it isn't scary. And part of the reasons why I think it isn't scary is that if you are in US East 1, if you are based in US East 1 where I think half the internet is basically based at this stage and there is a US East 1 outage, it's effectively like the internet is down. So even though you are down, also all of your customers are down as well.
So almost like people don't know that you are down because all of your customers are down. So that's kind of one reason that I'm not afraid of it. But the other reason that I am afraid of it is that we provide customer communication software and customer messaging software. So the times when customers are going to want to talk to businesses or businesses are going to talk to customers is pretty acute in terms of outage.
And so I really, I really, really want to be able to find a way for Intercom to be able to operate in that environment. And so whether it's moving out of US East 1 or running across multiple regions, it's definitely something which is on my mind and I would love to figure out a way to fix over the coming 6 to 12 months. So when we're talking about running less software, how does that apply to, I don't want to say service-oriented architecture because you already kind of pooh-poohed that name a little bit earlier, but I imagine you do have some distributed set of services. Maybe it's on Kubernetes, maybe it's on Amazon ECS.
How does that look? How does the service ownership and deployment process look? How does it look? How do we think about that?
So we definitely do have a bunch of discrete services inside of Intercom. Maybe not when people think about service-oriented architecture. I think about things being able to fail independently of each other or having really, really strong versioning or not so much tight coupling between different versions of different services. And so I don't think we are all the way towards service-oriented architecture or microservices as the maybe Wikipedia definition of them describes, but we absolutely do have a bunch of distributed services.
And the way I think about running less software, the way it helps this is that there are common building blocks across all of these different services. And so we are doing CRUD-like data storage in the same way. We are doing key-value data storage in the same languages. We are absolutely monitoring it in the same way.
We are deploying it in the same way. And what this means is that it makes it really easy for engineers to switch from one team to another and get up to speed really, really quickly. And so we actually try and have this run less software, maybe even use less processes or use standard processes. So I would go so far as to say each team would do planning in the same way.
They would do planning at the same time. They would do retrospectives in the same way. They would phrase all of their goals in the same way. They would write all of their roadmaps in the same way.
They would use the same software. They would use the same building blocks. And it basically allows us to have this really, really agile team that if we decide our Engage product is the most important thing we need to focus on over the next six months, that we can pretty easily move a bunch of our software developers from a team that we're not on Engage into Engage and have them come up to speed quickly, have them be able to maintain and evolve that software stack really quickly, be able to understand the team's culture, understand the team's processes almost seamlessly. So, you know, some shows with people who are building service proxy tools like Envoy.
I talked to Matt Klein from Lyft and Envoy standardizes the communications between different services. You know, it gives you a standardized way of doing load balancing, gives you a standardized way of passing messages between different services. But an equally valid way of standardizing how the different teams at Intercom operate with each other or just operate on their own is just to have cultural processes. It sounds like that's what you have seen more at Intercom in terms of standardization is that people just have a way of doing things and that way of doing things has permeated, whether or not it has permeated to the level of a standard microservice unit like a Docker container.
Yeah, I think that's very fair. I think maybe we are maybe more advanced on the standardization of process and thinking and maybe not as far along on the standardization of inter-service communication, though I would say dropping a message into an SQSQ and having it picked up why another service to be acted on asynchronously is a pretty standard way of different services in Intercom communicating with each other. But again, I think the things that slow you down are solving problems, people, cultural, teams forming, team dynamics, understanding all of your biggest problems are solved by multiple teams and multiple disciplines collaborating together. So definitely I think that the standardization or the ways of working or ways of building that we do have definitely help large groups of people come together to solve new problems that we haven't encountered before.
It's so interesting to talk to you about this because a lot of the shows that I did I think earlier were, well, I would say there's several different types of companies I talked to. So companies like Netflix and Amazon and Facebook and Google, they were formed in a time where it was not as much you could take everything off the shelf and just kind of build your, you know, build a product development culture in an atmosphere where you can take these building blocks off the shelf. But now I talk to companies like Intercom and I talked to Fiverr a while ago. I've talked to a number of other companies that have grown quite large off the backs purely of cloud services.
I guess you could argue Netflix is like that at this point. They made such an aggressive move to the cloud. But I mean, can you talk about some, are there any principles that come to mind that are true in this environment where you can take so much stuff off the shelf that are not necessarily true in the older world of software where you really did have to build a lot? Because now it just, it almost seems like build versus buy, you buy almost all the time.
Like, just buy. Yeah, let me have a think about that one. I would say maybe the biggest difference that I see between how we operate now at Intercom versus how I would have say operated at Amazon, particularly Amazon, maybe less so than Facebook. One of the core values we have inside of our, or in D.org, in Intercom is think big, start small.
And so we would think about a problem we want to solve, we would understand it from first principles, we would then think really big and design our, like, design our final solution and then pair it right back and scope it right back to, like, the smallest, simplest, fastest thing that we can do that will solve a problem that will solve a real problem for customers, almost kind of like pairing it back to your MVP, ship that, and ship it to a small group of people and learn and iterate and iterate and iterate and slowly but surely build towards that bigger thing. And I felt like when I was at Amazon, there was a think big, start big, rather than think big, start small. And you had to design that V3, like, design that, like, really big, all-encompassing solution and there was a lot more pressure for anything that kind of went to market or anything that you were going to ship into production. It had to be fully featured, fully finished, incredibly reliable.
Maybe cost wasn't so much of an issue, but it had to be really, really well polished and really, really well thought out. And there was much less of an opportunity, like, everything was so critical. There was much less of an opportunity to learn or make mistakes or really move fast. And so the average project would take six to nine months or it wouldn't be uncommon for, like, a project to take a year or even more.
Whereas now, I guess, in startup world, we could definitely do that, think big, start small, move fast. You have to run your own MySQL database. You have to figure out your own caching infrastructure. Does that make sense?
It does, it does, completely. So I know we're up against time. To close off, so in this atmosphere of running less software where you get to take a lot of cloud services off the shelf, I imagine that managing outages becomes less of a frequent issue. But since I saw you give this talk at SRECon, which is Cyber Liability Engineering, maybe you could talk about just briefly how you manage outages, what your response team around outages or incident response looks like, and how that affects the culture at Intercom.
Great question. This is actually a lovely story, I think. So when I started at Intercom three and a half years ago, we were averaging about a 99.91% overall system availability, which is still pretty good. 99.9% was our customer-facing commitment.
And I used to record every outage we had. I would send out emails every week to the company giving them an update on our availability. I would track all of the sources of our outages. And our outages were split between ones which were caused by infrastructure scaling events versus ones that were caused by us moving fast and us maybe deploying changes too quickly or without properly understanding what we were doing.
And over time, it's been really interesting to see the different types of outages, the different measures of availability change. And so over those intervening three years, we've outsourced more and more of our undifferentiated heavy lifting to AWS. We've learned more about how we develop software. We make very, very heavy use of feature flags.
Even though we are deploying code to production 100 times a day, almost every change is going out behind the feature flag or certainly a lot of the major changes are going out behind feature flags. So it's almost like dark launches and you have the ability to turn things on and off very, very quickly. And so now we're at a place where I think we're going to end this year with 99.97 availability, which is a significant increase from where we've been. I think this is going to be our best year for availability ever.
Our biggest outages where maybe S3, S3 related outages, we've had very, very, very, very few real operational outages. We still have, hey, a software developer has deployed code and it has had an unintended consequence and wasn't behind the feature flag, but even those are getting rarer and rarer. So, I mean, I'm an ops guy. I love high availability.
I love hard scalability challenges. I'm probably going to curse myself here, but things have been coming, things actually have been becoming really, really quiet these days. So I kind of hearken back for the days when things were a little bit more exciting. Well, be careful what you wish for.
Indeed. Okay, Rich, it's been great talking to you. I really enjoyed your talk. I really enjoyed this conversation.
The time flew by. I'm looking forward to doing more shows about Intercom in the future. Great. Thanks very much, Jeff.
Wow.