Envoy is an open source edge and service proxy that was originally developed at Lyft. Envoy is often deployed as a side-car application that runs alongside a service and helps that service by providing features such as routing, great limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community and has formed the backbone of popular service mesh projects such as Istio. Envoy has been mostly used as a back-end technology, but the potential applications of Envoy include front-end client applications as well.
The goal of Envoy is to make the network easier to work with, and the network includes client applications such as mobile apps running on a phone. Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP-2, GRPC, and QUICK.
Matt Klein is the creator of Envoy, and he returns to the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of back-end systems and cloud infrastructure. He discusses the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touches on the scalability challenges that Matt is seeing at Lyft. Matt has been on the show a couple of times previously, and you can find all those episodes with Matt and the episodes about Envoy, the episodes about service mesh.
You can find all those on the SE Daily apps for iOS and Android. We've refactored these apps recently, and they are in great shape. They are a great listening experience for software engineering daily, and they include the ability to comment and search through episodes. I will be commenting on each episode going forward for the foreseeable future, so if you're interested in discussing software engineering daily, check out the apps or go to softwaredaily.com, which is the web version of our platform, and share your thoughts.
You can also use these apps to become a paid subscriber and get ad-free episodes. Also, Find Collabs is the company I'm building. Find Collabs is a place to find collaborators and build projects. If you've got an open-source project or a music project or really any kind of project, and you're looking for collaborators, you want people to collaborate with you on your project or even start a company with, you can check out findcollabs.com.
With that, let's get on to today's show. I'm Matt Klein. Welcome back to Software Engineering Daily. Thanks for having me.
The goal of the original Envoy project was to make the network easier to work with. Give a reminder of the networking problems that Envoy addresses. Yeah, so what we've seen as most folks have moved to microservice architectures is that they typically run into a bunch of common scaling problems, and these are typically around networking and observability. These are things like not being able to understand where failures are occurring or not having consistent networking features around things like service discovery, load balancing, time out, free tries, things like that.
At the same time, what we've seen in the last five or 10 years is that folks have moved away from situations in which they typically use a single language. The times of the enterprise job deployment are typically over, or a single job enterprise deployment. What we see these days is we typically see what I call a polyglot language environment. That means that either via acquisition or companies that are allowing developers to develop in multiple languages, we see quite a few different languages that companies are using in production.
What that ends up leading to is that in order to solve some of these common concerns, again, around service discovery, load balancing observability, etc., companies essentially have two different choices. They can implement libraries in each language, and those libraries can actually go through, and they can implement all those common concerns, but then of course they have to do that in each language. Or we've seen a pattern that's become popular over the last few years, which is the side car proxy pattern, and that side car proxy can implement a bunch of that functionality, and then that can be done in a single language. Then if a company is using five or seven or 10 different languages, those frameworks can rely on the side car proxy to actually implement that functionality.
One boy was built as a, originally as a side car proxy to do these types of things, and then over the past few years, as it's become a lot more popular, we've seen Envoy being used in a variety of different situations. We see it being used for edge load balancing. We see it being used in a side car or a client side load balancing scenario, and then we also see it being used as an internal or a middle proxy scenario. The reason that we've seen it being used in all of these cases is that having a single piece of technology that an organization knows how to operate, it's very nice to be able to use it in all of these cases so the organization only has to learn how to use it once.
We are solving a bunch of common concerns, and we're solving them both for the side car proxy scenario but also for the edge scenario or the middle proxy scenario. These different scenarios, the edge proxy, the back end, side car proxy, these are not mobile application proxies. The original Envoy was mostly used for server side application, and mobile applications are more of a client side situation. How do the networking challenges of mobile differ from the networking challenges that are experienced on the server side?
What's super interesting is that they don't actually differ that much. Of course, there are lossy networks, and much more frequently we wind up in environments in which the mobile clients have to deal with either spotty networks or slow networks. But in terms of what the mobile clients have to do, it ends up being pretty similar. They have to do some type of service discovery to find back ends, whether that typically uses something like DNS.
They have to make RPCs, they have to have retried policies and timeouts and those types of things. I think the other thing that's happened over the past couple of years, in particular as many applications have moved to mobile first applications, is that we have realized from an industry perspective that all of the effort that we're putting into making the server side system or reliable is obviously fantastic. But if at the edge proxy we have a, let's say, 99.99% success rate or 4.9s, that's obviously great. But if the user of the mobile application is only seeing a 75% success rate, it doesn't matter how amazing our infrastructure is on that server side.
So we end up needing a lot of the same observability, a lot of the same things that we need on that server side. We need it on the client side so that we can make sure that we are delivering that fantastic experience to our users. There are many things that can go wrong on a mobile client device. You could have a poor cellular network, you could have a problem with the operating system.
How do we differentiate between problems in the network versus problems that are just happening as a result of, and when I say network, I should say the programmatic network rather than the cellular network or just something that's going wrong in the operating system or something that's beyond your control? Sure. So if I actually sit down and think about it, it's not really any different than what we do on the server side in the sense that if I have a service that talks to another service and I'm running within some type of virtual infrastructure, how do I know if it's my application? How do I know if it's the virtual machine?
How do I know if it's the physical network? So it's really the same set of problems. And I guess the quick answer, which is not very satisfying, is that sometimes it can be very difficult to tell. And all we can do from a network operations standpoint is we can try to build in the right metrics, the right analytics, the right logging so that we can view the network at different layers and we can try to correlate failure.
And that might mean trying to look for patterns. So are we seeing, for example, all of our users who are on the TMO network, are they seeing a particular problem that might be indicative of a network failure? Are we only seeing issues with users who are on a particular version of iOS, right? That would indicate that maybe we have some bug that's related to that version of that operating system.
So what it comes down to is that there's very rarely a smoking gun. And I think what's much more important is to have the right infrastructure in place so that, again, we can gather the right stats, the right logging, the right tracing, the right analytics, and then we can start debugging and we can start trying to understand what are the patterns that we're seeing. So when we start to talk about why we would want to run something like Envoy on the mobile client, a lot of the reason that we want to do that is that it gives us a base platform with which we can actually emit a lot of these metrics, a lot of these analytics and a lot of this logging so that we can do the right debugging to actually start to understand where the problems are happening. So there are tools for observability that are on the mobile device.
There are things like crash reporting. Why is it advantageous to create a module like Envoy mobile where all of these things would be centralized into one system? Sure. You know, the main rationale is really for the same reason that we've seen people start to really utilize the sidecar proxy paradigm, which is that if we have a polygon environment where we're using different frameworks, different programming languages, potentially different operating systems, what we can do as system developers and system operators is what we're doing, what I call reducing cognitive load.
And by that I mean that we are developing these super complicated distributed systems and the only way to really understand this effective chaos is to try to limit the number of variables. So, you know, we would like to avoid re-implementing code in multiple places. We would like to have single implementations of things like retried policies. So for the same reason that it's potentially advantageous to use something like Envoy as a sidecar or a system proxy because we get a bunch of this functionality in one place.
Instead of having to go and implement a bunch of this logic, you know, and again, that could be for networks and primitives like retries and time-outs or it could be for stats logging and tracing, instead of having to implement that, you know, on both iOS and Android and support every operating system version, we can implement this in one place and we can get the same output in every place. And that reduces cognitive load and makes it a lot easier to understand what's actually happening. And what's interesting about this is that there is actually quite a lot of private precedent for this. Google has a library called Cronent, which they have shipped for quite some time.
It's a section of the code that ships with the Chrome browser. They use that code in all other mobile applications. That's common C++ code that is used both on Android as well as iOS. Facebook also has a long history of doing something similar.
So you know, this is something that's been done for, you know, quite some time. But we think that with Envoy Mobile, we can take this paradigm and we can do even more. The back end model for deploying Envoy is in this sidecar container. In mobile environments, I guess we don't use containers or they're at least not as much of a present deployment system as they are in the back end.
So you're using, you're just bundling it as part of the binary. Is that right? If an application wants to use Envoy Mobile, they just include it as a library? Yeah, right.
So it's going back on some of the things that I've typically spoken about, which is that, you know, the library-based solution is not very productive and you have to deal with upgrades, et cetera, et cetera, et cetera. But you know, the current realities, I would say, of how people ship mobile applications are that the code has to be bundled. And so our initial target for Envoy Mobile is to actually bundle the code as a library component and then actually ship that out. I do think that in the medium long term, if we are successful with this project, I could see the sidecar pattern moving over to mobile.
You know, there's people that I've talked to over at Google that have expressed interest in this. I would imagine that Apple might be interested in also, but I think that we'll have to see how things evolve. But in the short to medium term, I think we are looking at a bundled library. Compare the deployment model for using Envoy on the back end with Envoy Mobile.
I understand it's a binary and so that's a little bit different. Can you just describe, let's say I'm an application developer. I've got like a food delivery app and the food delivery app has existed for six years. It's on the app store.
Now I want to add Envoy Mobile to it. What's that experience like? So to be super clear, it is very, very, very early days for the project. We made an explicit decision to open source early.
So we prove that we can get it running on both platforms. We have open source, our roadmap, but it's still, I would say, very early days. So in terms of what will happen for the user, what the API will actually look like, I think that's still being sorted out. But I think the idea is that we are going to offer a networking library and the goal of this library is to not expose any of the C++ code up to the end user.
Our goal is to build a future looking network API. So, and by future looking, I mean using the more modern mobile programming languages. So that'll be Swift on iOS and Kotlin on Android. And our plans are to surface a mobile networking library that offers various pieces of functionality.
So that's going to be making API calls, obviously dealing with different types of APIs, whether that be ResBase or a proto-based. And then eventually we expect to add a bunch of higher layer functionality. So one of the things that I'm most excited about is that as we move towards proto or structured APIs, being able to offer higher layer functionality. So things like streaming API caching or offline or deferred APIs, things like that.
And I think that will really move the state of the art of mobile networking forward. So in terms of how people will consume this in the future is that we expect to offer an API. It probably, you know, we may make it look like existing APIs. So for example, we would look at HTTP as something that we might want to emulate.
But I think that we are looking at this from a blank sleep perspective and just trying to understand what would a modern mobile networking API look like. And then as we add more functionality, again, we expect to add ergonomic APIs that will allow people to admit stats, logging, and tracing, enable them to opt into some of these more advanced features. And that would apply both to JSON ResBase APIs. And then for those users that are moving towards the proto or the structured APIs, we would allow them to opt into even further features.
As I was reading your article, I did have to look up a couple of the networking technologies that you touched on in the article. One of the things you were saying is if you get this proxy in your application, you can defer some of the work of low-level networking technology to this proxy. So there have been this set of networking technologies that have improved over the last five to seven years. And they're not necessarily available out of the box to anyone building a mobile application.
Describe some of these lower-level networking technologies. Yeah. So these are things like modern TLS, so TLS 1.3, which has both performance as well as security enhancements. So there's like quick or HTTP-3.
This is the next gen networking protocol, which is aimed for mobile or loss of environments. And what we typically see on mobile is that just because of the longer development times, the fact that handsets tend to be in the wild for longer, we typically see a slower adoption of these types of, I would say, newer or more advanced networking protocols. So if a mobile application wants to depend on something like TLS 1.3 or quick, it may be years before essentially all of the platforms across both iOS and Android and all of the iOS versions that they want to support with their application in maybe years before they can rely on those technologies being available. And that's not only years of potentially slightly worse security or slightly worse performance, it's years of, again, having that cognitive load, right?
If you're having to support your application across two different major vendor ecosystems, a bunch of different operating systems, that debugging cognitive load just becomes so much larger because you have to start understanding, you know, where's this bug report from, or how do I slice and dice based on the version of the operating system or the platform that the user is actually using? So being able to relegate or offload a bunch of this functionality to common code, whether that be a side-car proxy or even a library, it allows for a consistency of experience, and that improves both cognitive load, it potentially improves security and improves performance. So that's ultimately what we're going for. And then like I was saying before, we think that once we have this base functionality in place, just like we can do a bunch with the side-car proxy pattern of offloading more complicated functionality or more advanced functionality, we think that we can do that on the mobile side as well.
We're implementing caching for streaming APIs or doing a deferred or offline APIs instead of having to implement that and test that and understand that across both Android and iOS. If we can do that in a single code base, it allows us to focus on that code base, make sure that it's well tested, make sure that it has high performance, and that just seems like a better use of time. Can you shed more light on why this stuff doesn't get bundled into the operating system? If it's just, it seems like such a free lunch, right?
If it's just networking technology that improves your performance and your reliability and your security, why is it not an OS level improvement? To be fair, it is. Google and Apple, they obviously will work on technologies like TLS 1.3 or they will work on quick. But again, because of the longer cycles of how these handsets are built and if Google releases a new version of Android, then it has to go to Samsung or it has to go to the different makers of phones and they have to certify the operating system and then they have to get it onto the phone.
There can be multi-year cycles before a particular piece of technology even gets to a shipping phone for the first time. So, OS cycles tend to be longer and because of the risk of breaking existing handsets, Google and Apple typically don't backfill major features for a long period of time. What you see within the industry is a handset that might get shipped on a particular version of Android and then one of Apple's handsets might get shipped on a particular version of iOS. Those handsets may see one or two operating system upgrades in their lifetime.
It's not like Apple or Google continues to support all handsets and perpetuity for OS upgrades. That's not the way that works. It's not that they're not working on it. These things are OS concerns.
It's just that it takes a long time to get all of these technologies to every user and in that time frame there's some newer technology. So if you or if one wants their mobile application to always be at the forefront of that technology at the networking layer and again whether that's for security or performance or features, it's nearly impossible to ensure that every user of one's application has access to that. And that's why we see even Google who obviously runs Android, they ship this library to all of their mobile applications because they and Facebook both view it as a competitive advantage to make sure that they have a consistent networking experience across all of their applications regardless of the operating system version. So if you bundle the networking stack with the application, then as your application gets updated, then the networking stack gets updated and people definitely update their apps more aggressively than their operating system.
Exactly right. And again, that's not to say that in a future world, something like Envoy might not be bundled in a way where it's out of the hand from the operating system, but it sees updates, but it's a shared library or a shared proxy, if you will. I could see the industry moving in that direction, but that's not where we are today. When I make a request from my mobile application these days, is that request usually in JSON?
Is it in a protocol buffer? What format is it in? Hard for me to answer that's hard for me to speak across the industry. If I had to guess, I would say that the majority of API requests that go from a mobile application to a backend service are probably JSON, YAML, REST type APIs, I do think that there is increasing uptake of structured APIs, whether that be using a technology like Rift or protocol buffers, I think protocol buffers is becoming the default structured API.
So I think that over time, as it becomes easier to consume protocol buffer APIs, I think that we're going to see more and more and more APIs be based on them. As I've written about before, I think protocol buffers version three, or as people say, proto three, I think it's a really fantastic technology. I cannot recommend it enough. I just think Google has put a tremendous amount of thought into it.
And one of the nicest features that I find about proto three is that it allows APIs to be defined in a structured manner. So everything is tight, but also allows seamless conversion between that structured representation to JSON or YAML. So to me, it really gives you the pest of both worlds, which is that you can define APIs in a structured way, and you can seamlessly transcode them between JSON, YAML, and structured. And I think that once more folks realize that inherent power, which is that you really get the best of all situations, you can support backwards compatible clients that only want to deal with JSON, but your internal systems can be structured.
For me, there's almost no downside to it, and the tooling is getting better and better and better. So today, I would say that the majority developers probably still use raw JSON, YAML, REST type APIs. I do think that over time, we will see more and more APIs move to being structured. What is required of a developer that wants to use protocol buffers in their application?
So what is essentially required today is that the APIs obviously have to be defined, so that happens in a structured file. And then those APIs have to be essentially compiled. So what the protocol buffer compiler does is it compiles language-specific representations of those APIs. And that gets emitted out into all of the different languages that protocol buffer supports.
And then those APIs can either be used raw or Google supports technology called GRPC, which essentially is just a wrapper on top of protocol buffer that allows for descending and receiving a protocol buffer-based APIs. So from a user perspective, typically the easiest way to consume protocol buffer-based APIs is probably going to be via GRPC. So that will be defining the APIs using the generated GRPC clients and servers, and then just running wild. But again, what we've seen historically, and this is getting better every day, is that the GRPC generated code is of different quality depending on the language.
And again, we come back to this inherent trade-off of the library-based solution versus the side guard-based solution. And if we are making APIs and we want to co-generate those APIs in a bunch of different languages, there's obviously no way around doing that in each and every language. And as you might imagine, some languages currently have better support than others. So if you're writing in Java, obviously the support will be amazing.
If you're writing in a less frequently used language, the support might not be quite as good. So that is evolving over time. And what we've seen at Lyft, for example, is that Lyft is a very forward-thinking user of Swift. So we actually were one of the first, I think, companies to rewrite our iOS application using Swift.
And historically, for example, the protocol buffer-based support for Swift has not been quite as good. So Lyft has actually worked a bunch on open sourcing, Swift co-generators. But over time, I think, as more people want to consume protocol buffer APIs, I would think that the co-generators will get better and better and better in every language. And then I think it will increasingly become easier to use.
Does Envoy Mobile help with the usage of protocol buffers at all? What's the connection between protocol buffers and Envoy Mobile? Right. So from the Envoy Mobile perspective, I would say that there's two different angles that we're actually coming at this from.
One of the angles is from the lowest layer. So that's what we already talked about in terms of providing consistent networking across both platforms. So these are based networking technologies, like making sure that we support Quick or we support TLS 1.3, or we support modern extensions for propagating network quality or things like that. And that's obviously one angle that we are approaching it from.
The other angle that we're approaching it from is from the top down angle. And that's from the protocol buffer-based angle. And one of the nice things about protocol buffers is that it supports something called annotations. And annotation is a way of giving a particular API, which is messages and fields, and those fields can have types.
On a field, you can provide custom tags or custom annotations. And those annotations are where we plan on supporting some of this advanced features. So for example, again, caching offline deferred APIs, we imagine a scenario where if Envoy Mobile can determine the current network quality, so for example, if Envoy Mobile can consistently determine, we are on a fast Wi-Fi network, or we are on a fast LTE network, we might use the enhanced version of the API. Or let's say that the library determines that we are on a spotty network, maybe we can start, we can dynamically shift to a degraded API that uses less bytes back and forth and will probably load faster for the user.
So once we have this ability to provide these custom annotations, we would allow the Envoy Mobile layer to actually operate on those annotations. So there would be nothing company specific about those annotations. So company or organization A can define their API, but they can use a common set of annotations and a company B can do the same and so on. And then Envoy Mobile, given the definitions of the APIs, can actually operate on those annotations in an API agnostic way.
So even though the API might be talking about right-sharing for Lyft or might be talking about music for someone else, all of these applications can get a common set of functionality. And what we think, and again, as I was saying before, this project is in very early days, I would encourage everyone to check out our repo and obviously follow along. We think that we will probably support the GRPC wire protocol so that Envoy Mobile will always be compatible with GRPC servers. But what I would imagine may end up happening is we may have our own Envoy Mobile co-generators so that if you are using Swift or Kotlin, we can generate you library stubs or API stubs that will give you all this advanced functionality and you won't need to worry about the underlying mechanics.
So I would suspect that that's how this will play out. It is that again, there will be two angles here. We'll have the lower layer networking angle and then we'll have the top-down API-based angle and we'll be focusing on both. You bring up an interesting point, the idea of API degradation where if I'm using Lyft, for example, and if I'm on a great network connection, I'd be fine with my Lyft application heart beating with the backend with tons of information about where I am and my velocity and so on, just tons of information.
But if I get to a spotty cellular connection, I definitely want the Lyft backend to know where I am and that's maybe the core of what you would want to send. It's like, here's my Geo every 10 seconds or something. If you're on a really bad connection, maybe you just send it every 10 seconds, like as low as that. But the idea that you would want different volumes of information sent to the backend under different cellular connections, I would guess that most mobile applications don't factor in that option.
If for no other reason, then it's pretty hard to just like, what do you have? Like a switch statement, it's like, if cellular connectivity is three bars, do this, if it's four bars, do that. And so putting that into a usable API sounds pretty handy. Well, you know, what we see, and we've seen this across almost every type of application.
And I've personally seen this at multiple companies now. It's been widely reported at tons of other companies. And it's actually, it's a very simple thing, which is that we almost universally see that when applications are faster, when they are snappier, when they load, when the user perceives them to load quickly, business conversion is higher. And again, we just see this across almost every application.
And we can see pretty substantial differences. These are not small conversion differences. They can be many, many percentage points. So from a business perspective, and that's why I think you've seen investment in protocols like Quick and for other types of technology that are really focused on ultimately making the user's application snappier.
So when we see this, again, almost universally across every application, it's widely accepted within the industry now that focusing on making the end user's application snappier and to grade well is a very, very worthwhile thing. So then the question becomes, how does one do that? And like you were saying, it's certainly possible to have a bunch of switch statements and if statements and specific logic in every application trying to determine network degradation and those types of things. But we believe that there's a tremendous opportunity to do this within the library.
And that if we do this well, we can actually be a lot more sophisticated about it and a lot more general about it than any one application could probably justify doing within only their application. And we can provide the right hooks and the right tags and the right annotations so that again, just to go back to Lyft, you can imagine, and again, I'm not speaking about any particular plans here, I'm just trying to give an example, you can imagine with Lyft that yes, as you pointed out, we have core functionality that must happen. We must provide location data, occasionally we must obviously when we're requesting rides, we must show car locations. But maybe there's other things that are important, but they're not critical.
So for example, let's say that we want to show coupons or something like that or we want to give updates on the app or there might be all types of things that are important, but if we're in a really poor networking environment, maybe we just don't want to show those right now because the critical data has to come through faster. And we believe that by making the API structured and by annotating and tagging the API in certain ways and then having the network later be smart about what can be dropped both by the server and the client, we think that in these poor networking environments, we can greatly reduce the byte sent back and forth. And like I was saying before, ultimately, at the end of the day, what it comes down to is you just need to send less bytes back and forth. The fewer bytes in general, the faster things are going to load and the faster things are going to load with the right functionality, the better business conversion is going to be.
We've done some shows about the service mesh idea, which is that as an expansion on the model that we've been talking about where you have a proxy and you have a proxy on every service and the proxies can talk to each other. And because you have a standard proxy for communication between different services, you have a more reliable network and you have some homogeneity. And the next phase on top of that is the idea that you can do control across all of these different proxies. And so you have a unified control plane that interacts with all these different proxies.
Did Envoy Mobile change your vision for what the ideal service mesh would look like? Yeah. So this is certainly not our short term goal. It's probably not even our medium term goal.
But when I look out long term, and there's obviously a big if you're if the project finds success, if we can get wide deployment, I think there is a huge amount of long term potential here in terms of how people actually manage how they configure networking on their mobile applications. And some of your listeners will probably chuckle at this. But I think there is the potential to quote unquote expand the service mesh out to the mobile client. And I'm sure that will scare people greatly, right?
But by that really all I mean is that one of the reasons that Envoy, I think Envoy has become very popular for a couple of different reasons. But one of the reasons that Envoy has become very popular is because of its configuration API. And these are APIs that collectively we called XDS. These are our discovery service APIs.
And at least on the server side, we've seen now an entire ecosystem of companies, whether they be cloud providers or startups or individual companies, you know, who are building these management systems that manage fleets of Envoy's. And what you'll see is that if we assume for a second that Envoy runs on a bunch of mobile applications or even to the mobile applications for a company that's already using Envoy, if we can use these real time configuration APIs to do things like traffic shifting or potentially authentication or on the fly, actually, you know, change retry policies or maybe want to shift just a single route to a particular data center or we like to test something or something along those lines, there is the ability to really change how mobile networking is configured, how the policy is actually sent out. And these are not really science fiction things. So for example, to give a concrete example, you know, the state of the R today in terms of how major mobile applications tend to do traffic shifting, low balancing, low chatting is they typically use DNS and they typically use BGP.
And you know, people have done some incredible things, right? Like they use DNS to find the closest location and then they'll use BGP to, you know, make sure that we're routing to the right place and pretty amazing sets of technology. But in some ways, just and again, we can do a whole show on the complexities of how people do how people do edge routing. But to make a very long story short, there's there's a lot of complexity just in how people end up using DNS and how you can have intermediary clients that do caching in weird ways or with BGP, how, you know, occasionally you can get routes that are just black-old and then there's basically no more routes and et cetera, et cetera, et cetera.
And the thing about potentially offering something like XDS all the way to the mobile application is that although of course the actual XDS management server would have to be bootstrapped over something like DNS, there's probably no way around that, once the mobile application is able to get configuration data, even if it's heavily cached or, you know, we're using different types of mechanisms to actually scale that, we can do much more real-time traffic shifting. We can do much more real-time load shedding. And we can do that in a way that actually works, works similarly, right, to how server side is doing. If you look at some of the managed control planes that are sprouting up, so Google has a product called Traffic Director, Amazon has a product called AppMesh, these are, of course, these are targeted towards server side load balancing and server side policy.
But you could pretty easily imagine some of these control planes extending all the way out to mobile because if I have multiple data centers or I have multiple regions and I'm trying to load balance and I'm trying to actually shed traffic between these regions, either for performance or fault tolerance, why would I not potentially want that to apply all the way to my mobile client as well? So that's just a small taste, right? I think that if we're successful, I think that, you know, we may be able to rethink how some of this policy actually happens, which is pretty exciting. Whenever I talk to people about mobile application development, I get a sense that there is significantly less dynamism in how we can roll out these applications largely because of the app store gating process.
I think that has been eaten away a little bit with things like React Native, where you have some dynamism that just gets bundled into the application. And then this might be another example where if you bundle in this dynamic, you know, networking technology, then maybe you could issue more dynamic networking updates, which would be cool. But just because you're kind of like a backend developer who has waded into the mobile application space because of on-point mobile, what's your perspective on that app store gating process? Is that useful and how problematic is it for or to you?
I'll be perfectly honest in saying that I am far from an expert here. So I don't think I can provide a truly educated perspective on that. I fully understand that both Apple and Google, you know, they're trying to have some quality control and I think that's fine. I think, as you've said, you know, there's been a back and forth between application developers and, you know, Google and Apple in terms of what is allowed, what is not allowed.
You know, we talk about dynamic code generation. One of the things that we're working on right now on server-side Envoy is actually WebAssembly support. So and we'll have APIs on server side where a management server can actually ship out pre-compiled WebAssembly bundles to Envoy's and actually run them. And we believe that there's tremendous potential here for actually today with Envoy, we obviously have Lua and yes, it is possible today to actually update Lua code from the management server, that Lua has its own issues.
But from a WebAssembly approach, you can imagine a world in which people want to write extensions, you know, maybe those extensions are written in Go or Rust or C++, but they're inherently safe because they're WebAssembly and they're written to a particular sandbox or runtime. You can imagine a system in which in real time, right, like almost, almost like kernel level TCP dump style rules, imagine that I can compile a small WebAssembly program and I can send it out to my server-side Envoy's to actually do, you know, extremely efficient matching so that I can do traffic tapping or maybe blocking rules or things like that. Now, you could think in the far future, if I can do that server-side Envoy, why couldn't I do that out to mobile, right? Like, why couldn't I potentially have a part of my network logic that's maybe implemented in something like WebAssembly and it's shipped over XDS to the app and it's obviously cached.
Now, would that run a foul, the Apple or Google App Store rules? It probably would today. But as you pointed out, there's things that are happening in the React world where there is dynamic programming happening. So I think it's a very interesting topic and my suspicion is that, you know, Apple and Google, over time, you know, will probably adapt to the modern realities, right?
But I think that it's going to be a give and take over time, right? I don't see the App Store process going away. So I think that from, you know, the unbeknownst perspective, we obviously are going to do everything within the rules and then, you know, if we are successful and we want to do things in the future, then we'll obviously have those conversations. Yeah, I mean, I do have some respect for the App Store review process just because our mobile devices are these things that we've come to rely on so much.
So if you had application that, if you could skip the review process, you could get application in the App Store that would just instantly brick your phone and that's like breaking your lifeline in some condition. So I don't know, again, I'm not an expert either, but I didn't want to give the impression that I'm just a total cowboy and want to remove the App Store review process. No, of course. And you know, what happens over time also is that I think if you look back eight, eight ten years, I'm guessing that that the App Store review process was even more important because the operating system, the mobile operating system probably had less control and less safeguard over an application using too much battery or using too much network or stuff like that.
And I think over time, it's been pretty clear that the mobile operating system vendors, they're obviously continuing to invest in those types of sandboxing or those types of safeguards because you know, the App Store review process is only so good, right? You obviously want the right safeguards in place so that even if something slips through, like you said, you know, an application can't break that device. So it might even be the case that as the sandboxing gets better and better and better, maybe the review process can get a little less stringent. But again, this is not my area of expertise.
So I know we're running out of time. It's been pretty interesting talking to you a few times over the last couple of years about how the challenges that lift have evolved over time, the scalability challenges. Are you encountering anything new? I think how many years have you been there at this point?
Over four years. So it's the time the time just falls. Four years. So what are the newer things that you're seeing these days that are presenting challenges that didn't exist before?
I think what we're seeing now is, you know, whereas I think before, at least on the on-boy or the networking side of things, we were developing features and doing all types of things, right, that were aimed at, again, like whether that be retries or timeouts or feature after feature after feature. I would say that the major scaling problems that lift now, at least the ones that I'm seeing from an infrastructure perspective, they're less technology problems and more people scaling problems. And by that, I mean, it's trying to figure out, you know, we've gone from, I think when I joined, the whole company was probably less than 300 people for sure. So maybe 250, 275, and there's probably 75 developers or something like that.
Now, you know, I actually have no idea. Probably over 5,000 people as a company. And, you know, there's over 1,000 developers, maybe 1,500. So I mean, you know, we've probably at least 10,000,000 in the four years that I've been here.
And I think what we've seen from a scaling perspective is that even if the technology maybe works, we have just, we've reached a scaling threshold where, you know, there's never enough documentation, right? Like when you're dealing with 1,500 people, if even if one person only has a question every two weeks, if you start doing the math, that's a constant stream of questions. So where I'm actually leading with this is that I think what, you know, we begin to realize at this human scale is that we just need less than ops, right? Like we need a concrete example, something that's killed us is manually configured circuit breakers.
So trying to make sure that every application has the right safety guards in place or the right timeouts, it's just, there's, there's too much entropy. It's too difficult to manually, you know, ensure that all these things are correct. So now we're starting to invest in tooling, you know, to either kind of determine these settings from historical context or eventually just do away with the settings entirely and make them adoptive. But I think that's, that's really where these scaling problems are.
It's not in needing a specific piece of technology or a specific feature. It's more in how the system is configured. So I think at least from this perspective, development has moved, you know, really away from core Envoy. I think Lyft does very few Envoy features anymore in the effort has shift much more in terms of how the system is managed or how we expose all these knobs to our users.
All right. Well, we did a show about that. That was our last conversation. Just as a final question.
Have there been any like personality adjustments that you've made to yourself or philosophical adjustments to accommodate that? Because I imagine it just feels like if you let it, the environment is just pulling on you all the time to, you know, hey, Matt, do this thing, do that thing. Like, you know, I don't know. Do you have any words of wisdom?
Wow. We should probably get together and do probably an entire show on this. We already did. Well, yeah.
Well, I don't know. I just feel like there's even more. You know, it's I guess for me, and this is coming out of my personal perspective is I have had to learn over the past couple of years both how to scale myself, but also just, you know, how to let the organization evolve in its own way or its own path, right? And I think that just comes back to human scalability is that when you move from a small organization to one that's 10 times the size, I think it just, you know, things take on a life of their own, right?
I think for me, it's just, I would say accepting some of the realities of how modern companies either do support, how they do DevOps and trying to toe the line, right, of trying to figure out what is the right amount of documentation, when is documentation not going to work and we just have to automate something, right? And I think there's no easy, easy answer there. But for me, it's mostly just been trying to find the right balance of helping where I can, trying to steer open source where I can, try to steer the architecture where I can and try to be more reasonable about how many hours are in the day. And that's, and that's ultimately, I think what it's come down to for me.
All right, well, we'll have to continue alternating between envoy related shows and human scalability related shows. Yeah. Matt, thanks for coming back on. It's always a pleasure.
Thanks so much. Wow.