Engineering teams often build microservices as their systems grow. But over time, this can lead to a fragmented ecosystem with scattered data access patterns, duplicated business logic, and an uneven developer experience. A unified data graph with a consistent execution layer helps address these challenges by centralizing schema, simplifying how teams compose functionality, and reducing operational overhead while preserving performance and reliability. Viaduct is Airbnb's open-source, data-oriented service mesh and GraphQL platform built around a single, highly connected central schema.
It has played a major role in scaling Airbnb's engineering organization. Adam Miskovich is a principal software engineer at Airbnb and he worked on Viaduct. He joins the podcast with Gregor Van to talk about how Viaduct originated inside Airbnb, the architectural principles that shaped it, the challenges of scaling GraphQL to millions of queries per second, and why the team decided to open-source the platform. They also discussed the future of backend development in an AI-driven world, and how unified data layers may influence the next generation of engineering systems.
Gregor Van is a security-focused technologist, having previously been a CTO across cybersecurity, cyber insurance, and general software engineering companies. He is based in Singapore and can be found via his profile at van.hk or on LinkedIn. Hello, and welcome to software engineering daily. My guest today is Adam Miskovich.
Hey, how's it going? Nice to be here. Yeah, good to have you here. Today we're going to be talking about Viaduct, and that is a spin-out from Airbnb.
So we're going to be understanding what happened there. But yeah, Adam, I'd love for you just to talk to us a bit about. First of all, just your journey to new to like to Airbnb, and then where did Viaduct come from and how did that come about? Yeah, absolutely.
Yeah, so I have been a software engineer for, because it's pushing 20 years or something professionally these days. And I actually took a little bit of a non-traditional path to kind of where I'm at at Airbnb, kind of working in big tech. I have done a lot of work at a lot of small companies. I ran an agency, like I interacted with agency in Baltimore, Maryland for a while, building web and mobile apps for folks and interactive installations, worked at a company called Expo, some folks and listeners might be familiar with, doing React Native tooling, and then eventually kind of ended up at Airbnb.
So I kind of went from small company to big company instead of big company to small company, I think like a lot of folks do. So it's definitely been a learning experience working at Big Tech. And at this point, I've been at Airbnb close to eight years. So I've seen it kind of grow from a 500 person engineering organization to 3000, and also the company around us has grown a lot as well.
So yeah, it's definitely been an interesting journey. That can be interesting. It's actually quite similar to myself. Yeah, we're at an agency for a long time, which was a most agile small company.
Yeah, I'm now in a bigger company, but that's only 150 people, but still feels big to me. I think what's cool about working at like a interactive agency is you kind of get exposed to a lot of different stuff, right? And it like forces you to kind of be a generalist. So I was actually hired at Airbnb originally as a front-end engineer, and had done a lot of front-end development prior to Airbnb.
And at this point, I haven't done front-end development for, I said I was, I've been there about eight years. I haven't done it for about seven and a half. So it kind of takes us into the to the GraphQL story and viaduct and all those types of things. But yeah, it's been cool to work at Big Tech and bring kind of the generalist experience, you know, into that.
Yeah, for sure. And I think as you call it, it will as well hear that will lend itself really well to kind of, I'm sure why viaduct came about and like what it is today, because yeah, the more problems you're exposed to, the more realization that you know, you have to actually understand so many different business types and requirements and so on and so forth. And inside Big Tech, you're effectively working on different businesses if you want to look at it that way. So yeah, how did viaduct come about?
What's the story there? Sure. So about the second day after I joined Airbnb, I was pulled into a working group. I was called the GraphQL working group.
And some bunch of engineers that are going to be had been thinking about using GraphQL, you know, we were kind of in this interesting spot. We were starting to do microservices, kind of move out of our Ruby on Rails model if we wanted especially like the iOS and Android and web engineers, right? They really wanted like, you know, strongly typed APIs. There was some experiments with GraphQL in the Ruby space, right?
We had this thing that I think was open source at some point called Graphists, which was like this very opinionated kind of framework. It actually wasn't GraphQL, although you can imagine GraphQL layered on top of that. That was kind of this opinionated framework in Ruby on Rails for billing, our endpoints, our API endpoints. And yeah, so I was kind of pulling this working group and there was a bunch of folks in there, I mean, it was probably 20 or so folks from back end from front end, whatever it might be, thinking about how do we adopt GraphQL in this new kind of microservice world that we were moving into.
And we had, there's already an opinion that was relatively pervasive and then kind of became a bit more pervasive around how to structure our microservices that Airbnb. So you can imagine, you know, we came from this big Ruby on Rails model if they had millions of lines code in it and we started to kind of carve out chunks like lots of folks do, right? We had a service for listings, kind of some of our search components were pulled out, that type of thing. But we were pretty early at that point in like kind of building a bunch of services.
But the way we were thinking about structuring the microservice architecture as a way as you probably hear me call it a bunch during this conversation is kind of have presentation services, derived data services and data services. So data services at the bottom pretty straightforward. They wrap essentially databases, provide kind of a threat, we've thrifted Airbnb, so kind of provide a threat API over core data. You have the presentation services up top that are really, you know, I'd say highly started as ports of controller logic from, you know, MVC system inside Ruby on Rails.
And so they were very tightly scoped to like, you know, they exposed just like RESTful or RPC over JSON type of endpoints, right? And then you kind of have the middle tier, which is all the random derived data services that pull data from X, Y and Z places and much together. So the GraphQL story was really focused at the beginning around this presentation service layer. And there was a lot of trepidation at first because we were trying to figure out like, okay, well, we've already started to build these presentation services and we had a couple big ones at that point, right?
And we needed to figure out how to let people continue to build these things because that's what we said we were going to do, but how do we put kind of GraphQL on top of them? So our first general approach was very far from, from Viduct, which was just kind of convert the first schema to GraphQL and stitch all of the presentation service schema together into one GraphQL schema and one GraphQL on point. And we definitely weren't the only people to take that approach and mind you, this is essentially seven years ago, right? I was just going to say, I mean, because you say this was like day two working group.
It really was like, it was day two, yeah. So I mean, this was around when quite a few companies, I guess, were starting to, well, probably some had maybe done beforehand, but this was quite a big time for bigger companies to say, hey, we actually think GraphQL is where we should be going with our APIs. Right. So actually, let me set the stage a little bit.
So like this is pre-apologoration. So like it's pre all of that stuff, right? And remember GraphQL actually just had its kind of 10 year anniversary back in September of like open source. So yeah, so this is a few years into GraphQL.
There was no Apollo Federation. Apollo did exist, but it was kind of early. They were trying to figure out, you know, what their business model was going to be, how to bring a enterprise as that type of thing. The client space outside of like relay was still pretty nascent, you know, GraphQL JS was mature-ish, but like, you know, not used in a lot of spaces.
And so everyone was trying to think about how to bring big companies and enterprises, you know, into GraphQL. And yeah, the schema stitching idea was pretty popular at that point, right? That was how you did it. If you didn't want to have one kind of monolithic GraphQL server.
So that's kind of the space that we were working in, right? So we were, like I said, maybe not the first, but kind of on the forefront of what folks were doing then. And so we took these thrift APIs, we converted into GraphQL, and we switched them all together. And what the schema that kind of we ended up with was what I like to call like service oriented GraphQL.
I mean, it was not a entity graph, right? In the way that we did it, right? It was literally like service foo as like a top level field. And then the endpoints for service foo underneath, right?
And we always knew that we were going to like transition away from that somehow. But what this gave us was the ability to get clients using GraphQL across iOS Android and Web, get that stack in there, figure out the code gen situation on the clients, all of those types of things, right? And we could pretty quickly, and you can imagine since these endpoints were the same kind of shape as our rest endpoints, then like writing code against them, client-side code was relatively straightforward, right? And then you could easily kind of migrate from the old version to like this new typed version.
Given this was largely all internal, as an internal APIs. Yeah, it's all internal APIs. Yeah. I'm sort of like barring here, but like how was that?
I just communicated, especially back then, like, hey, we're moving to GraphQL. That's quite a big shift for probably the number of services that we're talking about here. So like were you the person having to deliver this news, so to speak? And like, or like, how was that done?
Yeah, it was me and a few other folks that were working on this. I kind of was the main sort of backend guy at this time. But to your point about like going and telling people, the whole idea here was that the conversion from threat to GraphQL was automatic, right? So for the most part, the backend folks didn't really need to know, right?
It was kind of interesting. We were like protecting the backend people back then. I don't know. It's very different now.
Like I wouldn't say we do that. But like back then, it was like client engineers want to do some crazy stuff, protect the backend people, let them focus on their microservice migration, right? And so that's what we did. You know, we really made it as trivial as possible on the backend side.
And then on the client side, people were much more eager to adopt the new tools. And so it wasn't really a struggle to get folks to adopt the new tools, right? They wanted it, right? Especially on the iOS and Android side, but then you know, web was kind of a fast one there.
Yeah. So that was kind of the kind of client platform that really wanted to see it succeed for sure. That was kind of, you know, the origins of GraphQL at Airbnb and where it, how GraphQL itself really got it's foothold, but then around, let's see, was it the summer of 2019, I believe? Yeah.
So basically this had been around for about a year. This like GraphQL, new GraphQL stuff had been around, and Airbnb for about a year. Summer 2019, spring and summer, kind of hiring a new CTO, he comes in, he is like, what's going on with like our data at Airbnb? Like how are we doing this?
and we had a fragmented data store situation. Offline data was crazy. We were having trouble with our data pipelines back then, and we were thinking about IP Owing at that point. It's like, well, we got to get our core data to make some sense, so that we can use it for financial reporting and all of the things that are required there.
We spotted this working group, you'll notice a trend of working groups. We spotted this working group called the Data Architecture Working Group, running an outside consultant named Rainy Stata, and really started to think about the whole stack end-to-end. Whether it was online side, APIs, whatever, whether it's the offline side, we were really nothing was off limits to think about changing. We had like eight people in this group, a bunch of different disciplines from all over the company.
We sat in a room for what seemed like months at a time, trying to figure out what we were going to do. A bunch of things came out of that. A bunch of improvements to the offline world, it's much, much better now and took a lot of years, but I got there. A bunch of changes in the online data source side, so born out of the Data Architecture Working Group was a rethink of our core online data system, which is called UDS, since we're unified data source.
That's a big project that's been going on for a while. At this point, a lot of data is migrated to it. Then on the API side, really this idea of what became viaducts, or what we call a data oriented service mesh, but really is a more simplified way to say it is just the unified data access layer, really emerged as a key need to try to simplify how we build APIs, how we expose data to clients. We really shipped the very first version of viaduct, which really wasn't anything super crazy.
It was really just a graph you all server that was separate from this service-oriented graph you all. We figured out how to kind of stitch it in. We shipped this first version of viaducts. I mean, really early, a couple months in to the project, we shipped it and I think he was working on the trips product at the time.
So basically, imagine you go and say there will be a number right in the corner on your home screen and view your current trip. That was probably the first viaduct powered feature back then, that and maybe wish list. So yeah, that was the original origins. Very cool.
Again, it's like our question, but since we probably used the word a lot, I did grow up in a country that have a lot of viaducts, but maybe you could just explain what is the viaduct and why did that come to the name? Viaducts, I'm not going to get somebody yellowy yell at me if I try to get some exact definition, but essentially it's a bridge. A bridge every where water flows over it, basically. No, that's an aqueduct.
Oh, there we go. Wow. Yes, the viaducts is a bridge over a span of, you know, it could be over water, but usually it carries trains or cars or something like that. Then I've learned something like that.
I called everything a viaduct. It does tend to have arches, though, much like an aqueduct. So I don't know, hopefully podcasts listeners don't go crazy. I mean, maybe there's some like technical definition of a viaduct.
But anyway, the general idea, the reason why we call it viaduct are traditionally pretty bad at naming things that are going to be. We always put air in front of them, but with viaducts, you know, it's really just, it's a connector, it's a bridge, it's trying to connect things together. And we had a lot of services and a number of services grew tremendously over time. And yeah, that was the idea there.
Yeah. OK, so let's kind of get into, I guess, there was like a sort of philosophy shift where, I mean, there's a great blog post which we'll link to that you wrote about all of this about two months ago. They're kind of three guiding principles sort of to that philosophy shift, which was a central schema, host of business logic and what's called the re-entrant API and what's it getting to that. But you said in that blog post, like from the beginning, we've been encouraging teams to host their business logic directly in viaduct.
This runs counter to what many consider to be best practices in GraphQL. So was that from the kind of beginning beginning or is that something that's kind of coming through in time? It's definitely from the beginning. So like I said, we had built the original GraphQL system, you know, very much integrated with our microservice architecture.
But at that moment, in Airbnb is kind of engineering journey. You know, there was a bit of microservice fatigue. And while we have a lot more tooling nowadays that helps with microservice development at Airbnb in those early days, it was tough to have quick iteration, to understand your dependencies. And also, you know, we had this kind of really opinionated framework for how you actually write business logic in the microservice world.
So it was just, I think a lot of people found it really tough to like de-iterative. And so when we built viaduct, you know, the idea was, you know, like, yeah, we'll figure out how to like scale it. It's like do things that don't scale, right? We'll figure out how to scale it at some point.
But for now, let's just write code in there and we'll have some opinion on how to write code so that it doesn't go completely insane. But at the end of the day, we wanted to build a platform that really was kind of, you know, we had many early ideas of like how it can just be like Airbnb is, you know, to use a buzzword that a lot of people hate, like serverless platform, right? And I'd say it's had its ups and downs of that decision. And I think we have a lot of interesting things in the pipeline with some of the work that we've done with Viaduct modern, which is what we open source.
But it definitely ended up being like a pretty core tenant from the beginning. Yeah. I mean, just to kind of call off anyone, well, we've touched on, you know, Apollo already, but you know, a global schema approach, that's actually unlike Apollo or like Graph Go modules. So that's kind of the differentiator here, right?
That's right. I think there is some interesting similarities with Apollo and Apollo generation. I mean, at the end of the day, like Apollo talks a lot and has a lot of success with this concept of a super graph, right? And in their case, yes, it is a super graph made up of sub graphs that are kind of hosted in services.
But Viaduct is not necessarily that much different when it comes to this general idea of having a unified graph and vending kind of essentially one schema to your clients. And in our case, our sub graphs are not independent services, but they're what we call Viaduct tense, right? So Viaduct, even though you're writing business like in Viaducts, right, it's not just a complete free for all, right? There is a rhyme or reason to how we organize the Viaduct kind of monolithic system at the moment, right?
Which is, you know, we have these things called tenant modules and tenant modules have schema, they have code, there's opinions on how you write your schema, their opinions on how you write the code. And, you know, they can be essentially packaged up. There's a little bit of nuance there, but if you look in the code base, right, they kind of look like little services, honestly, it's just that we host them in one larger platform. Got it.
And then this term, re-entrancy, could you talk to us about that? I mean, in the blog posts, you mentioned this is like logic hosted on Viaduct composers with other logic hosted on Viaduct by issuing GraphQL fragments and queries. Yeah. So this is, this is one of the things that I do think is relatively unique to Viaduct in that the whole idea is that as you build out this unified graph and you get more and more data into this graph, well, the less that you need to go elsewhere to get some data and the more that you can use the data that you already have in the graph to build features.
So the canonical example I always give, it's very trivial, but I think it illustrates the point, which is let's say you want to implement a field that returns, you know, the user's full name. Well, the full name is typically the first name and maybe their surname, last name separately, right? And so you kind of have your first name, you have your last name and then you have the full name. You got to combine those two things together.
There's a bunch of ways you can do that. If you imagine that all this data, the user data is stored in some separate service, right? You could query for the user up front and you could take the first name, last name, and then you could just like, you know, have some logic that combines those things together. But in Viaduct, what we encourage is that you are essentially declaring for that full name field, you're declaring data dependencies from within the graph.
So you're saying that for the full name field, I need the first name, I need the last name from the user entity. Viaduct will know how to fetch those things, compute them if necessary, and give that data to the resolver. And that general idea, which you've actually had, you know, since almost day one in Viaduct, the API has definitely shifted, but we had that idea from day one. It turns out it scales really well because as the graph grows and our graph is really big, I think we have 25,000 types and, you know, hundreds of thousands of fields or something like that.
So it's huge. And we have the majority of every of these kind of available online data exposed in this graph. You can often find what you need without going to some service, especially if you're operating, you know, higher up in the layers, right, you know, building presentational type of features. And so you can really write entire features and applications without ever making a direct service call.
And instead, calling back into the graph using this re-entrance approach. Awesome. That's a really good example. I like that.
Very simple, but very effective thing to understand what we're talking about here. You mentioned, well, you touched on the idea that there's a sort of modern version. We're going to get there. I think just before we do, maybe we could just still talk about the just general technical architecture, which again, you touched on in the blog post and the kind of three bits to this, the tenant API, the execution engine and hosted application code.
Could we just walk through those three, just sort of how do they come together to make Viaduct what it is? So as for Viaduct is, you know, really kind of an opinionated graphular server. It certainly grew from those origins. You know, it was really just GraphQL Java from the very beginning and then build stuff on top of it.
But we've taken a much more principled approach as we've kind of started to rebuild pieces of it. So the modern kind of way that Viaduct looks is those three layers, as you mentioned, you know, an engine, this like kind of tenant API runtime and then the code itself. And the reason to structure it like that really comes down to, we wanted a engine that was really lean as performant as possible might be implemented in some other language or framework or whatever one day and can focus on the things that are kind of core to the Viaduct execution model, which is, you know, this like high-performance execution and executing selection sets, which is a GraphQL concept, our concurrency model, how are we doing kind of parallel fetches, matching, which is very much tied into that. So figuring out how to avoid the N plus one problem, maybe caching in certain cases at the very least, intro, request caching and things like that.
And then the tenant API and runtime kind of sits on top of the engine. And it's what is kind of providing that like strongly typed interface, right? So, you know, for those that have looked at the first product, Viaduct is built in Kotlin. We're big JDM shop at Airbnb.
We used a lot of Java, but also used a lot of Kotlin. Everyone that writes code in Viaduct at Airbnb writes in Kotlin. And when you're actually writing code inside Viaduct, you want that strong typing, right? But the engine doesn't care so much about strong typing, right?
It just is kind of just letting data around. So that boundary of like where type data lives and where the engine can deal with just kind of more raw data, so to speak, is actually, actually ends up being quite important because if you're going to build this multi-tenant system, right, you want to avoid sending tight information back through the entire system because if you want to like say deploy a tenant individually from another tenant, right, you need that kind of boundary. And so we actually draw it a lot. It's not a perfect analogy, but we kind of have like engine space tenant space, kind of like kernel space user space, right?
And like I said, it's not perfect analogy, but it keeps us like kind of grounded in that. We need to keep those things separate and we need to create that very strict boundary between those two layers of system. And then you have the host code, right? That kind of is actually what uses the tenant API and tenant.
So let's talk about the modernization journey. If you like, there's, I guess, what's known now as like classic Viaduct and modern Viaduct, is that right? Yeah, it's at least known internally. Yeah.
Yeah, I think it was in the blog post as well. Yeah, yeah, we talked about it publicly, but it's, I think, to the outside world, classic Viaduct doesn't mean a whole lot because I've seen it. Gotcha. There we go.
Yeah. But yeah, and actually most of Airbnb as of the recording of this podcast is still running on class products. We're rolling out modern Viaducts kind of as we speak, going to continue rolling it out in earnest over following year over in 26. But yeah, most of what Airbnb is running right now is still kind of the classic.
Especially the classic, again, this is why that engine tenant split is important because actually we're running the modern engine, but the classic API inside of Airbnb. And so there's like this kind of shim layer that we're building and that we maintain with the eventual goal to push everyone to the modern API. So anyway, so why modernize? Right?
What are we kind of doing here? So Viaducts, the way it started, like I said, you know, kind of started out as a working group. We had a small team that thing like lived as a working group for a long time. We didn't have a real team, you know, but it found a lot of organic adoption inside Airbnb, which means it's successful, basically, because if it doesn't, you know, exactly, it was successful, right?
But I think it was also a victim of its own success. And if you look at how the code base evolved, it's one feature on top of another feature on top of another feature that we added because like some customer came to us and was like, we need this and we're like, OK, we'll help you. And I think, you know, in many ways, it's fine. It got us to kind of where we're at.
But it definitely has some scaling problems, right? Whether it's runtime performance or whether it's like build time and developer loop performance, there's just, you know, be completely honest, right? There have been a lot of struggles. And we've worked around a lot of them.
But what we realized when we started to think about Viaducts modern, you know, a couple of years ago was that there's certain problems that like are just insurmountable in the old architecture. And so the old architecture, just a kind of compare and contrast it with the architecture that we were just talking about with that clear engine tent. But basically, the old architecture, I know of that, right? And in fact, there was a lot of overlap.
It was very unclear when the engine began and when the tenant API stopped, right? And we did that on purpose. You know, I think it sounded like a good idea back then in that what you can imagine it looked like is that we cogent basically domain models, right? And then attached implementation to those cogent domain models.
And it worked really well. It actually had some like pretty ergonomic properties when it comes to just like, Oh, just like go override this method in this class. And then you've implemented your thing, right? But yeah, like I said, kind of scaling that and figuring out how we offer what we're really trying to offer with modern, which is more tenant isolation, more tenant autonomy, but still retaining kind of the leverage that we have working in this like opinionated centralized platform, which is kind of realized that it's just, we're not going to be able to get that with the classic system.
And one other thing to mention that I think will sound very familiar to anybody who has evolved a very large software project. It also became hard for the platform team itself to work on it. And that's what's down velocity, etc. It's like, I mean, this is a slightly nuanced example, but given you've worked in the agency, the last thing an agency ends up updating is its own website, basically because that's exactly right.
Yeah, absolutely. So it's that kind of problem where unless there is a team that has been put together, dedicated to this tooling, which is, I guess, how you can maybe look at it in some ways. Like that tooling just kind of only gets updated. As you, I think, alluded to sometimes it's a customer, i.e.
there's sort of money on the table that says like, we need this thing. So a lot of people go, oh, well, then now we can put time on this thing. Or sort of bandwidth is made available, but it sounds like as with so many companies and projects, it was going to be challenging for that to happen. So that's kind of leading into the modernization and skipping ahead a little bit here, the open sourcing as well.
But yeah, so I'll let you go back to the story. Yeah, for sure. All very good points. High-side 2020.
We could have done some of this stuff sooner, but I think what we ended up, like I said, kind of a victim of its own success. And with that victim of its own success, we spent a lot of time on reliability, right? Like keeping it alive, because at a certain point, it became a very, very critical dependency for Airbnb. And at this point, something like 80% of all traffic runs through Viaduct.
So of all very, yeah, traffic runs through Viaduct. So it can't get much more critical than that. Right. And so the reliability aspect of Viaducts really took the front seat while some of the developer experience pieces took a backseat, much to the jigren, I think, of a lot of development developers at Airbnb who I think, you know, just as a shout out to them, you know, AICU, and I think it's been tricky.
However, we've kept it alive, right? And now, and I would say, over the last couple of years, you know, we've been able to start to balance improving some things about developer experience while also keeping it alive, but building this modern API, modern runtime, or an engine to really set us up for a much, much, much better future. Because it really ended up being in a situation where Viaduct is a little too big to fail at Airbnb. And at the end of the day, people have to use it.
That's how it is at this moment. You know, this is how a lot of I think big company attack evolves, right? Even if it's not the perfect solution, it is the solution that you have, right? And so I see my job, I see our team's job, our organization's job as, well, at this point, we just want to make it awesome, because I think it can be really awesome.
And I think that's what Viaduct's modern is pushing us, we're pushing in that direction. And I think that the benefits that Viaduct provides and has provided to Airbnb are real and ones that we want to continue to capitalize on and not go back, right, to like, oh, just create whatever service you want to create and figure out how to string it all together. I firmly believe that there's a better world than that. And I think Viaduct is that world.
Awesome. Yeah, I mean, I believe that blog post does touch on real world Airbnb performance uplifts. So I do encourage anyone to go and take a read. And just looking at time, I want to make sure we do talk about what the modern that Viaduct is.
And then also when I kind of hear just about the open sourcing journey is something I'll see. We've had a couple of other companies on in the last six months who have gone from closed source to open sources and quite the same because it's like a kind of a new framework over an updated framework, but looking just at the new, like the modern sort of architecture, I believe it's a lot of us about sort of boundaries, you know, strong abstraction boundaries between, you know, what we've touched on the engine, the tenant API and the application code. Could you just like speak to us a bit about that? And I believe it's going from also like a dynamic engine API to a statically typed tenant API with Kotlin classes.
That sounds pretty important. And hopefully kind of, I'm imagining, cleans up a lot of the problems that have just ended up becoming problems with the classic. So yeah, we kind of touched on it earlier. I mean, those strict boundaries that I was talking about really end up becoming critical if you want to optimize the performance and also provide a kind of simpler mental model, right?
So people writing code inside the system. And I would say when I say people writing code, I mean both platform developers, but also developers, right? So at the end of the day, if you build a leaky abstraction, the non platform team folks, they end up having to like understand that abstraction like far too deeply. And that creates a ton of confusion.
That's a great article, by the way, leaky abstractions. Yeah, I was really listening to a very old suffering during daily episode and that our school was brought up. So I went back and read it. We were talking like years and years ago.
One thing I've learned in my career is how deeply important good abstractions are and how hard they are to come up with. And so, you know, I'm not to belabor the evolution point, but I think like sometimes you have to go through a bunch of different iterations of a thing, even a large thing in order to figure out what that right abstraction is, right? So you can think about some of these things from first principles. So anyway, I think with modern, you know, like the main benefit we're trying to give to folks is that it's a much simpler mental model.
Everything is a resolver. We have this like flow chart that we use internally or in some like external presentations, where if you think about what it kind of looked like building stuff in the classic API, you know, it's like this big flow chart of like you choose this thing, this thing, this thing, right? Ultimately, what we wanted was everything to be a resolver, we build that reentry and seek capability into that resolver concept and that's how you're actually coach. We can always provide other kind of smaller abstractions for, you know, let's say fetching data from service or something like that, right?
Like we can always have utilities to make those types of things easier, simpler or less boilerplate. But the core concept of the system is very, very simple. Everything's a resolver. So the other thing that's pretty core to modern is there's this like pretty fundamental concept that we call like async memoization.
And the insight is that when you're building a GraphQL server in particular, just the way that GraphQL execution works, which is kind of this like that first parallel traversal type of way of traversing the query and executing query. And then when you layer re-entrance this re-entrancey concept on top of it, you can end up doing like a lot of duplicate work. And so like take that first name, last name, full name concept I was mentioning before. Well, somewhere else in, like if you imagine the user type, you got first name, last name, full name, full name depends on first name and last name.
There might be some other field in that type that also depends on first name or also depends on last name, right? Why would you want to execute that resolver again? So, you know, if you're within the executing this like node in the graph, right? And you execute that same field with the same arguments again within the same request, you're not going to execute the actual resolver itself again.
So that's a pretty fundamental concept that like most GraphQL servers, I never really seen one do that. And turns out, you know, it has a lot of performance wins. What we really notice again, scaling viaduct scaling GraphQL, we have a massive schema already said that before, we have massive queries. I mean, we have queries that query 100,000 fields or 300,000 fields, right?
In one single query, right? They're returning like a megabytes of information. Now you can say, don't do that. Well, easier said than done, right?
Sometimes, especially as the platform team, we kind of are looking at, you know, these edge cases and like, how do we scale the platform to support those things versus just simply going to people? No, because sometimes they do actually have very legitimate use cases. So a lot of our work, right, around performance and things like that is like looking at, how do you really scale GraphQL, GraphQL execution to that size, right? To that level of complexity and you know, it ends up being a pretty non-trivial, both graph problem, you know, there's some computer sciencey stuff in there, right?
Which is pretty interesting. So by modern, you know, it aims to solve those things in a bunch of different ways. And like I said, you know, it kind of makes it easier to optimize and gives us kind of that stable kernel, right? Of the engine, while we can continue to evolve the API on top.
And then like you mentioned, the strong typing, we had strong typing before, but it was done in a different way, right? It was done in the like, I kind of alluded to it, right? It's you kind of override the domain models themselves, right? In this case, you don't do that.
You're just given value classes, right? That contained the data that you need. We can generate a lot fewer of those, right? Then we were doing for helps us with the build time problem, that type of thing.
And again, a lot of these problems that I'm talking about, these are things that only happen when you scale to the size that we're talking about at Airbnb. And there's a few other companies that are similar size Airbnb that have similar problems to us. I know from talking to them. But you know, most APIs, you know, or GraphQL servers are really any of you guys are, right?
They just don't have these types of problems, right? And that's definitely been a learning and one of the reasons why there's some, there's actually been a lot of work that goes into building the system. Yeah, I think exactly just sort of, or ever size, you know, the massive scale that this is designed to handle is such a huge piece of it. You know, it's, you know, the classic has been sort of battle tested through Airbnb and then obviously some learnings there, which are like flowing through to to modern.
And that's kind of what people should be sort of, if they're thinking, I have a massive GraphQL footprint brackets problem, maybe viaduct is something they should be looking at. And that sort of leads quite nicely into just the open sourcing office. Yeah. I mean, like, was that given when you were thinking about modern or how did that sort of come about?
It was pretty early in the modern story. You know, RCTL, Eric, he's been really supportive of open sourcing stuff at Airbnb. And we have a lot of open source projects that are going to be some definitely non-trivial ones. And I think he was always pretty passionate about, you know, kind of sharing our work with the world.
And it's not just about kind of the typical corporate open source thing of, you know, it helps our tech brand and stuff like that. It also is like a, it's an accountability mechanism almost, right? So it's like a way to share your work with the world and then like get validation on those ideas and be able to talk about them openly, right? And be able to kind of share deeply, you know, what we are doing and why we're doing it and operate and how we operate at the scale we operate at.
So that was his way that he kind of pitched it to me and others early on. And so yeah, so we were pretty much focused on open sourcing it from the get go. We kind of separated out the vibe of modern stuff, you know, kind of in our model repo to make sure that we were kind of not taking on internal dependencies and things of that nature. So we did have a bit of like that flexibility that I think would have been maybe a bit trickier than if we were to have open source classic very directly.
That being said, it's definitely been non-trivial to break viaduct out of the Airbnb bubble. And the real value I think Airbnb has gotten from it from a technical perspective is that that has strengthened the abstractions that we are talking about here because we can't be lazy if we're going to open source this thing, which of course we have at this point, we couldn't just kind of sit back and depend on other things or not think about how to support both potential open source users as well as Airbnb. And how is it sort of if I'm understanding correctly sort of classic and modern, they are different. How internally is that working, shifting things across?
Or is it just as new services come online, they take modern versus classic? Yeah, what we're starting to do is for new use cases, we're having people start to use modern, modern API. Like I mentioned, we actually like have very clearly split that modern engine from the API. So we are running inside of Airbnb, the modern engine, and kind of shimming, like I said, on top of that, the old API.
But yeah, right now it's very much a as new use cases come online, use modern, we'll never force our engineers to rewrite their code to modern. Instead, we will likely use AI tools and things like that. There's a whole like diatribe to go on about how we're doing AI based migrations, Airbnb, I won't go on that big diatribe. But I think that is going to be a pretty critical way for actually getting rid of classic and the code base over the next couple years.
Got it. And just to get listeners like a little bit of more concreteness to the scale, I won't say exact numbers, but we're talking multi-million lines of code in the Vi.co base, right, as kind of models. And we're talking around like a bit over a million QPS of GraphQL operations for second certified system. So it's definitely pretty hefty scale.
Yeah. Before we kind of just look at sort of feature stuff, you're Tevx getting started with Vi.modern, any things to call out there is all pretty straightforward from the repo or? Yeah, it's all pretty straightforward. The thing to call out is that what you see in open source right now, it's not exactly like the developer experience around it and like kind of the, if you imagine like kind of betting it in some server, right?
You're not just going to get Airbnb, Vi.co for free. There is a lot of like infrastructure work that we have not yet open sourced. We hopefully will open source more of it over time, but that there's glue. That's kind of what I'm getting at, right?
That makes Vi.out kind of operate at Airbnb scale. That being said, what is there is truly the core of Vi.doq than what we run internally. So, you know, it's kind of what you see is what you get in that case. Awesome.
Well, yeah, that's just github.com slash Airbnb slash Vi. So, I was saying to anyone curious or going to just like give a spin, you can head over there. I think just as we look ahead, we have managed to get pretty far in an episode without once saying AI. So, this is a change, but it would be good to hear you've got some sort of thoughts around GraphQL and AI and sort of maybe it relates to Vi.doq or maybe it's sort of just around like where is GraphQL kind of going from here?
Yeah, what are your thoughts there? Yeah, these are all kind of early thoughts, I suppose, but I'll kind of give some opinions. I think that with the way AI is going, you know, it's very clear. I'm an avid user of, you know, all the new fancy, agentic tools right every day to code, right?
And I think it's pretty clear that software engineering is going to change a lot. It's not going to go away. It's already changed a lot, but it's going to change a lot, a lot, right? In the next, say, three to five years, you know, and second go away, there's still going to be engineers.
We're still going to code. We might not write a lot code by hand, but we're still going to be responsible for code. And in that world, I think that having really strong patterns of how to build stuff is going to be really, really important, especially in enterprise type of scenarios, right? Large companies, you know, more than, say, 500 or 1,000 engineers working on a thing.
I mean, anyone can go buy code, sum backend and spin something up and have it do something, right? And don't get me wrong, that can actually take you really, really far. I'm a big proponent of like, say in the model if they'll do anything crazy, just like put everything on one server or whatever and scale your startup that way, right? But once you get to a certain size, and again, it's not just the ground, like QTS or something like that.
It's really, you know, this concept of sort of like programming in the large, right? Like you have to work with a lot of people and you have to build a lot of things that all have to work together, right? That's really where a system like Viaduct or there are other systems that are like Viaduct, I suppose, really can play a major role. So I see a pretty clear future for both simplifying architectures in large companies.
Services don't die, but maybe there's a considerable collapse of how many services there actually are. You know, we figure out how to scale these things, scale the number of people working on a single service a lot better than we've done in the past. And I guess not just scale people, but agents. We teach AI how to help us with our operations, right?
We kind of are doing some self-healing type of things, you know, making sure that AI understands our deployments, the observability systems, all those types of components, right? So I think that's why like this move toward building these really stable managed platforms, right? Is actually going to benefit a lot of folks in this AI world because our agents just want to write code. They don't want to like set up all the infrastructure because that's the type of thing that is like, that's not going away.
Like I forget who I was listening to on software engineering daily. It might have been maybe the, I can't remember. Well, anyway, whoever it was, they made a great point. You're a listener, which is always great.
So it was the, it was the ona, the ona folks that are building the USB get pod, right? They're building autonomous agents, right? And it's funny. I actually, one of my other lives there would be when I kind of like took a break from GraphQL for a little bit was building our internal remote development products.
So the stuff that get pod and now ona has worked on is near and dear to my heart. But they made a great point, which is that you spin up agents and you write a bunch of code, right? But like at the end of the day, you still have to deploy the code. You still have to observe the code.
You still have to do something with the code when the service dies, right? And then breaks, right? And you have an incident, right? And while AI can help with all of those things, managing that entire software development lifecycle is not going to be something that a single AI agent is going to be able to do for a long time.
I mean, I won't say ever, but it's going to take a long time for us to get there. And so having these managed platforms be like critical pieces of infrastructure in large companies. And then I'd say, when if you're a small company, you know, you'd be using, you know, the first cells and whatnot, fly out of those, whatever, to build your back and some top of, you just need it. Because I don't think you can do it without it, honestly.
I think that's a really good point. I think that we're probably still relying at the moment on a lot of as we should be, but like human in the loop and where, yeah, taking APIs as an example, like if we're already working with like, is it a grok? I'm not saying grok has to be that service. I just mean literally grok it, grok to the API well, then fantastic.
But there probably has to be a more systematic way around these models understanding like how and where they should go to fetch data and to fetch it safely and so on and so forth. Exactly. Yes. And so that brings me to the like kind of the GraphQL point.
I actually think, you know, there's been some like folks that are like GraphQL is too complicated. Nobody needs it except for the absolute largest companies, right? And, you know, I think there's some truth to a lot of what the critics say about GraphQL. However, I actually think that like, I mean, maybe you don't like the GraphQL protocol or whatever, but the general idea of having a strongly typed data oriented schema that represents all of your core business data.
And then you expose that to a query language or a protocol that both humans and machines know how to easily query and it's flexible, right? That's the flexibility thing is really the important piece here, you know, because everyone can say, well, you just do that with RPC or whatever, right? But it's the flexibility that GraphQL gives you to kind of build these queries that represent exactly what you want when you want it. I actually, I don't know, I'm biased, obviously, but I think like there could be a bit of a resurgence in GraphQL.
And I think that some of you maintain a GraphQL oriented platform. And I think the easier we can make it to write scalable backends using that technology. I think it'll actually benefit folks in the AI world. And I think to your point about accessing data, you know, think about folks as back engineers, especially working at large companies in complicated systems.
Yeah, like are you going to teach your AI about the nuances of every single service and every single API that every single service in your company vents and responds with? Or could you teach it about a unified graph of all of your data and all the relationships already encoded there, things like that. And that seems pretty powerful. For sure.
No, I think that's a really good point. I mean, it's not like we haven't seen technology, resurgence of late postgres seems like the obvious one that sort of kept getting overlooked for a good decade. I would say I happened to work with it for a long time. Yeah, big postgres time as well.
Yeah, but then, you know, along would be became along, no sequel, et cetera. And but I think that's a great place to leave it. I think that I think sort of highlighting where all this kind of fits into the landscape today is great. And I mean, as we sort of called out, if you want to check out vi.gov, that's the help slash Airbnb slash vi.gov.
Otherwise, are you on like pecs, Twitter, anything where you talk to developers, like where to find you? Yeah, I'm an ex at scabby. Usk. And we'd love to talk to you about what skit scaling big day oriented service mesh, you know, at your company.
Awesome. Well, Adam, thank you so much for coming on. Learned a ton today. Maybe in a couple of years, we'll be talking again and we'll see how the AI site is all played out as well.
Yeah, absolutely. I hope so. It'll be a very interesting next couple of years. That's for sure.
For sure. All right. Thank you so much again. Thanks a lot, Gregor.