X-Plane is a popular flight simulator developed by Laminar Research. It features a first principles physics engine, realistic aircraft systems, and a wide variety of aircraft. We wanted to understand the engineering that goes into creating a flight simulator, so we invited Ben Supnick on the show. Ben is a software engineer at Laminar, and he's been working on X-Plane for the past 20 years. He joins the show with Kevin Ball to talk about X-Plane and his career working on the simulator. Kevin Ball, or Kate Ball, is the vice president of engineering at Mento and an independent coach for engineers and engineering leaders.
He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action discussion group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc. Ben, welcome to the show. Thank you for having me. Yeah, I am so excited to get to dig in. This is an area of software I haven't touched before, so I'm excited to go. But let's start with you. Can you introduce yourself a little bit and share about you and X-Plane and what brought you here? I am a software engineer. I've been working on X-Plane for over two decades, which is like 130 years in software engineering years. If you told me back in 2005, you'll be doing this in 2025, I would have been completely surprised. But I started working on X-Plane back in X-Plane 8.
planning to get out of software engineering. My plan was to make a total career change and become an air traffic controller. And I had been an X-Plane user, an enthusiast, a kind of a modder. And I called Austin up and said, you know, I'm doing this training to become an air traffic controller. I can work part-time while I'm doing this. How about one last thing before I go? I'm going to design a new Siri file format for you and get you some data. And like, that'll be like a sort of capstone for all the modding I've done. And 25 years later, here we are.
I actually got the air traffic control training. And while I was waiting for the FAA to call me back, there was just more to do on X-Plane and more to do on X-Plane. So I kind of stuck around. I started working on the rendering engine. And by the time the FAA actually called me, we were knee deep in X-Plane 8 and making progress. And I realized I was on this flight team track. Yeah, I would love to sort of dig in with you a little bit on just even what it looks like to be on a project for 20 years, because that's a pretty unusual experience. But let's start with X-Plane. So how would you describe what is the core architecture of a flight simulator like this? A flight simulator.
Probably from a software engineering standpoint, it looks closest to a AAA game. That's kind of where our DNA comes from. But our business concerns are really, really different. So we're kind of outsiders. If there was a room with all the AAA game people, we'd be standing in the corner, maybe not one of the cool kids. Because we have the same requirements for high performance, as much graphics as we can possibly give them. Realism, physically-based rendering has been huge for us, like the whole games industry. But to a certain extent, games live and die based on being fun. And if you don't do that, you're toast. Most games are going to be toast anyway. If you're going to do a sequel, you might just upgrade the whole engine. Whereas...
We are kind of a platform because people make content for the flight simulator. People make aircraft. People modify airports. The modding community is so big for us that it's almost like we're a platform as much as we're an app. So we need to have continuity basically as much as we possibly can, right? And that's where we differ from the games, where a game engine programmer might say, well, we've got this new technique. We'll have to modify all the art. And there's these constraints. So I get my art director in and say, can we do this? And the art director says, all right, it'll cost this much. And the producer says, well, we can afford it. And then they tell all the artists, go modify these things. Never use the alpha channel. Here's the restrictions. Here's the bugs you have to work around.
That's how AAA games get made. We can't do that. There's aircraft that people have already spent man years on that are already selling, and the stuff is what the stuff is. So we have to ask ourselves, how do we upgrade the engine and still keep this content working? So it's kind of one foot in AAA games and one foot in platforms, which is sort of a unique niche that we sit in. Yeah, that's really interesting. So before we dive into any of those, are there any things within the core software piece that would be not recognized by somebody coming from a gaming background? I think from a gaming background, it's all going to look pretty familiar. Our physics engine.
is completely homegrown. Austin Meyer, our founder, benevolent dictator, the only laminar employee for, gosh, like at least 15 years, developed the physics engine for himself to simulate aircraft. And his background is actually in aeronautical engineering. So he has this deep understanding of why airplanes fly. So the physics approaches used there are very different from what you might get in a Havoc or some kind of general purpose physics solver. Those systems tend to be very generalized and they're trying to give you a bunch of very simple Lego bricks.
to do some physics in your game, X-Plane really deeply understands airplane things. Like we model airfoils, we model wind flows, we break the wings down into elements. So if you came from a Gain engine, you go, this is the physics engine. And then you probably go, this is not like any physics engine I've ever seen. What's going on here? And it's because it's very domain specific. That makes a ton of sense. When you have a realism requirement there that most games don't have, it doesn't have to just feel real. It has to accurately represent what somebody is going to experience in one of these planes.
In the case of the physics, it actually goes a step further because X-Plane is not only used to simulate aircraft. And that's the lion's share of our use case is people training for real aircraft with X-Plane and people playing being pilots at home. We have gamers. We have a lot of older users who used to be real world pilots and they've lost their medical. You know, if you've got cardiac problems or eyesight problems, you can't legally go fly a single engine plane. And these people discover that they can rebuild what they had in simulators and still get some joy out of flying. But we also have people who design new planes.
using X-Plane. And that's unusual even for flight simulators. You know, electric VTOLs is an area that's had a lot of growth recently because the batteries have gotten good enough. And once your engine is a simple, inexpensive, small electric motor and not a complex, unreliable gasoline or diesel engine or turbine, you can put four engines on. You can do all sorts of novel designs.
X-Plane can give you pretty good predictions of how that's going to fly just by inputting the design. We didn't need to know the answer ahead of time and then spit it back to you, which is the conventional simulation approach. We can actually be predictive. And that puts an even higher requirement on the physics because we have to get it right because no one's going to tell us what the answer is. That's fascinating. So I'm kind of curious then, what is the format for those airplanes look like? Is it similar to what someone is going to see, for example, as they're speculating things out for manufacturing or like what goes into that? The format is proprietary to X-Plane and it's pretty specific to X-Plane's needs. So we have an aircraft format with just, it's really just a giant.
key value property table full of stuff. And we need airfoils and their locations. We have a separate subfile to describe the airfoil. The airfoil is described as a table because we don't predict what the airfoil will do from first principles. That requires computational fluid dynamics, and that's a whole other ball of wax. So once your airfoil has been measured, maybe you have a wind tunnel, maybe you did CFD, maybe you just downloaded the answer because there's a lot of airfoils that fortunately are published. But you've said, I'm going to use this airfoil. You can tell X-Plane, here's my wings. These are the airfoils. This is the shape. There's a lot of shaping you can do. They can twist. They can be tapered. They can move while you fly. They can be swept. Here's the control surfaces. This is how much of the space you have. And so all of this detailed data really describes specifically...
how the aircraft operates in aircraft terms. So you describe flaps, you describe ailerons. And once you have all this data, including, you know, here's your engines, here's your engine specification. X-Wing can take this and we can simulate the vehicle from the ground up. So we're going to compute how much force did this jet engine put out and where on the airframe did it do it? What was the wind coming into the propeller? How was that wind modified by the propellers as it goes through? Was it redirected? Is it going to then hit the wings after? Is it going to hit the tail? Is the wing going to redirect the airflow away from a different part? And essentially we have this collection of parts and we try to derive from the parts what is the net effect on the whole. So there's a number of aerodynamic effects that are emergent and we don't...
code them in. There's this tendency of aircraft to pull to the side because of the twisting motion of the propeller. And we didn't say this is how much it pulls. We said, what does the air do? And this kind of comes out by itself. That's fascinating. And it's fascinating because, as you highlight, you don't want to do the CFD underneath it because that's super expensive. But you are doing, it sounds like, a fairly low-level simulation and seeing those emergent effects arise out of it. How do you make that perform it? I mean, you guys run all the way down to running on an iPhone or an Android. What do you do to make that all work? We run on the mobile phones, but...
it's 2025, the phones are really fast. Like the phones are much faster than the computers that were running X-Plane when I started, by a lot. Even an Android craplet, like a low-end Android device, is still much better than what we had with X-Plane, say, 8, which would have been a single core, two gigahertz Pentium or something. We don't live in the dark ages anymore. The physics actually take a relatively small portion of the performance budget of the aircraft. And I think the main reason for that is that the actual number of stations you need to simulate the aircraft...
isn't that bad. If you can do 40 different simulation points on the wings, that's more than enough to capture the individual effects. But 40 is not a big number for a computer, right? Like we kind of think of it as like, oh, that would take me a long time to calculate. But when computer programs are slow, it's usually because the data input stream is really big. There's 10 million of something. So as X-Plane has matured, the physics has started taking a smaller and smaller amount of the CPU budget. And we've much more spent the Moore's Law surplus on graphics because our users have a unquenchable thirst for it to look more like the real world. There are...
some physics things we do now that we could not have done in the single core Pentium 4 days. So modern X-Plane has fairly complicated interactions between the different surfaces. When the wind stream is deflected... by one part of the aircraft, that carries through to the next. And those propagating effects are kind of a higher, bigger order than individual blade theory. So if you went back and looked at the original iPhone version of X-Plane back on the 3GS, I think it was, which seemed like a pretty nice phone back in the day, but now we would see this as a small computer, or even the original Pentiums, we were doing all the elements, but we were doing no propagation. And that is a computationally easier problem. But even with propagation, this is still a much more manageable problem than draw everything on the entire earth within a hundred mile view, which there's a lot of stuff there. And users have been always happy to have more. So that's been where we've kind of sunk our dividends.
That makes sense. Well, and as you highlight, that is a similarity to gaming, right? You can never get good enough graphics. You can never do it. But with your constraint of, oh, I have to be backwards compatible to all of these old files, how do you take advantage of those Moore's Law dividends in the graphic space? So there's a couple of things. First of all, it's a window of compatibility. And the window might be bigger or smaller depending on kind of how much stress we're under with that compatibility. I would say having good separation of concerns has helped with the compatibility. So I try to think of things in terms of what is this content modeling and how are we interpreting it, right? If you say this model.
is a drawable geometry with a lighting model attached, your compatibility options are zero because it is exact in the spec of what this model is, how it will be drawn, and so you're not allowed to change things. But if you say this model is a physical material existing in the real world with some properties on the material, now the lighting model is not part of the model, right? And so you're allowed to make the lighting model better. So anytime we can use the real world as a reference... for the content, we have the option of doing better. And anytime we've said the content's behavior is exactly specified, we're stuck. So we can do a better job of simulating the aircraft if you tell us what the aircraft is. And if you tell us how the aircraft works, we're stuck. It better work the same because you told us the how. And in the graphics field, there's a pretty good separation between the kind of what of what's in the world and how it looks. So what we found is that in X.11, we moved to physically-based rendering.
we needed people to specify their materials in a physically-based manner, and we could look at the previous non-physically-based manner and say, this is the least bad approximation we can do. We can say, this color texture looks a lot like an albedo, so that's the best mapping. And if all else fails, let's do the least bad compatibility path we can. But then going from x.11 to 12, where we went from it's physically-based to it's photometric, meaning not only is it a... set of math on the lighting that is based on what real world optics do, but we're going to try to have all the calibrated numbers be in real units. Like we're going to specify the intensity of lights in candela. We're going to specify the luminance of a glowing thing in nits. When we did that, to the extent that the previous content was still giving us physical materials, giving us albedos, we could interpret those correctly because we weren't changing those rules. So a lot of that content just worked. And we have to look and say, what don't we know that we wish we did? And the answer is, well, it's got a self-illuminating texture and they never told us the luminance. So we're going to say, okay.
For all aircraft surfaces, we're going to assume the luminance is 2,000 nits for everything in the cockpit. And that's the least bad compatibility we can do because we know it's an aircraft. We know the pilot can see these things. What's a realistic value they would need for the simulator to function? And you need to see your displays during the day. 2,000 nits. It's not perfect, but it's pretty good. And in that way, we can kind of keep stuff working as best we can. And every time we make a change, we have to ask ourselves those.
There's one other aspect, which is a lot of AAA games use a lot of artist time to make things work, right? They have kind of a mission, they have a budget. And if the art director says, I want all the rainforest leaves to look like X, it might be the way they do that is they have code. They might write an algorithm that processes the art, or they might tell the artist, just go do it by hand. They can make that decision. In our case, I try to be conscious of how the dividends in computer graphics are driving up the price of making content. Because when I started as a modder, one person could make an airliner the entire thing in about six months. It was half a man year to make an aircraft because there just wasn't that much you could do.
The mesh came from the physics model. You couldn't attach custom meshes from an authoring program. You got about 1K of color texture and that's it. There were no physical materials. The cockpit was a 2D drag and drop thing. It wasn't a 3D interactive cockpit you could click on. There was no code to write. You got the systems in X-Plane. And so there just wasn't that much to do. And that's all that the computers of the day could do. And as the computers get better and our users expect more, one of the problems has been that the authors making airplanes have sort of borne the brunt of this and that the labor to go into the airplane is going up. So when we're going to add things to the feature, the other thing I think about...
besides how am I going to map this content, the old stuff, is how much labor are we going to bring in? Are we doing something that's going to be automatic and just works? Or are we going to ask them to go and do a tremendous amount of work and kind of erode the margin and the viability of making this content at all? So, for example, we added proper 3D rain to the windscreen in X-Plane 12. And we take the airflow over the aircraft from that CPU physics model, we put it in the texture and send it to the GPU, and now we can drive the particle system realistically, which is...
totally fun because you're sitting there on a rainy day and the raindrops are falling down the windshield and then you crank the engine and they go and they go right off the windshield. It's very satisfying. And it just works because the GPU has the real shape of the windshield. It has the real flow of the engine and we can just do the simulation. But it's also a feature that's relatively cheap for the authors because the shape of the windshield in 3D was something they had already built for us. They were already building the 3D model and we're reusing it. So this is very cheap. When they do the windshield wipers, we need the path of the windshield wiper in the 2D space of the particle system, which is kind of a hard thing to get because the geometry of the windshield wiper...
is potentially very weird. That mechanism could be almost anything that people engineer. And every airplane is a hundreds of million or billion dollar engineering effort. So there's no reason why they wouldn't just build everything bespoke. So we looked at this and said, there isn't a way for us to get this information in real time from the 3D mesh that's even remotely performant. So we bake it. We have the path of the windshield wiper in the 2D space of the particle system as a color gradient where the different grayscale levels tell you these pixels would have been affected by the wiper at this.
timescale. So now we're starting to make it a problem for the artist because you have to give us this special, magical, exactly right formatted texture to make it easy for us at runtime. What we did do is we built a script into Blender to produce this automatically. So you could start with your windshield wiper animation, which is something you already built. You do a very small amount of annotation. You say go bake and then Blender can sit there and go chunk, chunk, chunk, chunk, chunk, move the wiper, find the pixels, fill them in.
and export them. So we can still try to protect our authors from having to build something by hand. But in this case, it's going to be done ahead of time. But we're looking at the authoring cost for the entire feature and asking, by making this possible, by making it something our users expect, are we driving up the cost of making add-ons? Or can we give the authors tools to do this in a reasonably affordable manner? And now their airplanes are better, but they're still able to produce a product whose price point isn't really changing with hopefully a similar amount of man hours.
It's fascinating because it's an area I hadn't thought about as much so much of software, right? The downstream dividend of things getting more powerful is you can write software cheaper because you can skip steps, you can use scripting languages, you can do all of these different things. But in graphics, the expectations have risen faster than the tooling. That's right. I think it's telling that in software, what have we done with Moore's Law? We made our own jobs easier. My father...
was in hardware for his whole life. And he'd kind of like scowl at me and say, you kids with your virtual functions, you know, we didn't make the chips faster so you could just waste all the cycles, right? But the truth is, that was kind of the most useful thing you could do would be to make software cheaper to make or to make software more reliable. I think the dominance of managed languages is telling us something about what people need out of software, right? And it's very expensive to have humans write bug-free, unmanaged code. So maybe we use more as well and not do that. On that topic, what is the underlying software stack y'all are using for Explained?
X-Plane is coded entirely in C++. It's pretty much a big monolithic pile of C++. We use a smattering of library code for kind of the things you would expect. Like, we've got libpings, so we can open pings. We've got Racknet for a networking sublayer. We have things like that. But it's very much ours, almost surprisingly so. And I've had people ask, why are you not using SDL2 for your cross-platform layer? Like, why do you do it yourself? And the answer is, well, the code is 30 years old, and there was a time when the thing you're asking us why we didn't use didn't exist. So...
At some point, I think we've erred on the side of keeping what we had, which was already in production, which is a trade-off. Maybe our solution isn't as good as a new thing that came out as library code, but also there's often a lot of client code on top of a library, which is why it's often cheaper to rewrite the library or shim it than to go rewrite all the client code. We were OpenGL-based for a very long time because OpenGL was the first hardware API that Austin used for rendering. There was a time when X-Wing was software rasterized on the CPU, but by the time I got there, it was OpenGL, and OpenGL runs on multiple platforms. When we wanted to move to Vulkan...
and Metal to modern low-level graphics APIs, we had to make a decision about whether we were basically going to just shim Vulkan and Metal under what we already had, or were we going to go rewrite a chunk of the software stack? And we chose to do the hard thing and rewrite a chunk of our own code. I think if we just wanted to say, hey, it's on Vulkan, that could have been an eight-week Skunkworks project because the API is very narrow and you just shim under it. But all of the architectural contortions that you pick up from being a long-lived OpenGL application... would have been still there in X-Plane. We would have had a great tool in Vulkan and gotten none of the benefits because we'd be holding it wrong pretty much always. So instead, I think we spent three or four years of continuous work on this, Sydney, Houston and I. And what we got in return was a real rewrite of the rendering engine itself under modern principles. Vulkan lets you understand the performance cost of everything you do. Whereas in OpenGL, you make a call, some things happen, some pixels appear later. That's basically the whole contract. It's very loosey-goosey. And because that abstraction is not well suited to modern hardware and you can't fault SGI.
They failed to look in the crystal ball in 1991 and predict what would happen in 2025. Like that's, we'll give them a mulligan. But because of that, OpenGL doesn't have good matching between what's expensive and what you do. So realistically, what happens in an OpenGL driver is the driver follows along, it squints, it tries to understand what the app is doing. It does a lot of work to reprocess the information. And at some point it has to go do a lot of work. And a lot of the times that's at the very first triangle you draw, they go, okay, now we understand the whole problem. Let's go compile some shaders. Let's go make sure these objects are in the right part of VRAM and all this work that you wish you'd done.
at load time is getting done at draw time. Our users, especially professional ones, really want X-Plane to be like a metronome. Rock solid, 60 fips, don't stutter. Stutter-free operation is a requirement from the FAA for professional sims. So a driver that will sometimes stop and go compile a shader because on this particular hardware that's required because of this state in a way that you couldn't predict, it's really hard to hit that 60 fips limit. But we didn't have any code in X-Plane to go prepare the shaders in the background because there is no prepare shaders operation in OpenGL. You can tell you're building the shaders, but...
it's going to half compile them and then wait and then do the other half later when you draw. Because in OpenGL, the specification for a shader is incomplete. In OpenGL, you give it the source code, but you don't tell it the render target, you don't tell it the fixed function pipeline, it's missing most of the information. So it goes and it does it later when it has the info. So we had to restructure X-Plane and not just switch to Vulkan to get that actual performance win. So we went in and we rewrote a lot of code. It took a while, but in hindsight, it was worth it because we have something much newer, much better matched to the actual hardware and putting new features on top of it has actually been very straightforward.
I think that's an example of what happens when you have this very long-term view as well. And you mentioned, I think this might actually be an interesting time to dive into what goes into maintaining a project for 20 years. How did you go about that three to four-year migration in terms of planning, phasing, isolating things so that you didn't break stuff that was running? What did that look like? Well, Austin has always said that he wants our time horizon to be infinity when we think about the efficiency of what we do, right? If we say, oh, there's this thing we can do, it's going to be really great in the first three months, but it's going to be inefficient.
in a 10-year horizon, like it kills him to think that we're wasting our 10-year time to hit that three-year thing. Like he wants us to think about the long game. And that means we can kind of think about a roadmap of where we want to move the code. So when it came to the Vulkan port, I think it started by trying to understand the technology. We had a pretty good idea of what we weren't happy with. Like we could really see the pain points of what was already there. So we came up with a sort of refactoring roadmap for which pieces we would rebuild and when. And we had to look at how are we going to do this in pieces? How are we going to never have the app so thoroughly broken that we can't test? Because at some point...
If you do that, you end up breaking a bunch of things never noticing, and then it's really hard to put Humpty Dumpty back together again. So there's sort of a huge value in never breaking the universe. We also looked at whether we had to stick with OpenGL. One of the biggest arguments we had was, do we have to keep OpenGL working when we ship Vulkan? My answer was yes, and this is what we ended up doing. We ship Vulkan as a free update to version 11 while keeping OpenGL. And what this got us was the ability to have a very, very long beta. Like, my God, I think that beta might have been nine months, which is atypical for us. But we just needed to have Vulkan out there and...
find out what it was like on real machines. Because something you get from a shipping product is, you know, what's this like out in the wild? Because the wild is really different than your lab machines. And in the case of Vulkan, we had no idea what we would find. And, you know, we could not have predicted what we would learn from that beta. So having the old system running at the same time meant there was no customer going, I paid $60 for this and it's brick now and I want a refund or I'm going to fight with my credit card company or God knows what. Like, it gave us the time and the space to do the exploration. I think if we'd said, okay, X-Plane 11 is OpenGL, X-Plane 12 is Vulkan.
it would have been an easier engineering problem. We would have said, hey, in this new thing, our requirements are simpler because the union of OpenGL and Vulkan is a little awkward. But on the other hand, God, it better have worked for the first time when you hit go because you just paid for the new version, right? And there's no safety net. You can't go back to OpenGL. And in hindsight, we did not hit that. We needed that nine months of beta to understand how Vulkan performs in the wild. What happens in Vulkan when you're doing memory management and the user launches Chrome and Chrome goes and puts stuff in VRAM, right? Like Vulkan tells you how much...
video memory you have. And by the way, it might just cut in half and you have to survive that, right? You have to write your own code to cope with that situation, preferably without exploding in a million pieces. You know, we did not have enough information in the company. We didn't have enough experience with Vulkan before we'd written Vulkan to correctly predict that. So we had to try it in the field. So knowing that we could then say, okay, which pieces are we going to roll out? And what we found is that there were refactors we could make to our code to use OpenGL in a better way in ways that probably didn't make a huge difference for an OpenGL driver, but we're also fitting the shape of Vulkan. And once we made those refactors and shimmed all the
OpenGL code into an abstraction layer, then we could rebuild that abstraction layer for Metal on OSX and Vulkan on Linux and Windows. And then we could kind of hit the lever and see, you know, can we ship all the trains over to the new track? And since we were going to do Metal and Vulkan for Mac and Windows, we knew we would need an abstraction layer no matter what. So having it also handle OpenGL and OpenGL ES in the mobile market, which we thought we'd be on for a while, that was sort of the natural path. And it gave us a way to...
do the refactor in two pieces. First, kind of making a nice API boundary where previously the OpenGL and client code was completely fused together. And once you completely cut out that boundary, then you can do a shim and you can switch between them without recompiling the app. And that kind of makes the refactor trackable. Yeah, no, that technique of introducing a boundary and new abstraction layer, decoupling things, and then introducing these two different ways is that I found that to be highly effective as well. I love what you said about the sort of long-term time horizon. I feel like that's something very few of us in the software industry are in companies that support that, but that's incredible.
Yeah, I think we're very lucky to have that freedom. And it's one of the ways that we're different than games, where often a game's time horizon is we're going to do one title. And while I have not worked in the games industry, my brother does. We have people who work for us who have been in games. And you sometimes see some pretty scary code because, and people write it for good reasons, right? Like we have to ship this, we have to ship it on this date and no one's ever going to come and refactor this. So give me the big rule of duct tape and here we go. And, you know, it holds together and it ships. In our case, we know we're going to pay for ripping the duct tape off. And sometimes you end up using it anyway, but we have to think about what it's going to be like, you know, in a week, in a month, in a year. Yeah, I wish more software business models.
took that approach. I think it is definitely a healthier one for the code and for the engineers. You mentioned testing. Can I ask what your test stack looks like? Our test stack, we have a couple things, and it's never enough. I would always love to have more testing. We have Catch2, which runs unit tests, and we have partial coverage of our lowest stack of software. We've been catching up on test infrastructure because when the company was very small and X-Plane was very small, testing was kind of, let's give it to the users and see what they say. For a very long time, Austin was the only person working on X-Plane. So testing was, he coded it, he tested a little bit.
And they gave some people and they emailed him at his personal email address if it wasn't right. And that was everything. We built that up a bit. So we have catch two for unit tests. We have a Python-based runner of a scripting language because any...
sufficiently large application has a command shell in it, and X-Plane is no exception. And the command shell, besides providing kind of strange debugging tools and things, also provides a whole bunch of stuff for testing. So we can write these automated scripts, and some of them will do things like we have one that opens up every aircraft with the engines off and does the procedure by invoking commands to start the aircraft, monitors the parameters of the airplanes it starts, and fails the test if something's out of whack. So if somebody makes a change and now the electrical system is broken and the battery doesn't cause there to be amperage on the bus... test fail. So we have a suite of those. They can also take screenshots so we can get image output. And we have Daniela, our sound engineer, does a lot of web stuff too. And she built this fantastic browser-based diffing program so you can pull up the before and after and spot the different regions. So we have black box testing. We have unit testing at the low level. We have some integration tests in the middle. We have a tool you can run on every airport and it will catch invalid frequency conflicts. So when you import new airport data, you can run a one-time check. I would say the struggle we have is that because X-Plane is a platform,
you can make an almost infinite combination of aircraft. And a lot of the bugs that are hard to catch is, well, it turns out that if you have an airplane with three engines and two are electric, one is gasoline and you have two clutches then, and that particular combination is difficult to cover. And what we don't have is a very, very, very wide suite of aircraft. X-Plane ships with, I want to say about a dozen or so, really carefully built handcrafted aircraft that our team does a great job on. But because they put so much work onto them, you know, I can count them on my fingers. And so the test coverage value is less. And at times we've built simple airframes just to do testing. But I think that's an area where the coverage is poor.
We have two QA engineers and they both kind of supervise users of the automated testing resources. And they also do manual testing. We do have tests where there's a, you know, old school test plan, right? It's a doc with words for what they do. And in some cases, you kind of need a human to go do this because automated would be too complicated. And most importantly, like they just sniff out bugs. Like they're really good at that. You know, we joke about, you know, well, how many milliseconds is it going to take Danny to find the first bug? And they, you know, they'll come in and they'll kind of shake it and get that spidey sense that a good QA engineer can get. And that helps us kind of catch things that would fall between the cracks that the regular test infrastructure wouldn't catch.
There are people who just seem to have a talent for breaking things and finding things are broken. It's uncanny when you watch them. Actually brought another thing to mind. You mentioned that many different combinations someone could do in these third-party content, whether it's an aircraft or an airport or something like that. Is there a correctness validation that makes sense in those files or things that are describable in the format, but maybe not physically possible? What does that even look like? There are semantic validations that we can do and that we do do.
I would say my own work in the scenery code has in the past been pretty slim in terms of semantic validations, mostly due to schedule pressure. And I've been trying to pay my debts off and add more validation. You know, Austin has a fair number of validations in the physics system because he doesn't want to deal with junk data coming in. But there's also like a gray area between this is clearly wrong. Like you cannot have a stall speed of zero. There's got to be a stall speed versus this is very silly. Like, are you sure you really mean this? And sometimes we say, well, we saw this weird bug and we looked at it and the data was crazy. So we made the validation a little more skeptical. And then we hear back that, no, no, I built an RC plane.
You know, it's one foot long. So all the numbers really are small. This isn't a typo. Like, you know, this is actually how big the prop disk is because it's made of balls of wood. So there's definitely a tension between how much we can validate the model and how open-ended X-Plane can be to very strange stuff. Yeah, for sure. And if people are using it to test out things that they might build, right? Like this can be built, but falls off out of the sky is a valid outcome. That's right. Yeah. And so flies well is definitely not a validity parameter. On the sort of flies well side, another thing that you kind of briefly mentioned is...
FAA certification, right? So X-Plane is not just for hobbyists trying things. It's not just for people manufacturing, but people are actually using this software to certify themselves to fly. What's involved in that? What has to be true at the software level to enable that? Surprisingly less than you'd think, but more than you think, not in the areas you might expect. So first of all, they don't necessarily certify software. They certify devices. So we are a software reseller and we sell a professional version of X-Plane to an integrator. And they're going to select the computer hardware.
select a version of our software, build a simulated airframe, build the controls, maybe a fiberglass enclosure. Maybe they're going to buy a motion platform if this is a high-end system. They might have multiple computers networked together running separate displays to get more visuals. And then they're going to take this entire thing to the FAA for certification. And...
The FAA is less picky than you would expect about the accuracy of the flight dynamics compared to the real aircraft, but they are often very picky about the operation of the simulator. So this one always cracks me up. If your flight control hardware is not working, meaning your joystick is unplugged, your simulator has to not start. It has to log that as an error and not run. And the inspectors will come and yank the joystick out of the USB port, and if we keep flying, they'll fail you. And I've always thought to myself, like...
Who's going to go in and do flight training time? And the joystick isn't working and they didn't notice. Like, what world do we live in? But that is a requirement, like, because otherwise it's clearly broken. So, you know, we can get tested on that. So the specific regs are quite long. And my experience with the certification process is that it's a little bit ad hoc, that different inspectors have different parts of the regulations that they will check. And so we sometimes get calls from integrators saying, oh, my God, we got called on this thing. It'll be like, we haven't ever seen that before. But, you know, sure enough, it's there. So we try to make the most useful, sane, generally highly functional simulator. But I would say that the regulatory structure.
doesn't match X-Plane's business model because we have a tool that's meant to simulate all airplanes. And the really traditional simulation model is we're Boeing. We built the 737. We spent about a gajillion dollars on the R&D. We're going to get, you know, a eight-figure fee every time we sell one. And we're going to make a level D maximally certified 737 simulator that we have made exactly like the real airplane in every way, including using the physical flight deck in that airplane. And this simulator is a one-off device to simulate this one airplane. And it's going to have...
God knows, seven, eight, nine figure price tag too. And the certification process is kind of aimed at that, I suspect. Like, here's a simulator of this thing. So what there isn't is a way to say, here's software that's generally useful. We certify that in the general case, it is a jack of all trades. So we're more of an arms dealer. Like we're selling shovels to the prospectors in terms of this process. And I don't mind that because dealing with the regulatory requirements is its own complicated thing. So we're happy to do the software engineering we're good at and let somebody else do the certification process. That makes sense.
From a device support standpoint, does that introduce, like these are custom hardware, but underneath it, they're just running devices that you already know how to manage? Or what does that look like? Yeah. So we are on Mac OS X, Linux and Windows in the desktop space and Android and iOS in the mobile space. So we're running on a lot of operating systems. And most of the integrations are using Windows. Occasionally we see people using Linux, but the professional integrators are usually pretty flexible about doing what we ask them. They want to succeed. They want to have a system. They get certified. They don't want to spend a lot of time screwing around.
If they're using Windows and we say, well, upgrade to Windows 11, they'll do it. If they're using Linux and we say, use a different distro, they'll probably do it. We can recommend hardware. So in terms of the overall machine, usually it looks a lot like an ideal machine in terms of the specs are very good because the cost of the hardware isn't that great compared to the whole project. The operating system isn't crazy. It's hopefully not loaded down with a bunch of really crazy mods and add-ons and malware or something. So it's a pretty good starting environment. We use USB HID for our...
hardware input. So it's relatively straightforward to wire up just about anything to explain in terms of joystick or control input. We also have a DLL-based plugin system, and we're starting to do REST and WebSocket-based communication with the SIM as well, and we've had UDP input for years. So they have a couple of tools they can get that they can use to get custom data into the SIM, but a lot of the times their joystick hardware is going to just work. Whether they manufactured it themselves and got a hidboard, or they bought something from another manufacturer, usually you can make a config file to further specify how we interact with the joystick, and you can tweak a lot of the settings in our settings, but it's pretty flexible, so that usually goes well for them.
That makes sense. We've talked about a few different places where you have this interface layer that can be configurable and can let you work across the different types of hardware. Graphics, obviously, being one input device. Are there other areas where you are ending up handling this diverse set of devices? I know a lot of people, you know, there's a lot of standard libraries out for doing that, but you rolled your own thing. So what does that look like? We generally try to keep the platform-specific layer as thin as possible because anytime it's platform-specific, we're coding it five times and not.
one time. So, for example, our user interface is rendered into the app in Vulkan and Metal, using our own UI kit. And so we're not writing separate Win32 forms and Mac Swift UI code. And then we need something else on Linux. Maybe it's GTK or GNOME, or there's a lot of chaos over there. We need something else on Android. We're not doing that. We're willing to say, the X-Plane UI just looks like X-Plane. It's not going to be platform native, but we got to write it only once. So we keep that hell very thin. We don't need that much casing because...
four of our five operating systems look a lot like POSIX and Windows isn't that different anymore. So the platform story has gotten pretty good. I believe the HID code we're using comes from an open source library, which does give us separate HID paths on multiple platforms because how you talk to the USB driver varies, but that makes it uniform. Vulkan and Metal. look different from each other and are abstracted, but Vulkan plays on Windows, Linux, and Android. Metal is on all of Apple's platforms, so we get pretty good leverage from that. We have a thin set of file system wrappers, but we don't need much. We have a wrapper around the memory mapping primitives to get files in, but again, it can stay pretty thin. And we're not doing tons and tons of file operations, so our needs are relatively straightforward. Racknet abstracts the network stack, and we also have a pretty thin socket wrapper, but Windows looks enough like sockets. So overall, I would say the...
platform layer we can keep it thin and it's been okay we need a way to get a native window and get some mouse input off of it but once you write that once you know the amount of change in that space is pretty limited you know windows code really pretty much runs the same from you know when i started it was still win32 i mean people slag on microsoft but they have kept a platform running since forever and i think people don't realize like how much they're holding up by doing that You know, we feel it a little bit because we've been on OSX where they've changed the ABI maybe like four or five times. They've changed the UI kit several times. They've changed the language multiple times. And you have to rewrite your stack and your user aren't going, oh, I'm really excited to get the exact same features, but rewritten differently. Like you don't get anything for that. So, you know, as much as we sometimes say mean things about Redmond, like I do recognize the value of having the thing keep working so we can focus on features and not just rewrites. Yes, no, they definitely understand their business consumer, I think, better than just about anyone else in the space. You mentioned features. What's on the roadmap? What types of features are you all working towards?
Hopefully, by the time people hear this, but as we are recording, our newest free patch, 12.2, should be coming out. That's a patch that's very graphics intensive. We try to have the patches be in sort of a theme area. Marco, our product manager, kind of looks at things and tries to gather up the features and, you know, find things that will be good for users, but also hit a theme and limit the testing, right? If we touch everything, we have to test everything. So the most recent one is a graphics one, and there's changes pretty much everywhere. I don't think I can just rattle all of them off because there's been so much. But this is the end of a pretty long...
probably even a year-long effort to refine the rendering and lighting stack of the sim and really do write what we kind of prototyped in X-Plane 12.0. X-Plane 12.0 was a difficult release for us because Microsoft had come into the market with Flight Sim 2020 while we were finishing the Vulkan port. And we were completely taken by surprise by this. They really stayed in stealth mode right up till their EA announcement of the new product. And it was just, how soon will you guys have the new version and how many features can you have in it? And that's, you know, that's a pressure cooker for release. So we did all new photometric lighting and clouds and...
Quality was something that kind of got thrown on the floor to get that out there and just show that we were in the marketplace. And people used it and they could see the potential, but there was also a lot of weird stuff. And so now the team has had the time to go in and really do a great job of working all the bugs out of the lighting and going back to the real world references and using new algorithms. And it looks like a whole new sim. It's really a big upgrade. People who've used it in the company, you go back to the shipping branch and you're like, what happened? Now it just looks flat. And it's funny how in graphics, everything looks fine until you see the new thing. And then we hedonically adapt so quick and you go back and go,
we thought that was good. And I know there was a time when we first saw X-Plane 11 and went, wow, that looks so much better than we could have imagined. But then once you've seen X-Plane 12, you go back to 11 and go, yeah, that doesn't look right at all. Look at all the things we didn't do, right? So the lighting, the color saturation, the clouds have gotten a big update. The interaction between them has been reworked. It's going to be a really nice visual update. Behind that, what I think could be 12.3 for us, but...
Your mileage may vary. We have a number of features for systems and weather that are nearing completion. So better modeling of how clouds work, better interaction with real-world data when we can pull it in. I believe we will finally ship the new weather radar, which is a very accurate simulation of the radars that are on aircraft so that the pilot can see storms ahead that runs on the GPU and probes that weather data, but then simulates the returns and the occlusion. So you get a very accurate view of when there's a storm cloud ahead of you and it's big enough, the pilot doesn't really know the weather behind it because the radar doesn't penetrate it, right? And we simulate that. Or if there's a mountain in a way, you actually get a radar return.
off the mountain. You can't see the storm on the other side, but you do know there's a mountain. So Philip has done a fantastic job with that realism. We're going to ship that. There's new features for the FMC. And I think we're going to finally ship the new network synchronization layer, at least one part of it. So Pavel has reworked how we synchronize state over the network because you can do land flight, you can do X-Plane plus external visuals. And for years, we only synchronized the aircraft, which is the most important thing. But as X-Plane has kind of grown a zoo of other stuff, that starts to be inadequate. And we get bug reports of, hey, I've got a three monitor, three machine setup.
And the jetways aren't in the same positions across the monitors. They don't line up. And it's like, well, that jetway is a simulated object that can extend and go to the airplane and move. And the machines didn't know what the state was on the master machine. So we have new synchronization. So it's going to be exciting to ship that for the first time. This is one step in a roadmap for improving external visuals. And the team keeps working on features. So we have releases beyond those, which, you know.
I think Marco will slit my throat if I talk about, but we can see what's coming. The code isn't ready yet. And, you know, it's in progress, but we can say, this is gonna be a valuable feature for our user. We've, you know, put shovels to the ground, we're engineering it, and we'll see in what order they come out. Because it can be hard to predict how fast things will get decoded because in software engineering, sometimes you don't know what the hard problem was until you crash into it. You know, if you, if you already fully understood the problem, like it's probably because you already coded it, in which case, why are you coding it again? Like there's just not a lot of repeat problems when people build a house.
It's not that different than the last house they built. And so the estimates can be very accurate because you can use all that experience. And maybe if it's the first house with three printed concrete, that's gonna be really hard to predict. But if it's the six millionth two by four frame house, it's a well-solved problem. And in the simulator, we're pretty much never solving the problem we already solved because we already have the code to do that. So it's always new. So it's kind of always a learning experience, every new feature. Awesome. So we're getting close to the end of our time and this has been absolutely fascinating. Is there anything we haven't talked about yet that you think we should touch on?
I think we've covered a lot of the landscape. I guess the other area where we've seen a lot of change besides Moore's Law drives us to use new hardware, new technology to try to give more to our users. And I should say, I am very thankful to have that problem because I started in video. And when I was at Avid Technology in 1998, video was rapidly becoming a solved problem. And it was hard for the company to come up with new ideas that they could sell to their customers. And what happens is somebody comes into the market with a much cheaper product using the fact that technology is faster and they've got a lower cost engineering structure.
You know, you get eaten alive from below, right? The bottom will come up from below and that's where you got to look. And this was after my time there, but I think to a certain extent, their bacon got saved by HD. This needs to like operate all this equipment. It's like, okay, now there's something we can tell our customers that they really, really need. But other than that, once you solve the problem, you're kind of in trouble. So, you know, we used to pay for programs to unzip files. Like that was the thing you paid for. And now, like we take it for granted. It's built in, right? That's a market that's gone. And that's great for all of us. That's great for users. But as a programmer, you kind of have to look at what you're doing and say,
what's driving this? Like, will people want more? So we've been very lucky that our users want higher fidelity simulation and we can go use this hardware and do a better job. The other area where we've changed is kind of new programming techniques and language innovation. So besides moving to Vulkan to use graphics hardware better, you know, we've really changed how we approach concurrency. And that's also driven by hardware because when I started, dual core machines were exotic.
Like there wasn't really a point of coding for them because no one had them. Then we reached a point where, okay, people have dual core machines. Separating the loader from the simulator would be worth it. You know, we can load in the background and not pause your flight. Now, 16 cores, sure. 24 cores, 32 cores. People have massively concurrent machines. And fortunately, the language technology is growing too. So we have access to coroutines now in C++. Structured concurrency has become something that people understand. So you can program with language support for these multi-core machines in a way that isn't the more equivalent of go-to for concurrent programming. So as we develop new systems, we're also kind of coding them differently. We're seeing people...
use structured concurrency in the sim, and be able to write concurrent code from day one that's pretty safe, hopefully. I mean, multi-threaded code is always really hard, but this is at least not quite as dangerous. And it's been interesting to see how our techniques can change based on not just better hardware, but better tools and kind of better community learning. And that kind of helps us to address that challenge. Awesome. Let's call that a wrap.