Asahi Linux is a project that aims to port Linux to Apple Silicon chips, which use a custom ARM-based architecture. The project is fundamentally important given the popularity of Apple Silicon Macs, and it's also a heroic effort because Apple Silicon is an entirely undocumented platform. Alyssa Rosenzweig is a well-known computer scientist who describes herself as a graphics developer passionate about software freedom. She is currently a contractor at Valve, where she develops open-source software to improve Linux gaming. Alyssa is also a contributor to Asahi Linux and works on reverse engineering the Apple M1 GPU, among other contributions to the project. Alyssa joins the podcast to talk about reverse engineering hardware.
Asahi Linux, new advances in gaming on Asahi, and more. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him. Alyssa, welcome to the show. Hi, it's great to be here. Yeah, thanks so much for being here. I'm very excited to talk about this project. I talked to lots of people. I think this is probably one of the most exciting things that I've got to talk about in a while. But before we get into that, I thought it would be helpful for listeners to cover a little bit of the history of Apple and chip designs and set the stage, so to speak. So feel free to jump in if you have something to add or additional context. I'm sure you know way more about the space than I do. But essentially, the x86, which was created by Intel back in 1978, is the most popular and widely used chip architecture in the world. Intel AMD chips are based on this, and Apple's been using the x86 chips since 2006.
Because Apple was using x86 chips for their Mac computers, you know, those computers could, of course, run OS X, but they could also run things like Windows and Linux on them because the CPU instruction set was known. The ARM chip architecture, Acorn Risk Machine, was created in 1985. Apple started using ARM chips for iPhone back in 2008. Then in 2010, with the first generation iPad, they licensed the ARM architecture because it gave greater control over performance, energy efficiency. It was a huge benefit, of course, like smartphones back in the early days. And then in 2020, Apple transitioned their entire line of Mac computers from x86 to ARM, starting with the M1 chip. And this was like a from the ground out rebuild.
giving them greater control over the software hardware. But this also was completely proprietary technology where only OSX could run on the M series of chips. And Linus Torvalds even said when asked about having Linux run on these chips, said that it would probably be more work than it was worth it to do. But then along came a bunch of crazy engineers, including our guest today, that said, you know, not on my watch Apple, and started working on this super ambitious project called Asahi Linux, aimed at reverse engineering the M series in order to get Linux to run on Apple Silicon. So Alyssa, tell me about the project. How did it get started? How did you get involved? And do you have any additional context to share on the sort of the history that I shared?
Sure. Well, as far as I know, that history is right. I don't know the dates by heart. Yeah, Apple had announced that they were transitioning to the M1 chip and having ARM desktops, which suddenly made the ARM hardware, the Apple ARM hardware, very interesting to a subset of us in the Linux community. Obviously, the M1 chip is nothing new. It's just a scaled up version of what Apple's had in the iPhone for years prior.
With the locked bootloader on the iPhones, we can't run anything but iOS on it, at least not on any of the current devices. And even on the ones where we could because of boot ROM exploit, it's just not so interesting to us. I don't know. There are people who are very interested in putting Linux on mobile devices, and I think that's awesome. I think that's cool. It's not my jam. There are a lot more people that are running Linux on desktops if we ignore the Android space, which is, of course, where we can't ignore the Android space. Anyway, so when this chip came out for the M1, it suddenly put itself on our radar.
Mark Han, Hector Martin, announced that he was going to work on this new Asahi Linux project. And he launched Patreon and got a lot of excitement really fast. And it looked like this project was actually going to happen. And in his initial postings in the project, right after the chip had launched, this would have been back in December of 2020. We're all stuck at home with nothing to do but reverse engineer hardware. And Mark Han had said publicly that he was going to have to work on all these different upstream projects. The Linux kernel, yes, but also things like Mesa, where all of our GPU drivers live in user space. And he had just put out a call for any upstream maintainers that wanted to show them the ropes for their parts of the stack.
you'd be very appreciative of that. And, you know, I bet. I was, at the time, working on Panfrost at Collaboro. I was a student, but that was my part-time job, at least. So we started talking over IRC and giving Markhand the rundown of, this is what I expect a GPU driver development project would look like for this new Apple hardware. And long story short, I ended up buying myself an M1 Mini for that Christmas, and you know what happened next.
Yeah, so back in 2020, when most people were making sourdough, you decided, like, let's reverse engineer some Apple Silicon. So how many people were actually involved? Or like, what's the scope of the core team that's working on this project? Well, for the previous comment, there is a little bit of Panfrost code in the Asahi Linux driver and the Mesa driver. And there's plenty of other drivers in the Panfrost driver. So in a way, I am making sourdough. Sorry, what was the question after that? How many people are sort of involved in the core project? Yeah, it depends how you define involved. There are only a couple of us who are...
getting paid directly for working on Asahi. There is maybe a half dozen people who are very active in the project, if not full-time, part-time. And then when you include the broader community of people who are working on whatever their little piece of the Asahi Linux umbrella is, I'm sure you'll get dozens there. Okay. Why is a city like porting Linux to Mac under these conditions such a hard problem? There are two broad issues here. The first is technical in nature. There are a number of architectural deviations between the Apple hardware and...
what the rest of the industry is doing. And the reason for that is simple. If you are Intel or AMD, even Qualcomm, you need to make sure your hardware can run Windows, Linux, and Android, and not just run the OSes themselves, but also run essentially any user space on top of them. And even Qualcomm has Windows and ARM to keep them honest in some sense in terms of following the progression of the industry. With Apple, Apple only cares about running the Mac OS and running iOS and whatever other OS they have, watch OS, TV OS. They only care about the Apple ecosystem. And unlike Microsoft,
which has to care a lot about backwards compatibility for various business or political reasons. I'm not a Microsoft person. I couldn't tell you. I do know Microsoft cares a lot about backwards compatibility. Apple does not. I don't think that statement is controversial. Apple has several times dropped backwards compatibility for different software, different hardware. And Apple has the blessed Apple way of doing things. And they just don't care so much about.
what the rest of the industry is maybe doing or even what they were doing 15 years ago. And what that means is that they can iterate very quickly on the hardware and they can do unusual hardware designs, which from a technical perspective is very neat. But it also means that if we want to be able to just run some random Windows game that came out 10 years ago, it might not have the hardware support we need for that. So there's a lot of challenging technical problems of trying to port standard software, standard APIs, port standards onto hardware that was really never designed for it. The second part of the answer is that we're doing this all with our eyes closed. Apple does not document their hardware in any meaningful way. I'm sure they have documentation internally for other hardware. It's not like nobody at Apple is a technical writer, but certainly there's nothing they've released publicly. We don't have any hardware documentation under NDA either. We have no reference code from Apple. All of Apple's own drivers are proprietary software with vanishingly few exceptions. Occasionally there will be something in a kernel source drop that's useful, but by and large, all the interesting drivers are proprietary. So we have no reference code on how to use the hardware. We have no documentation on how to use the hardware.
We have to figure everything out just by either guesswork or observing what Apple's driver does from the outside. And we can't get too close either because black box techniques are generally preferred for legal reasons. Okay, yeah. So you have the just like base level problem of, hey, we have to like map a bunch of stuff to hardware. that wasn't designed essentially for it to work with this piece of software. And then on top of that, you don't even have insights essentially into how the hardware functions. So the mapping process you're mapping essentially into this black box. In terms of the backwards compatibility, I don't think anybody would argue with you that Apple doesn't really care about backwards compatibility. I mean, in some ways, because they're not burdened by backwards compatibility, they probably have some advantages in terms of this M-Series rebuild. They can really optimize it to run. They may have to worry about standards, backwards compatibility. So that gives them an advantage probably with optimizing energy efficiency, compute cycles, whatever it is essentially that they particularly care about. For sure. In terms of like...
With something like this that's very challenging technically and there's scale to it, where do you even start a project like this? I think Mark can put it best. The hardware doesn't have any public documentation. So the first thing to do is document the hardware. The problem of writing drivers is challenging for sure, but there are lots of people who have written drivers for hardware that has documentation. So really the start of the challenge is reverse engineering the hardware, not worrying about how we're going to drive it, but just worrying about what the hardware itself does. And I do mean hardware. I don't mean reverse engineering Apple's drivers, Apple's softwares, Apple's APIs. It is...
irrelevant what Apple does on their software team. All that matters is what the hardware is capable of and how the hardware itself works. So how do we do that? Well, the good news is that we do have macOS running on the machines, and Apple does have drivers for their own API, which doesn't support everything within the standards, but it does support almost everything the hardware can do, almost. So what we do at a high level is we have little test applications that will run on macOS, and we will intercept the communication between Apple's drivers and the hardware itself.
By intercepting it, we can dump out all of the interesting information, all the data structures in memory, the registers, what have you. And when you do this for the very first time on unknown hardware, well, you just have a big pile of hex values that you don't know what to do anything with. Great. You save it. There's your first dump. Go back to your test application, change one tiny little thing, get a new dump, and compare the two. If you're lucky, there will be a single tiny little change in the output, and you've just reverse engineered your first bit of the hardware.
Obviously, something like a GPU is massively complicated, which means you will have to do thousands of iterations of this before you really understand the hardware. But there aren't any leaps of faith for the most part. It's just one building block after another. And when you do this thousands of times, you get a pretty good picture of how the hardware works. You document it as you go along. I don't write formal hardware documentation because I have better things to do than become a technical writer. But we do have XML descriptions of the hardware data structures, for example. And we do try to keep those as accurate and complete as possible.
And so then once we have this understanding of what the hardware does, we can understand all these little data structures in it. We can write drivers for that. We can start out writing drivers that mimic what Apple's drivers do, you know, monkey see monkey do, because we know that is a correct way to drive the hardware. But once we understand how the hardware actually works, we don't have to stop there. And at this point, we've been reverse engineering the M1 GPU for years now. While there are certainly bits we don't understand yet, there's no giant piece of fixed function hardware that is just a big mystery blob. We understand roughly how the hardware works and roughly what's there, which means...
It's no longer relevant now what Apple's drivers do or how Apple approaches problems. The reverse engineering is sort of done. And that segment of the project is going a little quiet. And now it's an engineering problem of we are driver developers. We have a hardware that can do these things with this constraint. We know how to drive it. And we need to make it do this other thing. How do we do it? What's the best way to do it? And sometimes that means doing something very similar to what Apple did. Sometimes that means doing things very different to what they did. Certainly running DirectX 11 efficiently on this hardware requires quite a bit of creativity.
The sort of starting process of like intercepting the messages are going from the driver to the hardware. How do you actually intercept those messages? The macOS driver in user space, whenever it wants to do anything with the hardware, it calls into kernel space with a system call. And the way it does that is through Apple's IOKit framework. So IOKit is a open source Apple framework, which makes things marginally simpler because these are public entry points, IO connect, call method, whatever. And then...
All the different methods are undocumented for the GPU, but the general signature of this ioconnect call method is public. So what we have to do is write our own version of ioconnect call method that calls into the real OS one, but also right before calling in, we'll dump everything to standard output to a file, wherever. And we then use a little bit of a dynamic link or trickery to run our test application, but with our version of the ioconnect call wrapping the real one. And then we are right in the middle. The other way we can do this is actually at a system level for doing kernel work that...
won't work because you can't intercept the kernel, user space talking to the kernel, the kernel is the thing you're working on. But you can't intercept the kernel talking to the hardware. So I believe Marcan wrote a fascinating hypervisor where we have the capability of booting macOS in this thin hypervisor we control. And so in essence, the entire system is being wrapped. And then any register writes, any memory access, anything is going through our custom hypervisor. And we can instrument that to dump the things that are interesting to us. I believe Asahi Lina has written a tracer that's able to dump the GPU stuff at a system level as well.
If Apple changes things and we can no longer do the IO connect trick, we're not out of options, although that would certainly be inconvenient. Okay. And in terms of the GPU architecture, how does that compare to other platforms that you've worked on? In many respects, the Apple GPU as shipping in the M1 is one of the simplest GPU architectures I've seen. This is both a blessing and a curse. It's a blessing because it's very straightforward to drive this hardware. Drawing a triangle on this hardware is, in many senses, easier than on other parts. There are just fewer...
things going on in the hardware. And so it meant that we could get up to speed pretty fast. And when I first started working on this GPU, I thought this hardware architecture was amazing because I was used to dealing with hardware that had millions of stupid sharp edges for literally no reason. And it was refreshing to have an architecture that didn't include so much nonsense. Do you think that's in part because they did this sort of from the ground up rebuild and they don't have to worry about any of the backwards compatibility? Well, the hardware's not from the ground up. Well, yeah, it's very clearly a power PR derivative, but I'm not sure it's that. I do think that Apple...
has brilliant hardware engineers. I have a lot of respect for the engineers on Apple's hardware team, for sure. They have a set of engineering problems posed by iOS originally, and the solution that they built is very well suited for its domain. But I said it was a blessing and a curse that it's so simple. And the curse is, of course, once you get over that initial phase, you're just kind of stuck. You run out of hardware to do things, but you haven't run out of features that the drivers expected to have. So there is piles and piles and piles of functionality, which would be in hardware on most other vendors, but which is just not in hardware on Apple. And so we have to implement that hardware ourselves in the driver.
you know, generating shader code or something. And that can be nice. It's very flexible. It means that I can implement things in my driver that Apple doesn't implement in theirs because, well, it's just software. It's also very annoying because especially something like Vulkan strongly assumes that you have hardware that, well, implements Vulkan.
I see. And then, you know, what were, you know, when you were starting this project, besides some of the things that, you know, obviously we talked about of just like, hey, but you have to kind of do this step-by-step dumping of the communication to the hardware and get a sense for like how it actually works and document that. Like, was there any big surprises in terms of that you discovered about how the hardware functioned? In some ways, the simplicity was a surprise. It's unheard of to have a architecture that is this simple, really. If I rattle off other GPU architectures, pretty much you can tell me, Avenger, and I'll tell you why their hardware is terrible on some axis, right? And...
It turns out the axis for Apple is it's simple to a fault. It's very well suited for the design space that it was built for. And it turns out that running AAA games is not what it was built for. And I think Apple has realized that as well, which is why they've made some of the changes in the M3 that they have. What features of Asset Analytics are still under development?
So the big ones, one is support for Type-C display output, as well as Thunderbolt. We support HDMI on machines that have a physical HDMI port, so at least on the MacBook Pros. We don't support Type-C to HDMI adapters yet. We have patches for that. It's in progress, hopefully by the end of the year, but there's a lot on our plate. That's the one that people complain about anyway, so I'm guessing that's the biggie for people. Personally, I only have one monitor, so I've never hit it.
What about like, you know, there's a lot of custom components in Apple hardware, like the Apple neural engine, the secure enclave. Like, are those things that you're also looking at in terms of the entire project? Certainly all those components we can use if we want to, we don't have to. The neural engine was reverse engineered by Eileen Yoon. So on her GitHub, there are notes for the hardware and sample code to drive it from Linux, which is super cool. It's not really clear where we go from here because there's not an established standard-based ecosystem for machine learning acceleration on Linux. I expect it will happen given the current AI craze, but that's not the case now. This is very different from graphics where we have a standard for graphics. It's called Vulkan and it's called OpenGL.
And it's called OpenCL for compute. And if you want to get graphics on Linux, you implement those three APIs, end of story. With something like the neural engine, there's not a standard for that. It's not clear what we should do with it yet. And as talking Linux, as I said, there are a couple of us who are getting paid for this and doing it as our job, but most people are not. For most people, it's a hobby project. And something like the neural engine, that's very low priority for the project as a whole, which means...
the very scarce, you know, full-time development resources are going to be going towards things like Type-C display or the GPU. As for if somebody has an interest in working on machine learning stuff, then yeah, they can pick up and work on their own engine. And that's, I guess that's what happened with Eileen and she went and reversed engineered and she did it. So, but we do know how that hardware works, but will that ship? I don't know. We'll see. Maybe if somebody wants to.
In terms of the full-time people, how do you sort of organize? How do you decide which parts of the project need attention and who works on what? The project is a bit of controlled anarchy. It's definitely not a corporate structure. By and large, the split is I work on GPU user space. Lena works on the GPU kernel space. Markhan works on everything else. That's the split. What about in terms of community contributions? Has the open source community contributed meaningfully to the project yet? Absolutely. As I said, there are...
at least six people who are very active on the project. And if we include everybody who has done significant contributions, if not in the present than in the past, certainly well over a dozen people have contributed in significant ways to make this project possible. I keep mentioning the type C work that was originally done by Sven Peter and is now being worked on by Jan Grinnell. Neither of whom are me, neither of whom are Marcan, right? Neither of whom are Lena. As somebody, like if I was interested in this and I wanted to like install it and use it, what's that process like?
There is a script you run on macOS. So for better or worse, it is a curl pipe sh on macOS that asks a couple of questions on macOS to work with the partitioning mainly. Eventually, it will give you some instructions of what to do next, which will be to reboot the machine and hold down the power button to prove to the hardware or prove to the firmware that, yes, I'm the machine owner. I'm really trying to install this. This is not a rootkit, right? Because, of course, the process to install an OS and the process to install a rootkit, from the firmware's perspective, there's no difference except for one of them the user did intentionally. So Apple's firmware makes sure that, you know...
you're doing this on purpose, then it reboots, and then you're just taken into the standard KDD or GNOME installer to finish up the process, and that's that. I don't know if it's easier than installing on an x86 PC. I haven't done that in a lot of years. It's certainly easier than installing on any other ARM system I've used. As a user, like today, like how much functionality, would there be certain things I would notice that in terms of like limited functionality, or would I be able to actually use this day-to-day? It depends on what you're doing with the machine. If you ask 10 people if Asahi Linux is ready, you'll get 10 different answers because, you know.
If you're a developer and you are building Rust packages all day and that's your whole life, it'll be great. If you are a gamer and you want to play Cyberpunk 2077, maybe it's not ready quite yet. And you should strongly consider selling your Mac and buying a Steam Deck instead. If you are, you know, it just depends on what you're doing. I would say all of the standard basic functionality is there. Some people would say that the Type-C out is standard functionality. I only own one monitor, so it's not standard for me, at least. There's a lot of stuff that works that might be surprising that it works. You know, there's a package you can install to get Widevine for watching DRM videos.
for better or worse. For me, I've been using Asahi as my main machine for years, and I have nothing to complain about. For you, I don't know. Depends what you do. Okay, fair enough. Look, how did you personally, even before Asahi, how did you kind of build up these skills or interests in terms of being able to do these types of projects? So before I worked on Zahi, I was working on the Panthros project for Mali GPUs, which was a different landscape for sure, but the core methodology was the same. ARM had their proprietary hardware, ARM had their proprietary GPU driver, and we did the usual wrap the driver, reverse engineer the hardware, write our own driver, which went from very humble beginnings all the way back in 2017, started working on that.
So I was in high school at the time and over time built that up. Today, there is a well-staffed full-time team of engineers, not just at Collabora where I was later working, but also I'm now seeing contributions that drive from ARM employees with their at arm.com email addresses. So it's pretty cool to see the very far from the original proprietary stack. So no, I did not start with the Apple. Prior to that project, how did you basically get your entry point into like programming and coding? Like what sparked that and how old were you? I started coding, I was maybe seven. I'm really showing my generation here.
was not doing impressive stuff when I was seven. That should be clear. I think I was writing websites with PHP. That was cool back then. And over time, developed interests in compilers and graphics and reverse engineering. Prior to doing any graphic related reverse engineering, I had a project in high school where I had a zillion different websites that I had to use for school because I don't understand why there's this many websites for school. And I wanted a unified front end for all of them. So I went ahead, reverse engineered a bunch of the...
HTTP, you know, REST APIs that these used, wrote some Python to plumb it all into one unified place. And cool, that made my school a little bit easier so I can focus on more content for us, I guess. And certainly there was a progression there. There are a lot of people that I think are impressed by what I do at my age, but I've been doing this since I was seven, which means, you know, if I have 15 years of experience, maybe that makes more sense now. Yeah. When it comes to these kind of projects...
Especially when you were talking about sort of just like trying to figure out how the hardware works. Like there's a lot of like sort of repetition where you're changing a little thing, getting the hex dump, figuring out what changed, documenting and stuff. How do you kind of stay motivated to continue to work on something like that? Everybody has their special interest. It's ironic I've been working on gaming because I just don't find games all that interesting. I will play a game for about a half an hour. I'll have fun and then I'm just bored of it. And I know there are people that will play games for 10 hours and will only stop because it's bedtime if that stops them. More power to you. I hope you can use my drivers to help in some way, but.
That's not my thing. And I'm sure they would get bored trying to do what I do with the reverse engineering. I just get a thrill out of it. I find it fascinating. I'm stubborn.
It might just be that I'm stubborn. You have to be stubborn to succeed in this industry. And then in terms of people who might want to get involved contributing to the open source project for community aspect, what's your recommendation in terms of getting involved and jumping into something like this? This is definitely not a good first project for things. There are lots of people that will show up and want to help and have no relevant experience whatsoever. And it's not clear what to say because it's...
Even among reverse engineering projects, even among drive development, Asahi Linux is in many ways doing things on hard mode, just given all the constraints we've talked about earlier today. So the people who have contributed successfully, I think have had significant program experience on more approachable projects first. So yeah, I would say try to build the skills. And if you think you have what it takes, you won't listen to me tell you not to, because like I said, you don't succeed in this industry unless you're stubborn. And if you're stubborn, you'll ignore me saying not to, which is of course how I got started working on graphics was I ignored somebody else telling me not to.
So I think that's the tradition now. I mean, you have to have certain determination to do something. You know, going back to what you said, like, you know, maybe this is not the place to start. It's like, oh, I've never like, you know, done anything like this before. But if somebody was interested, you know, this piqued their interest. Hey, I want to do some sort of reverse engineering project. I want to learn more about hardware and how some of the lower level parts of a system work. Like, where does somebody like that start? What do you recommend? Find something to reverse engineer and go at it. Even if it's already been reverse engineered by somebody else, just don't cheat. Do it yourself. Hardware reverse engineering is...
Definitely challenging because of all the extra constraints, but the same principles, the same skills transfer directly to and from other kinds of black box reverse engineering. I think a lot of people who are successful in the hardware reverse engineering space also have experience doing something like reverse engineering web services, like I mentioned, or in the game modding community, same skills. I can't speak too much to those communities because hardware is what I've known for years, but probably similar to jumping into any kind of programming type of project, like just pick a project to start with that's like of reasonable size and scope that you're interested in doing. And then you'll probably develop whatever skills you need along the way if you're stubborn enough to be successful at that project. And then essentially rinse and repeat until you build up enough skills to tackle bigger projects. For sure. Is there anything else you'd like to share about the project or your personal journey with this? I'm very excited about what we've just released with the alpha for gaming on Saki Linux. With the stack we've just dropped, this is tying together a lot of work that's been in flight for years, not just on the graphics driver side, but also project-wide and across projects. A lot of open source stack together to do something that should not be done, really. We shouldn't have done it. So this ties together.
the Vulkan drivers that we've written for this hardware on top of DXVK for translating Vulkan to DirectX calls as part of Proton, which includes Wine, for translating Windows system calls to Linux. All of this is x86 code being run under FEX, which is a x86 to ARM emulator, which is running inside of the virtual machine to handle page size mismatches, which is running on our bare metal. And miraculously, this giant pile of things means that Direct3D Windows AAA games are entirely playable and a lot of fun. And it's pretty crazy to see how far we've come with this. When I started working on graphics for Asahi, people would constantly be asking, when can we play insert AAA game title here now? And I'm like, I don't know, I just drew a triangle, man. And I'm my own worst critic, I think. And I never thought we would get to this point. So the fact that, you know, Fallout 4 is running on my machine with drivers I wrote, I'm pretty pleased with that. And, you know, I've been working on facts as well as just working on performance work for the past year or so. So for me personally, it's great to see tying together all these different projects and tying together, you know, my maximum opus. Yeah, it's amazing.
Thanks for sharing that. And thanks for sharing so much detail about how this project came together, how you get involved in some of the challenges along the way and also the wins along the way. So thanks for being here and cheers. Cheers.