David A. Patterson is a pioneering computer scientist known for his contributions to computer architecture, particularly as a co-developer of Reduced Instruction Set Computing, or RISC, which revolutionized processor design. He has co-authored multiple books, including the highly influential Computer Architecture, a Quantitative Approach. David is a UC Berkeley party professor emeritus, a Google Distinguished Engineer since 2016, the Rios Laboratory Director, and the RISC-V International Vice Chair. He received the 2017 Turing Award together with John L. Hennessey, quote, for pioneering a systematic quantitative approach to the design and evaluation of computer architectures with enduring impact on the microprocessor industry. In this episode, he joins Kevin Ball to talk about his life and career. Kevin Ball, or KBall, is the Vice President of Engineering at Mento and an independent coach for engineers and engineering leaders. He co-founded and served as CTO for two companies, founded the San Diego JavaScript Meetup, and organizes the AI in Action Discussion Group through Latent Space. Check out the show notes to follow KBall on Twitter or LinkedIn, or visit his website, kball.llc.
It is my absolute honor today to welcome Turing Award winner David Patterson to the show. Thanks for having me. I'm excited to have you and I have a bunch of questions. I just I love getting to geek out with people on stuff. But I'm curious before we start, you have a long illustrious background. You've been in a whole bunch of different domains of the tech industry. How do you introduce yourself these days? What do you bring forward? I think I usually start with I was a Berkeley professor of computer science for four decades. And eight and a half years ago, I started working for Google. So I've almost got a decade at Google. So I've got a half century of experience. I love the encapsulation as a half century and the field isn't that much older than that.
Yeah, the field's about as old as I am. So it's in the late 40s. It's the very beginning, the ENIAC and things like that. And the first, when I decided to study computer science, I had no reaction from my relatives. It was not like in the 1960s, everybody said computing was the future. It was just this kind of side thing that few people were interested in. And now my relatives think I was very wise in the field. Yeah.
Absolutely. So the subject that you won the Turing Award on was related to risk and MIPS and that sort of domain. And I know I saw that you've been deeply involved with the RISC-V project, or at least you wrote the book or one of the books on it. So I'd love to kind of dig in a little bit and get your perspective on RISC-V, what it means, and what people are trying to do with it. All right. So when I explain it to my relatives, what I say is when software talks to hardware, there's a vocabulary. And the technical name for that vocabulary is instruction set. And the words of that vocabulary are like the keys of the calculator, add, subtract, and multiply, divide, and all that stuff.
And so the so-called reduced instruction set computer debate, which we had in the 1980s, was what's the best instruction set or vocabulary for microprocessors? And the prevailing philosophy was that it should be very sophisticated in this idea closer to software. And John Hennessy and I, who shared the Turing Award and people at IBM, argued that that was the wrong model, that we should instead keep the instructions actually relatively simple. And that's kind of reduced, simplified instruction set. And that the compiler would be able to map programs into it. And if we kept the instructions that simple, we could iterate them faster.
it could potentially be more power efficient and things like that. So that was the debate. And what it came down to is for the more sophisticated instruction sets, you can think of polysyllabic words and vocabulary, you needed fewer of them to execute the program, but they might run more slowly. So what was that ratio? And it turned out that risk style tended to execute about 30% more instructions, but we could execute them about four or five times faster. So that was the net win. So that's risk in a nutshell. Yeah, absolutely. And it has continued to evolve. I would say probably if we look at, you know, processorship today, the debate is over.
essentially. Risk is more or less one. Yeah. And new instruction sets over the decades, but nobody's tried to do it. The ones, these very sophisticated ones, they're still similar at the core, similar to the very risk one that we did at Berkeley in the 1980s. Yeah. So looking now, risk five, relatively recently, the first version finalized and out in the world. What is different? I think I saw in a brief that it learned a set of key lessons from previous generations and avoided some key mistakes. So what do those look like? Well, we did four risk architectures at Berkeley in the 1980s, and Hennessy did a couple called MIPS at Stanford.
And what happened was, you know, the origin story for RISC-V is in 2010, we were going to do research around parallel computing that was sponsored by Intel and Microsoft, the switch over from single core to multi-core. And that was what was funded. And we were going to need an instruction set to do our research. We could see that Moore's law was slowing down and thought what the future looked like is you'd have a core instruction set, and then you'd add special purpose instructions for specific domains. That's what we thought was going to happen. So we needed a core. So it would make sense to use x86 kind of based on who our sponsor was. But A, it was, you know, a ghastly, hard to extend. And Intel wouldn't let us use it, nor could we use ARM. You know, ARM.
was popular, but you couldn't extend it. So we had to invent our own. And so led by my colleague, Christopher Sosanovic and two grad students, and simply in Andre Waterman, they said, let's do a brand new risk architecture, learning what we've learned over the last 30 years about things that were probably mistakes back then and what we learned since then. And because we did four risk architects in the 1980s, they called it RISC-V. So I think the fact that they call it RISC-V gets me more credit than I deserve. But the idea was, you know, as researchers, we thought everybody would want to use it. Everybody in academia would want to use it. Other researchers wouldn't use it. So we made it available, kind of the Berkeley tradition, things are open source. And that's how it got started.
About four years later, we were using it in our classes and our research. It didn't get really other universities. It didn't really pick it up, but it was out there. But we started getting comments like, why did you change the instruction set from the fall semester to the spring? Why did you make some changes? Like, why do you care what we were doing in our classes and our research? And then in talking to them, we found out there was this thirst for an open architecture that people could use rather than have to get a proprietary architecture. And once we realized that, then we thought that was a great idea and wrote a paper kind of inspired by the Linux.
Software should be free. This is what we call the instruction set should be free. And then soon we started a foundation around 2014, 2015. And now 10 years later, fortunately, it's actually caught on. It's people like the enthusiasm around open source software. This has this kind of religious fervor, philosophical attraction. Same thing for the open architecture. Most of the people involved really like the idea that potentially we could have a lingua franca across all computing, the biggest to the smallest. And if we could have one, it better be open. As we see, instruction sets of proprietary are tied to the fortunes of those companies. And who would have thought?
Intel would ever be vulnerable, right? That just seemed impossible. But yeah, that x86 is tied to that company who's having difficulties. And there have been many other instruction sets that have gone away because of the fortune there. So the idea of an open architecture that's kind of community-oriented, it's a standard. It's not like Linux, which is an implementation. It's a standard like USB or something like that, but a lot of enthusiasm around it. So if you look at the very core of that instruction set, it's similar to RISC-1, but since we have so many transistors today and we had that idea of adding features when you need them. So we have RISC-5 allows optional features for all kinds of applications like encryption.
or machine learning or things like that, options that you can add. But at its core, it's that same risk philosophy. Let's maybe talk some about that extensibility, because I think one of the things going on right now with Moore's Law ending with a bunch of these other things is we're sort of moving into a world of domain-specific accelerators. And probably the biggest one of those is around machine learning, people using GPUs or TPUs and things like that. But one, what does that look like for people using RISC-V if they want to tap into a domain-specific accelerator? You said it's open, they can do that. What does that actually end up playing out like? And where are you seeing this happen?
Yeah. So what I ended up doing at Google for almost a decade is working on domain-specific accelerators for machine learning. When I retired from Berkeley and wanted to keep my hands in the technology, I thought Google would be an interesting place. I had gone to sabbatical recently. And they just kind of, because of my experience, they had me report to Jeff Dean, who's kind of a famous software engineer. And I think just because of my stature, but he was in the machine learning part of Google. And not that I thought this is something I wanted to do, but I didn't have any strong opinions.
But he was in Google Brain that he founded. Jeff was a big and early believer in the potential of machine learning AI. He was one of the first movers in that. And so Google was the first mover in that. So I ended up spending the last eight years learning all about domain-specific accelerators for ML AI. So now RISC-V in particular, its first foothold has been in embedded computing. So there's DSP extension, there's digital signal processing extensions. There's compression expenses to keep the instruction set smaller.
extensions for machine learning, kind of to my surprise, I was involved in the IEEE floating point format standard, which in the 1980s, before that, different computers had different floating point formats. And so imagine porting floating point programs when the floating point didn't do the same thing. So it was a real kind of mess on big computers. And the floating point standard was set up around, wow, microprocessors are starting to have floating point. Let's standardize it. And so thank God we did. But I thought the data types were settled, which was single precision doubles precision. Maybe down the line, there'd be bigger things, but that was it. Well, to the surprise of many, machine learning, because it's doing symbolic processing.
It's not whatever it is, 50 bits of precision. The range is very important, but the precision isn't all that important. So Google created a new floating point format, which I never expected to happen in my lifetime, given the standard. But at first, a 16-bit format that they called Brain Float, which is a different format because that's where it was done. And since then, Google and NVIDIA and other companies have made it even smaller. So 8-bit floating point, 4-bit floating point. And there's only 16 values. How can that be floating point? But amazingly enough, people are figuring out ways to get AI done with very, extremely neural data types. So data types has been a...
significant area of innovation in architecture. And I think when Bill Daly of NVIDIA talks about the giant games that we made in machine learning, he credits data types as one of the significant ones. So if you had an instruction set that you couldn't change from, you'd be in big trouble. So you'd need to expand these data types. So those are examples of the type of extensions going on.
That's a great example. There's all these interesting papers around, yeah, trading off between the number of parameters versus the precision of the parameters. I saw somebody doing essentially, what do they call it, like one and a half bit, where they just basically had one zero or not set, and each parameter was just that. And that makes a ton of sense that you're being able to bake that in at the instruction set level, because this is extensible, allows you to experiment in those domains. Yeah, so if you're a computer designer, what you love is people who say, I can't get my work done. I need a much faster computer. You're not happy with people. Yeah, things are fast enough. My laptop is good enough. I'm going to keep it until it breaks. I'm going to keep it a decade, right?
You're not a fan of those people. The machine learning people are voracious. If you give them a factor of 10 tomorrow, they'd say thank you. And then like, what's next, right? They can use up everything. And it's a brand new area. We don't know the best architecture for.
machine learning and AI. It's a wide open space. And so this idea that doing this numerical analysis, it's got these two phases, training and what's called serving your inference. Training is kind of like sending your kid to college. It takes a long time. It's expensive to spend a lot of money. But once they're educated, then they go out into the world and then hopefully answer questions much more quickly than it took them how to learn that material. So there are these two kinds. And numerical precision can be different. What you train in and then what they call quantification. Maybe if you train it using 8-bit, you'll be able to serve it or do inference at narrow bits. And there are papers that talk about...
one ternary, three states, like one and a half bit serving. It's amazing. So what's exciting, you know, from maybe in the 1990s when Intel x86 dominated everything, it was kind of boring. There's this wide open area on both the software side and the hardware side where it's not clear what's going on. And it obviously has gigantic commercial impact as those of us who watch Nvidia stock can see. So it's both, it's pushing the state of the art in both, you know, software, hardware, and simultaneously being delivered to people and having an impact on people's lives. So it's a very interesting time.
I'd love to dig in a little bit what you're talking about there in terms of separation of inference and training, because I think the vast majority of the industry is using off-the-shelf GPUs. They're using the same hardware for those. I think I saw a paper talking about Google's TPUs that y'all custom designed actually having some focused on inference versus training. What are the different parameters that you're optimizing there? So what's happened in the machine learning community? The big thing, kind of a breakthrough was in 2017 when Google came up this new model that's called Transformer. And a specific idea that if you think of an image, you can pay attention to.
different pieces of the image and do more computation there rather than uniformly. So that the actual title of the paper that introduced transformer is attention is all you need. So this has proven to be a breakthrough model. And what's happened in the last seven or eight years is people just pushing that model and expanding the number of parameters dramatically and all that stuff. So what's happening simultaneously besides the split of training and serving is this rapid increase in the size of these models. So what that turns out to is typically is memory capacity. So it used to be, well, I think there was a transformer model might've had a hundred million parameters, some number like that.
And people are already going to billions, and people are talking about hundreds of billions of parameters. And so when we were talking about the data type width, these parameters are probably either, well, they could have been 16-bit, this brain flow 16, or 8-bit, or people would like to get them into 4-bits to shrink both the memory capacity. And if it's not only smaller, it takes up memory space, but also you get more memory bandwidth. You can fetch more of them per second. So that's that shift that's going on, driven by this increase in these so-called large language models. Now, for surveying, it tends to be, for training, it's really computationally intensive. So if you look at our textbooks,
There's a phrase that's called arithmetic intensity, which is the number of operations for every byte fetched. So training, it could be hundreds of operations per byte fetched. So every day, hundreds of flame-borne operations per byte fetched. For serving, it tends to be not that high arithmetic intensity. So what that means is it's more memory bound. So serving tends to be more memory oriented. Training tends to be more compute bound. And so given that there are, well, first of all, you can do serving on a training chip. There's nothing, if you can do training, you've got everything you need to do serving. But by specializing, you can reduce the costs and you can produce the energy and the carbon footprint by doing that. So you can have a smaller chip or reduce the size. All these chips have big matrix multiply units. So you can get away with a smaller matrix multiply unit. You can get it right with a lower power chip and try and focus on memory centric architecture design.
for serving. I would say right now, you know, NVIDIA rules the world on training. And I think their favorite solution, how do you do serving? Well, use our old training chips to do serving. That's their advice. That may benefit them economically, but that's their advice. It might be just talking their book a little bit there. Yeah. So I think there's, whereas, you know, if you think of it historically, if you think about the PC as duopoly between Intel and Microsoft, right now it's pretty much NVIDIA and NVIDIA, but they focus on the training side and very high powered, very big chips. So the serving side is there's more opportunity, I think, for hardware people to innovate.
And with these kind of trade-offs I talk about, maybe you get away with a smaller chip or you have closer to the memory system, so you get good memory bandwidth. So that's kind of examples there. And again, this exciting era, certainly NVIDIA is the commercially available dominant thing. Some companies like Google have built their own training chips, and Google has also built special versions of it for serving. You mentioned something there that I'd like to dive in a little bit further. So a piece of it being around becoming memory bound. And I know this is a domain where the sort of ratios of compute to memory have been shifting over time, even on...
my laptop, if I'm running into trouble, it's almost always because I'm running into memory constraints. And I saw a paper that you were involved with recently around memory-centric computing and redesigning the ways that we set up at least cloud servers to be focused around memory-centricity rather than processor-centricity. Can we maybe talk about what that looks like and what that means? Yeah, I think the original tagline of that paper was, the CPU is not central anymore. So we're kind of used to measuring software by arithmetic operations, be very computation-centric. But increasingly,
you know, we are bound by memory, either by memory capacity or by memory bandwidth. And what's happened over time with the slowing of Moore's law is the rapid improvements in DRAM memory, which we saw for decades in the last century. It was like clockwork, four times the capacity every three years. Today, it's going to be more than a decade, four times every three years to more than a decade between the eight gigabit DRAM and the 32 gigabit DRAM. So it's really slowing down. And then along with that slowdown capacity, the bandwidth is improving this rapidly. So one of the famous...
kind of admonitions in computer architecture was by Gene Omdahl. And he wrote a kind of a one and a half page paper that stated what he thought was fairly obvious and has since become called Omdahl's Law. If you have a piece of the pie and there's a part of the pie that you accelerate and the rest of it you don't, he said, by what people say Omdahl's Law, that limits how much faster you go. If you're going to make two thirds of the pie go infinitely faster. you're only going to go three times faster because it's one third you don't touch. And it's kind of this, people call it this sad law because architects run into it all the time. They get very excited. I can make matrix multiply, go 10 times faster. Very exciting. And then all the rest of it, that doesn't need to go. So what's happening is, as Moore's law is starting to slow down, the logic is still getting pretty good. The actual arithmetic units are okay, but the cache memory technology that's called static RAM, that's not improving very much. And then the DRAM, which is in a separate kind of technology, is also improving much more slowly. So what they've done...
is to try and boost up the memory, especially for these accelerators. They've gone to a novel packaging scheme and it's aptly named high bandwidth memory, but it's actually a little physical memory where you stack the dies on top of each other. There's a stack of dies, like four or eight, and they're trying to get to 12 or 16 dies. And you have multiple stacks right around the computation unit. So they're very close and they've got a thousand wires. So standard DRAMs would have 64 wires. This is a thousand wires wired in these little stacks. It's very specialized memory. And it's kind of at the heart of all these accelerators to try and...
provide the bandwidth that you need, particularly for training and for inference. So it's, again, and not only is this technology running the areas, it's also in the business pages as we go along. SK Hynix has successfully can make it with eight chips and is delivering it. Samsung, which was the original, I think, creator of the high bandwidth memory, they're having difficulties delivering on this technology. So as a result, SK Hynix is doing well in the market. So this is all what's the AI stuff is.
tied directly to the business pages in stock prices that we see. Absolutely. Yeah, it's really driving it. And yeah, Amdahl's law is, I think, intimately familiar to any software engineer who's been told premature optimization is the heart of all evil, right? Like, oh, I got this loop going really fast. Why is my code not going any faster? Yes. You know, I think it's the law of diminishing returns. Like when Gene Amdahl wrote it, it was just like, oh, he was talking about parallel computing, right? And he says, there's a part that you can make parallel. Great. The part that you don't make parallel will limit how much performance you're going to be able to deliver. And, you know, it was obvious to him as a really smart computer designer that...
He just had to write it out because people are getting very excited about these parallel processors and ignoring the part that they weren't doing. Yeah. But it's this law you keep running into over your career. Like, oops, yep, screwed up again on those law. Yep. OK, so with the accelerators, that's really interesting, right? They're packaging huge amounts of memory going close to it. I think I saw another. Yeah, what's interesting is because you can actually plug a lot more DRAMs into your PC than you can plug next accelerator in the so-called dual inline modules. But you can put it.
dozens of them. But because of the physical distance to get that thousand wires, it's on a special package. So what's the downside of the stacks is the capacity isn't very high because it's not that many dyes, there's not that many stacks relative to the DIMMs. So incredibly fast, but very limited capacity. So how does that constrained capacity of HBM go with what you claim was billions of parameters or hundreds of billions of parameters? Yeah, that's the problem. You need a lot of GPUs to get enough capacity to be able to solve these big problems because of the limited memory capacity of the HBM stack. So, well, it's challenging.
I guess if you're selling GPUs, it's not challenging. Just buy more. There you go. Not a problem. Continuing to dig into this sort of memory and data centricity, I think I saw something around building out kind of database architectures around a memory pool rather than once again, being sort of CPU centric. What does that end up looking like? I'm interested in this shift of if our constraints are now memory, which they've been on and off again, but increasingly like data has so much inertia, like that is the fundamental constraint that's not getting faster from Amdala's law. Like how do we shift our hardware architectures and our software to better manage that? Yeah, so the paper that you're talking about is actually being presented at CIDR, which is a database conference.
I can't remember what it stands for. And I was at a workshop in Germany where they had a few architects and a lot of database people. And we got together and talked about this memory-centric approach. And the authors of the paper, you know, there's a lot of database people there who are better informed about this than I am. But one of the enabling technologies is this new kind of a successor to the PCIe bus. It's called CXL. And the idea is to be able to create a coherent address space across several servers. And what we argue in that paper is one of the downsides of database accelerators in the past.
is that you needed memory bandwidth for these accelerators. But you would have to put a lot of physical DRAM with each of these accelerators. And if you didn't use it well, it would be very expensive. So what we argue is that CXL allows you to very easily have, and CXL is standard on all the servers. have pools of DRAM that could be shared across CPUs rather than having to put a lot of DRAM at every physical one. So you could have given a pool of DRAMs and maybe accelerators, database accelerators make more sense is that you only need to use memory from the pool rather than you have to justify the cost of having DRAM that is only can be used in these narrow situations. So I'd say the bottom line of that paper is pooling of DRAM is realistic. We should be thinking of being more memory focused, thinking of the problem of how do we get access to the data rather than thinking of it as computing is the focus of what's being done. Yeah. Well, I wonder, and
You said you're not as familiar on the software side. So redirect me if I'm going off into an area that isn't in your domain. But does that require the software explicitly managing the memory? Because I feel like a lot of right now, at least higher level software, can optimistically ignore memory hierarchy. And then you have to start being aware of it when your performance falls down. You're saying, oh, I've broken cache locality or something like that. But when you have this. Yeah, this is a little different. So how do computers work, right? It's how could you have gigahertz processors in this DRAM that takes 100 nanoseconds? How do you make that work? Well, caches were invented. And we just keep sticking in levels of caches and trying to hide.
give the illusion that you have this incredibly large memory and it's incredibly fast, and it's up to the hardware memory hierarchy to hide that from the programmers. For some tasks, that illusion works pretty well, but for others, it doesn't. And so programmers do need to be aware. But I think this one isn't so much the memory hierarchy as it is just the memory capacity, which I think programmers have had to worry about for a long time. But this idea that, well, I think the argument is if you re-architect your software...
to be aware of the pooling of memory across different servers, you have this ability to be able to get a lot better cost performance for data intensive applications. Yeah, that would make sense just in terms of being able to trade off more memory versus CPU and not having to stack them. I was also wondering if you make memory ownership visible to the software, could you hand off between different processes without having to do a mem copy where you basically say like, here's your address, go. Yes, that's kind of this distributed shared address space. I think there's...
I believe there's a whole bunch of CXL protocols, but I think the latest CXL protocols will allow that type of access to the shared memory. But what's different is, you know, it's shared between different servers, and that used to be an impossible thing to do. And the way that servers are constructed, there's just a limit to the amount of DRAM you can stick on the one-on-one server. But using CXL, you can get this illusion of having much more memory for individual pieces. So it's kind of a practical way to get a tremendously bigger memory footprint without a gigantic cost of some kind of supercomputer. Yeah, I remember earlier in my career, I was involved with a lot of high-performance computing and stuff.
Shared memory models, if you wanted to do it across lots of CPUs, you had to go to like SGI had these like mega things or something like that. Anything else you were passing messages essentially over a network bus in some way. So it sounds like in some ways this could enable shared compute, shared memory style programming models, but across some number of servers. Yeah, so that was John Hennessey. So John Hennessey did this project called Dash, which was a big shared memory processor. And this was this kind of switch to parallel computing or that was kind of in the air. And we had a contrasting project at Berkeley called the network workstations. And this is in, I guess this is in the 90s, I guess. I think that's right.
And John thought, well, the hardest problem with parallelism is programming. And if we keep it a coherent shared address space, that's going to make it easier. And so we, at Berkeley, what we said, well, that's one way to go. But we think independent, it's so cost effective rather than have these big servers with this, you know, interconnect to be able to provide that. It's tremendously more cost effective if we could use, you know, we said workstations, the PCs, put a bunch of them together, use local area networks to connect them. And that's going to be, the cost performance of that's going to be amazing. And so how did that settle? Well, what happened is, you know, the internet came along and internet services wanted to.
The parallelism was based on a number of people, not on a single program that you had to parallelize across other processors. So it was a huge throughput demand. So what ended up happening is the network of workstations. That's how all internet services standardized around that. And John's model was more efficient than DRAM, but the price of DRAM at that time was so low and you could scale up. And it was also very reliable that a single server could fail and the software could keep working around it. So it was much more reliable.
much more scalable and much cheaper than the coherent outer space model that those SGI machines did. But it's an example. How do we settle the basic computer architecture? Well, we get companies to spend hundreds of millions of dollars or billions of dollars to put it in the marketplace. And then we fight it out. Oh, that's how we do it. Absolutely. Well, and as you highlight, a huge amount of the demand in compute growth for many, many years was internet-driven, essentially embarrassingly parallel. You just split it out across things, stateless servers that maybe access some shared state. And then you've got databases or things like that that have to actually have that coherent view of state.
Now we're at a scale where that's not super cost effective, needing this pooled memory for databases. Are there other domains? We've talked about machine learning, particularly at inference time being very memory intensive. Databases are another one. What other domains do you think this type of memory centricity is likely to make a lot of sense? Yeah, that's kind of the question when people talk about things. So my examples have always been, besides being interested in data analytics, databases, I don't think I have any other areas that are obviously memory intensive. My guess is...
That'll be increasingly the problem going forward for lots of applications. But it's hard to, I'm not sure. I don't think I have any. No worries. So looking at this and looking at, you know, so much is being driven right now by machine learning. And as you highlighted, like folks doing machine learning, they'll take as much as you can throw at them. I saw something recently where like one of these machine learning coding tools, Cursor was saying, hey, Anthropics throttling us because they are literally out of GPUs. They can't run enough inferences, right? Like everybody is struggling on that domain. But I'm curious, what do you see as like the big unsolved problems for the next five or 10 years?
Well, one of the questions I had is when, you know, in 2016, when I joined this machine learning organization, it's like, well, we'll see how significant this is. And it's very hard to see when you're in the middle of it, whether this is a paradigm shift or not. Retroactively, it's much easier. Like a decade later, you look back, wow. And in my career, the microprocessor, the internet, maybe mobile phones, smartphones. And when you look back, you say, wow, that was a giant change in our technology base. So now that I've been there eight years, I think this is one of those things.
So agree, this is a big paradigm shift. I think that's becoming more and more clear. What are the still big unsolved problems here? Yeah, well, because we're at this, well, I'd say a couple of things. We're not at an upper bound on intelligence. We're not at, you know what, that's good enough, right? I think with intelligence, I don't know if there is going to be an upper bound, but we're certainly not there yet. It's still, there's things where it screws up. There's things where we'd like it to be a lot smarter. So I think on the machine learning side, can we figure out how to deliver something useful that people can depend upon?
You can ask a questions and trust its answers. And can we deliver that economically? And can we reduce the carbon footprint of our solutions as well? That's a topic I ended up being involved in quite a bit of the carbon footprint of this. So I think if we only focus on the part of the industry that's machine learning, there's giant challenges there. What's the best architecture? Can we improve the algorithms to reduce the cost of training and serving? Can we come up with architecture ideas that can reduce?
and hardware ideas that can use hardware. There's just a huge set of problems there. So it's this very exciting time. Like I said, for my career, they've been kind of boring times. This is not one of them. And what's great is if you're a researcher, then if you've got a good idea, people are anxious to hear it. There are times when everybody's making a lot of money and it's kind of boring. It's hard to get people's attention because they're making a lot of money and they don't need to change anything. That's not where we are today. So it's a very exciting time if people are interested in hardware to get into the space or algorithmic advances.
I'd be curious to dig a little bit deeper on the carbon footprint side. I saw something recently that Microsoft is projecting they're spending $80 billion on data centers this year. I haven't seen similar numbers from Google and Amazon, but I know that they're also deeply investing in capacity for this. And there's a lot of worry out in the world around what's the environmental impact, what's the energy impact. So can you talk a little bit about the research you did in that domain? Yeah, I know I've worked a lot on this topic. So I got started in it because I guess in 2021, there were papers coming out. They were making alarming claims. And I would ask my...
friends in machine learning at Google, like, is this true? So there was a paper that came out in IEEE Spectrum, which was the flagship of IEEE, one of the big organizations in the world. And it said, it was 21. He said, I think it's by 2024, training a model would cost $100 billion and produce as much emissions as the city of New York in one month. Like, oh my God, is that true? So we started investigating it. And we found that there was a particular paper that inspired these concerns. And it was actually a paper by a group at the University of Massachusetts.
who were trying to guess what it costs for one of the Google projects. So in machine learning, there's a thing that you do to try and find better models, more efficient models. And that's called neural architecture search. So you're using kind of machine learning to find better models, more efficient models. And so it was actually a more efficient version of the transformer model called Evolve Transformer. So they tried to estimate what was the carbon emissions of that search. And because they didn't have internal Google information, there's a thing that we call, in that paper, we call the 4Ms, which affects the machine costs, which is the model itself. There's a machine it's running on.
And then the third M is mechanization, which is an M word that means how efficient your data center is. And the fourth one, which was a big surprise, was max, because the cleanliness of the energy is highly dependent on geography is where you are. If you're near a hydroelectric dam and you're solar or wind, you know, the energy is going to be much cleaner. So those four M's. So they didn't have facets of the four M's. And so they did an estimate based on averages. So that was fine. So they were a little about a factor of five higher than Google was because we optimized a bunch of those things. But unfortunately, they misunderstood how we did the neuroarchitects research.
So they were off by another factor of 18. So we did a small proxy model to search the space, and they assumed it was informal. So the paper itself that everybody based their work on was off by a factor of about 90, too high. But then they misunderstood what the paper was about. They thought it wasn't searching for a new model, which they do that occasionally. And then they put the model out, they publish it and put it on GitHub and people download it and use it thousands of times. The people who read that paper thought that was the training of the model. And so not surprisingly, it takes more than a thousand times as much energy to find a model than to train them. So we multiply together that the conclusion was too high by 120,000. So you end up with the claims like it's going to cost $100 billion in New York. So now the problem was, now that we knew that, how do we get the word out? So there's no real...
good mechanism because this paper appeared in the conference to say, hey, by the way, remember that paper everybody's citing? It's off by 100,000. So I try to go around and give talks tonight, but it's basically an unsolved problem. So that's got me into the space. I think what happens today is people just have a hard time because you hear these, you talk about tons of emissions. It's hard to put it in perspective. So there's something called the International Energy Agency, which is like an organization for 50 countries. And what they said was recently, how much energy is going into data centers? Well, it's about...
1%. 1% of all electricity is going into data centers. That doesn't include the internet, doesn't include crypto. This is kind of Amazon, Google data centers. And then AI is only a piece of that. Our measurements were that it was about 15%. So it's less than a quarter of 1% is AI. That's where we are today.
And so this IA agency, they said, you know, looking into the future, even with strong growth, you know, it'll get bigger, but it's compared to the other things that are going on. It's not that big a deal. So like air conditioning, air conditioning is going to grow a lot in the next five or 10 years, and it'll be much bigger energy driver. And also they talked about high electrical things like aluminum plants, just plain old economic growth was going to be another thing. So when you put things in perspective.
It's going to grow, but it's small relative to other things that are going on. But it's very hard to communicate that kind of message today because people will notice things. Because what happens is that's the worldwide average. But in particular regions, if somebody builds a lot of data centers in the same region, then that utility company could be taxed. And that's new. So you'll see some cities where, wow, they want to build a lot of data centers here. We have a limited energy supply there, and that's going to tax it. So it's kind of a local problem. But if you have the big picture, it's probably not a global problem. But nevertheless, the growth is so much, and companies want to be able to build these data centers. They're investing.
or investing, or they've made plans for nuclear energy. So these so-called small modular reactors or even fusion, which is hard to believe. I saw that. Yeah. I'll believe it when I see it, but that would be incredible. Well, yeah. The other thing that's in the press all the time is quantum computing. And one of my friends said, you know, there's a chance that fusion is going to work before quantum computing. So it's been one of those that we're going to solve it in the next 10 years, about the last 50, right? Right. But the people are more optimistic in the last five years that we're five years away than for decades, we've been 10 years away. And they are using ML to help.
figured this out some, but it does seem like it's, I can't believe it's going to be real. Obviously if fusion happens, it'll be amazing. But the small, I have a, one of my nieces is a nuclear engineer, the small modular reactor thing, you know, we've had reactors in submarines for decades. They're training 18 year olds to operate, you know, nuclear reactors. So there's, there's another part of the space that technically could be very helpful, but you know, the kind of the sociologically putting reactors into neighborhoods maybe bridge too far. Yeah, no, absolutely. And I think, you know, in that power domain.
So actually, first off, let me restate in case anybody missed it, right? If you're worried about the environmental impact, the paper that you're probably tracing your worries back to is off by 100,000. And the papers, you know, in academia, we count citations. It's citation rates going way up. Our paper says, by the way, there's a little flaw there. We're way behind. We're not catching up. So that's kind of, I was talking to a guy who does data center stuff. And he said, that's what it works. If somebody makes a mistake in a calculation and people read about, wow, I didn't know it was that bad. And then that's news. And once it's news, it is very hard to fix. Even today, if you were to search Google.
about what's the cost of training. You might get this so-called the same as five car lifetimes. That's that original paper that's off by 120,000, but Google will help you find that erroneous result. So another topic that I like to talk about with folks who you mentioned, right? You have five decades of experience here, almost as long as the industry. And you just said something, you've seen boring times, you've seen exciting times. And we're in one of those exciting times. I'm kind of curious with that perspective, what do you see going on in the evolution of the tech industry right now? Like what is changing the most? What's staying the same? What are you excited about?
Right now, it'd be like asking about right after the microprocessor was invented, which I asked some of my grad student friends. I was a grad student when it was invented. They told me I said I thought it was a big deal, which I'm very happy that I said that. Got that one right? Yeah. Because it was kind of a toy. Actually, in the early years, there was real computer conferences. And also, they had these kind of pretend computer conferences where these toy computers were at. They weren't real computers. But yeah, because it's right there, it's kind of hard to see around the AI stuff. And it's got nothing to do with AI that's very exciting.
The quantum computing stuff isn't really about AI. You know, quantum computing is like zero degrees Kelvin almost. And, you know, AI success is machine learning, lots of data. So quantum computing and lots of data don't really fit together. But, you know, I think what Jensen said was pretty plausible. What did he say? It was 15 years is too early, 30 years is too late, 20 years. And I think, so it may have a big impact on pieces of computing in a decade from now or say, but you're not going to have quantum cell phones, right? This is big science. And there'll be some problems that are, that it can solve that we thought were unsolvable. But most things.
As far as we know right now, most will still be regular general purpose computers for many other things, including, as far as we know, AI and machine learning. I'm not a person who thinks that's going to be this gigantic paradigm shift. It'll be an amazing accomplishment to do it, amazing scientific accomplishment, but I don't see it as changing the industry. We just don't know how far AI is going to go, right? It's how much of our own technology will we throw away and replace with AI? So for the people closer to AI, like in the vision community, have friends in the vision community, once there's that tipping point with AlexNet and the ImageNet competition, where it won, and within two years, everybody.
had to abandon what they were doing yet. They completely changed all their courses. So it just, everything went away and it became machine learning. So it was absolutely revolutionary. And that, how much of computing technology will that happen to? Or, you know, where are the walls when it can happen? Because when it does happen, it's hard to see the limits. It's hard to see how far it can go. So we don't know where that wave of AI, how far it's going to spread into the computing field, but it's definitely affecting lots of pieces. People are talking about using large language models to make it easier to do the, what's called register transfer language of hardware design, which is, you know, that's not something I would have.
seriously thought about. So it's hard to ignore how pervasive AI is going to be, but we don't know how, but if we knew the answer to that, I think we'd have a better understanding, but there's this chance it's just going to infect all of our underlying technologies doing things very differently that we've done from the past. And so it's hard to predict given how extensive this change we might be undergoing in the next few years. Yeah, absolutely. I feel like it's already dramatically changing the way people do software development. You can write software with this thing, and it does a pretty good job and can speed you up dramatically. But it also shifts the types of architectures you want to write, which then changes the education path of what type of software is good software and how does it work? Yeah, I was actually involved in another paper, kind of out of my depth, that some friends went writing. And it's about shaping AI for the public good is the theme. And I think the title is shaping AI for...
billions in archive paper. And there was actually, I did an article in Economist about that topic. But I think the way we think a great use of technology is to act as kind of assistant with the human involved. So there's a human expert and it's working in conjunction with the human expert, which makes the human more valuable economically. So rather than getting rid of hundreds of programmers, let's make hundreds of programmers tremendously more productive. And in that paper, it talks about, you know, people worry about job loss. Well, in economics, there's this question of whether it's the product you're doing is elastic or inelastic.
The question that that means is, if you make it more efficient and cheaper, what's going to happen? Well, if it's agriculture, you're making food, there's so much food that people can eat, probably the number of jobs will go down. But in topics like software, where in the past, it's made the number of jobs go up. Even though you bring the price down, there's tremendously more demand for it. So we think if research focuses on elastic fields and improving human productivity, that could have this very positive effect that's going on. But I certainly see it right now that if you're an expert.
And you can use AI to make yourself much more productive. And you can see when it screws up, it's a very powerful technology. On the other hand, if you're a novice, how do you avoid the hallucinations or the mistakes it makes from having you do something that's embarrassing? So it looks like that's where it is right now. Who knows where it's going to be in a few years. But I think it's this reinventing how we write software and how we design hardware is an example of this extensiveness of where AI might go.
Yeah. And I think you're spot on, right? We're entering a period of software abundance, but there does not seem to be, you know, software is like dreams made life. We have no shortage of dreams. There's always more software to write. To me, it's this, I talked about this, no, like that's smart enough. We don't need to go any further. I think same thing for software quality. I mean, the things over my career, the embarrassing stuff is how insecure our technology is, you know, enabling Purple Hill or people in Lithuania to steal money from your grandmother, right? We helped make that happen. So if we can make a serious dent on the security problem, that would be amazing if that could happen. And maybe AI can help us. I mean, that's an example of a quality improvement that would be.
wonderful for our field. So we're coming to the end of our time together. Is there anything we haven't talked about that you would like to share with folks? Let's see, upcoming, I think one of the, in the embodied carbon space, I'm very proud of, I think within a couple of weeks, Google's going to publish a paper that talks about the two parts of carbon footprint is when it's operating and what it costs to manufacture. And people have speculated a lot about.
how expensive it is to manufacture chips. We're going to release a paper or a blog, I think. We're going to have done what's called a lifecycle analysis. And the title of the paper is Cradle to Grave. So the whole lifetime, we'll have the data out there to show us how expensive it is to build these AI accelerators like GPUs. What are the carbon emissions associated with it? How does the operational piece compare to the manufacturing piece? And that'll be the first time that data's out. So it was nice that Google has both environmental experts and computer experts, and we collaborate together. This, I hope, will come out over the next month. And then reading about thinking, we tried to make a thoughtful paper under the assumption that there's people who are...
open-minded and would like to hear about AI, it's less clear to me now based on the kind of Twitter-like instant reactions based on one sentence in the paper, how many open-minded, thoughtful people are out there. But I think hopefully, if you were to read about that, it'd give you something to think about. We try not to talk about the upsides and the downsides, not just the upsides of AI or the downsides, only the downsides of AI. Yeah. There are a lot of outrage merchants out there. Yeah.
Yeah, there was this, Andy Kongwinski is a former PhD student. We did this paper a long time ago on cloud computing, where it was not as controversial, but it was controversial when we wrote the paper, like everything's cloud computing. There's nothing there. And so we kind of explained it. What are the upsides? What are the downsides? How researchers could make it better? He wrote that paper. He said, we need to do that again. But yeah, like you said, because of the outrage merchants. So I don't know if we're making a dent in the conversation, but we gave it a shot. I hope it gets better uptake than the carbon footprint corruption. Yeah, that would be a goal. Can we get as many citations as the paper that's off by 100,000? If we got there, that would be success. Awesome.