Chips, Memory, and Power | Pat Gelsinger

In AI, digital age, energy capacity, equals economic capacity. Why build a new data center and buy 8-1 million GPUs if I can't power that? You're gonna see more and more defaults happening on many of those data center projects because the energy will be there. Whenever you have the technology to make something easy, that means the portal may move somewhere out.

Nothing's a chip anymore. It's a rack that took me three months to design it, but it's nine months until I can actually start to use it. Exactly how many major new memories I've had over the last 30 years? It's a zero.

Memory innovation for the first time in 30 years is not gonna come up. I declared the death of copper about 25 years ago. Eventually I'll be right for all the hardware guys. But it's like Miranda Sats in front of AI, in front of accelerator chips.

I'm sure I don't even know them all. Why are you guys funny? You've funded your furniture too. Historically, they have not been a hundred competing process of endos in any industry ever.

Is this a temporary thing that will convert back to a view? A-she-it as... AI is making it faster to design chips, but actually building them, powering them, and connecting them, maybe getting harder. In this episode, A16Z's Raghu Raghuram and Guido Abbinseller sit down with Pat Yelsinger, general partner at Playground Global, and former CEO of Intel NVMware, to discuss the next bottlenecks in AI infrastructure.

Pat takes us back to his early days designing Intel's processors, when engineers were inventing the tools they needed to build the chips themselves. He explains why AI could bring a similar shift today, even as manufacturing timelines, memory bandwidth, and power become harder constraints. They also debate whether the explosion of specialized AI chips will last, why Pat believes memory innovation is finally approaching a breakthrough, and how optical networking could change the architecture of AI clusters. In drawing on their shared history at VMware, they close with a question.

What would it look like to rebuild virtualization for a world where AI agents, rather than humans, are the primary users? I'm here with our esteemed guest and dear friend and the former boss, Mr Pat Gassinger. Welcome Pat, Pat is currently the general partner at Playground Global, but you're very well known in the industry for being the city of Intel, of course leading VMware, and many other things. So welcome.

Hey, thank you, Raghu. Great to be with you and Guido. And to me, this feels just a little bit like old home, right? We superimposed a year of change, right?

But you both looked the same? I feel energetic, so what's the idea? There's never been a day when you stopped being energetic, so there's no news there. But yeah, let's start actually from your Intel career.

Recently, I'm recently on this Career, this Horowitz and Recent Academy, which was to find talented people between 16 and 22, and then put them into a modern educational setting so they'd be well prepared for either doing things that they're all joining the companies of today. You had a similar experience, like you went to regular trade school and then jumped into Intel when you were about 18 or 19 years back. Yeah, it was really a sort of magical period. I'm 16 years old.

I accidentally went to scholarship. I go to tech school, right? Skip my last year and a half of high school and Intel comes recruiting when I'm 18 years old. And I never went on an airplane.

I'd already fallen in love with computers at tech school. And the interviewer, right? Ron Smith was his name. He writes on his page.

And I was number 12 that he interviewed that day. And if you've interviewed 12 people in a row, you can't tell male from female by the end of it, right? On number 12, he says, smart, aggressive, arrogant. He'll fit right in.

So I got invited to come to Intel and just became really glorious. Starting as a technician, moved into the design team at the end of the 286, engineer number four on the 386, architect and design manager for the 486. All of that went to my bachelors, my masters and PhD work. So it was the best career that you could possibly have.

You're learning my day and you're putting it to practice at night. That's right. That's why I did it. I made a new, completely new era of chips.

I backed muscle, what you loved and what you built. There was a big gap because you're breaking your ground. Yeah, in a lot of ways. And it really was a pretty incredible time in the industry.

And I remember one of the professor classes, he postulated a new carry look at Adderdy's side, right? One of my classes is Stanford. So I said, well, I'm working on the 386 carry look at Adder. So let me go try it.

So I come back two days later to the professor and I said, it doesn't work. I started arguing with my professor at Stanford. It was like, what next two class periods were just me and the professor arguing over why his design didn't work. He had a book that was being published with that design.

It was very annoying. Oh, it was a time-susterer. And we started built in self-test. And I remember Ed Mikoski as Stanford was my professor.

And he was sort of the father of that. And we had wonderful arguments around why most of his ideas were bad. They were practical for real chips, but I was putting a built in self-test to work in the 386. John Hennessy was my thesis advisor.

And obviously he did okay. And I helped his career and became president. And he's not okay since then. So later on, we were talking about the SIS courses.

I had the same experience when I was teaching at Stanford. You come with something that you think is the truth. And then this student comes that works in industries. Well, actually, not quite.

So the next lecture you come back with, okay, I looked this up. You're right. I had this. Yeah, that's part of it.

You know, that interaction. And I think that's part of what makes Silicon Valley so unique, right? You know, as you have so many companies and entrepreneurship, it's like in the water here, right? Starting things up and new ideas and challenging the professors.

And you can't be a professor at a place like Stanford to not expect to get challenged and enjoy it. There's a good traffic in and out that I have every tower. Yeah. Yeah, now 486 was your big accomplishment.

If you've obviously lots of people worked on it. And you guys broke some new ground and chip design there. And increased what the function of the chip was supposed to do and so on and so forth. Yeah, it was a pretty magic period as well because it was before what you would think of as the EDA industry.

Oh, wow. Yes. And the end of a little bit of the 386, but the 486 was really the first chip to implement what you would think of as modern EDA techniques. We did a high level design, right?

A description of RTL description, but there was no Verilog. So we invented HDL, right? The Intel hardware description language. Oh, wow.

I wrote my own language. Well, you had to build a compiler for that language, right? So we created a compiler for the language. There was no automatic placement route.

So we had to invent that and we worked with Alberto Sanji, Avonji, Vincentelli, Berkeley, and some of the students for the first place where the first routing, the first automated timing management. So the created EDA to something? Yeah, to a great degree that, you know, that was really one of the hallmarks that not only was the 486, this compatible pipeline microprocessor. But we ushered in many of the foundations of what became the modern EDA industry and was really a pretty magical period of the industry.

Yeah. Yeah. So if we look at today, will people look back as Blackwell and saying, this was of the last generation that was done before AI took over the design process for micros? I've been a similar transition right now.

I think there are surely aspects of that. And I think when you look at things like Jalapeno today, right? You sort of say that's sort of a first principles use of AI in the chip design process, right? We'll throw out a lot of those things.

Now, there are pieces of design that are still really hard, right? And that's a lot of the analog aspect. Surdies is probably the best example of that. You know, we're still not AI-able, right?

You just need lots of silicon data, right? To get those. So you either get so conservative in your analog design that you're able to say AI it or, right, you got to do that as the hardware. But now many of the logic functions can really be done with AI tools and techniques in pretty incredible ways.

Right? Your transistor budgets are big enough. The design tools are getting smarter than a few experts guiding the tools and how to apply it. And I really think it will be somewhat like the 46 that way, right?

We just sort of look back and say, yep, that was the beginning of a new era of chip design. Whenever you have the technology to make something easy, that means the bottleneck moves somewhere else. Can we guess at this point where the bottleneck will be in the future? There's something which seems like AI is incredibly good, right?

Like going to software layers, you know, writing kernels, a lot of the, I think a lot of the sort of being able to specify the objectives at a higher level, which then gets translated into a very long, right? So what is the new frontier? What is the thing that will be difficult going forward? Well, when you look at today, so let's say we don't want Pat, we're off to do a great new chip together.

We're going to do the next one. Right, right. This is the one that's going to leap ahead because we have understanding of the AI. We have understanding of the AI workloads.

We have understanding of how to compose that into the right multiply, accumulate, register structure. Perfect visibility model workloads of tomorrow as well. And all that kind of stuff. And let's say we turn our AI tools loose.

We are great within three months. We have an awesome design, right? For that we've been a little bit conservative on all the analog components of it. Now we still have a couple of really major bottlenecks today.

One is, right? We still have nine months of silicon processing time. Right? Something like that.

Right? So I can design the thing in three months, but I can't actually get it into real silicon at scale for nine months. Okay, that sucks. Right.

So we have to really see that bubble neck and prove if we're going to have this innovation. Because we know. One of the validation of this is going to be on that. Well, first, I can't get anything out of fab in less than three months.

And that's if I have supercharged design flows. Right? So I said, you know, how do I compress that? Right?

Yeah. You know, for it. And then I got to get into advanced packages, 3D packages, I get in complex. So that's like another three months.

Right? You know, and then I have to put it into a rack scale solution because nothing's a chip anymore. It's a rack. Yes.

So we're nine months. It took me three months to design it, but it's nine months until I can actually start to use it. So how do we start to compress those aspects of design? I think we need new forms of lithography to enable that.

We need design flows that don't require $50 million of mass cost until I can get things into prototyping. So you know to me, how do you compress that to a month or two? Right? Because I can now do the design in three months.

How can I have that done in a month or two? Because it takes me a year and a half until I actually get scale and software. Okay. My understanding of the AI workloads is no longer applicable to the chip that I design.

Right? So you know, always. You just play off live right now. I think it was an edge up.

Right? It's almost, you know, we can look at it historically. It was like, you know, Graphcore wasn't a bad design, but the world moved on, right? You know, and right, you know, in that.

So all of those aspects somewhat obviate the fact that AI has made the design piece easy. All of those manufacturing scale issues to become a huge bottleneck. Right? Another bottleneck, of course, is memory.

Yep. Right. You know, memory bandwidth. We don't understand anything.

Right. And you know, for that, as I've described, HBM is a hideous memory. That's just the best one. We don't have to go about that.

Yes. It's a terrible memory. You know, I get terrible bit density. I'm limited to shoreline bandwidth.

The power, right? Issues, thermal issues. D-RAM doesn't like to be hot. I'm saying everything in one spot.

Yeah. That is a memory compute workload. Yes. Right.

So, right. So, you know, for that, so I need to, you know, compress that development and embodiment into real scale. You know, I need to fix memory, right? Associated with this.

So to me, that's another critical bottleneck for us to attack and power, right? You know, power in, you know, thermal and most of the power simulation aspects today are bad. Right. You know, you don't have good 3D modeling tools for power, power dissipation, hot spots, etc.

You know, my voltage in, right. You know, I'm essentially guard banding almost my full power envelope. Correct. Right.

So essentially I'm losing, you know, 40% of my power in guard banding. So I got to get much better at power modeling, you know, and then my power dissipation techniques are bad. Right. So, you know, The low remodelers are changed too.

Yeah. So all of these things, you know, to me, those are the next big bottlenecks, right? You know, how do I improve each one of those? And now, right, you know, if I've done that, okay, now we're on to an exciting era for what's next.

So, you know, thank you for those great AI tools, but, you know, I got big challenges, even when those are fully deployed. And so, we've seen sort of a rebirth in the number of chips and chip architectures and so on and so forth. Yeah. Yeah.

Yeah. So, you know, I'm trying to make different designs assumptions either in the components that you talked about or in the part of the credit cards that they're targeting. So do you think there's room for all of these? Or are we in a world where partly because of what we talked about, you know, tooling, we now have the ability to build more different kind of chips.

Yeah. The time is enormous, whichever way you want it to. Yeah. That is even more provocatively.

Historically, they have not been a hundred competing process of enders in any industry ever, right? Yeah. Yeah. Is this a more heterogeneous future?

Well, we see, is this a temporary thing? It'll converge back to a few. Yeah. And, you know, I see it as a more temporary thing and it will converge, is what I expect.

And there's probably three different reasons that I see that to be the case. You know, one of them is, you know, this emerging heterogeneity that you see in the compute environment, where, okay, now I have specialized pre-fill versus deco. Well, you know, and now people are saying, well, we really need to specialize tier for mid-fill, not just pre-fill, right? So now I have early pre-fill and now I have mid-fill, which is, you know, different.

Let me speculate and verify. Oh, yeah. Listen to open ag here, right? Yeah.

You know, and all of a sudden my compute fleet becomes more and more granular across the workload. And I think any time that you've seen that in history is not sustainable. And then all of a sudden, right, when, you know, as you look at that, and now people are saying, well, as I go to reasoning models, I want something that looks more like a CPU again. So, you know, all of a sudden I didn't want that level.

You know, that didn't become the predominant portion of my compute capacity. I needed more of it over here, right? And model and model workflows and so on. You know, so I really don't like, I'll say at scale specialization, right?

Because I think the workloads are going to continue to moderate and migrate so significantly. First, you know, second is I think today's models are about to go through some evolutionary breakthroughs as well, right? You know, I think the LLM is sort of reaching the limits. And that was people look to how do I really model 3D, right?

You know, where, you know, a flatish LLM doesn't work well when you go to molecules and chemicals and, you know, right, imaging, you know, so I think we'll see limits of that that will cause different shifts in the algorithmic domain. I also think that some of the most interesting workloads become where you start, you know, also bringing HPC like things back into AI like things where all of a sudden, you know, 64-bit provision precision counts again, right? So I see... What are we an example of that?

Well, you know, imagine that my AI model is now inducing the five most interesting domains for a chemical analysis, right? Okay, those chemical algorithms are going to run with high precision, right? You know, which is going to be the dominant piece of the workload, right? Getting to the five algorithms I need to run or running the five algorithms, right?

On different chemical or biological systems. So, you know, I do think that, you know, or as we optimize our AI systems, they will lead us back to some of the things that have been more traditional, high-performance computing. So I see that aspect of workload as well. So, you know, all of that said is, you know, this extreme specialization to me in the compute architecture, I sort of don't like it because I don't think it's going to be what the workloads will look like two, three, four years from now.

So that's one reason I don't see all of these AI chips as they get more and more specialized for portions of the compute workloads necessarily be right. You know, second is, right? There's like a hundred of them now, right? You know, there's multiple opt-of-the-ones, there's probabilistic ones and so on.

You know, they're not all going to win and they're not all going to win because at the end of the day, you have to get scale on these things and scale requires, you know, you have to win, you have to get capital, you have to get workloads on to it. So I think it sort of defies logic that you're going to have a hundred of these things. So I see them, you know, coming back as a result of, you know, simple, you know, capital market share, etc., which are the winning teams, you know, winning designs, winning architecture. So I see the workload driving that, you know, I see the natural, you know, industry consolidations, you know, that will occur.

And then I also expect that the winners will pick some winners, right? You know, where, you know, an open AI and NVIDIA, any anthropic will say, I like this one because it isn't just hardware, it's how do the hardware and software evolve, right? You go forward and I do expect that this going to be, you know, here's this 10, I could pick for this, I like that one. And it does require investment in the software and the workload evolution that take advantage of those platforms.

So for those three reasons, yeah, we're going to see, you know, a narrowing of the field in that sense. The big one on complexity as well. Yeah. So we go ahead and make a total sense.

I think I agree in most mind that we still try to store a little bit of controversy here just to make a good little story. You don't start encouraging. No, no, no, no, no, no, he's not going to change that, right? Yeah.

So obviously, how do companies, what do you say? So consolidate, right? And completely agree with your argument that at the end of the day, the large consumers will pick winners to some degree, right? Today, I think to be a first year, chip startup to some degree, whether you're that or not is defined by how much traction you have with the big guys.

That's it. And this was an argument to say like in any kind of transition period, you probably have a bit more explosion of the project as well. Now, all that said, I think there's an argument you can make that basically says, historically, we standardized on one or two architectures because different instructions that were hard, you have to build multiple compilers, like they basically, the software complexity dictated that you can only have a certain one of our platforms. That part seems to be changing with AI today, writing an optimized kernel for, I mean, I think I'll let you write an example, but that's like, look, humans can't do the program this thing anymore, but an agent's one more one night can do it.

That's all we need, right? So basically, you can say, if you can build a chip that has a certain capability, it's not much, much easier to build the software that fully leverages that. Could result of that, either we have a little bit more heterogeneity, just because it's the easy to program that hetero-dentering on. And I do think it changes some of the, you know, where the bottleneck lands, right, changes in that regard.

The first bottleneck is the one I point to that we've already talked about, still takes a year and a half to build these things, right? And then the second bottleneck is not only does it take a year and a half to get these things into racks of scales or something like that, now to make them useful, okay, well, could I borrow 50 billion? Right, I mean, you're not talking about small capital expenditure, you're talking about very large capital expenditures and data centers, you know, power, you know, commitments, so I kind of build this thing at scale, right, so that's going to defy a lot of heterogeneity as a result. Now, you know, clearly big players, we saw this with what, you know, NVIDIA did with Grock.

Okay, you know, we're going to sort of own some of that heterogeneity, right? And you know, two years from now, you're not going to tell what Grock is inside of NVIDIA, right? It's going to be like another example, right? You know, so like that, so you'll see the architectures, you'll sort of hide the heterogeneity, right, inside of it.

But to me, it's just fine, right, you know, I mean, that's what layering, that's what extraction has always done associated with it. You know, but I think those forces are powerful ones to, I'll say, expose a lot of the heterogeneity. No, right, I think we'll get abstracted underneath, you know, some of these, but at the end of the day, it's so hard to build these things at scale. Earlier, you talked about HBM being hideous memory.

Mm hmm. By far, the best one we have today. Exactly. The dark thing about democracy.

We can't recall when there was the last memory innovation. It might have been obtained, I think you probably had to do that. Yeah, that one died. Yeah, that one died.

We killed it. Yeah, we killed it. So do you think memory innovation is around the counter? Or what needs to happen for, obviously, we understand the scale problem because it's even more larger there.

But if you keep the scale aside, even from a technology point of view. Yeah, and let's, you know, look at that history. I've probably personally been associated with at least five different new memory architectures obtained just being one of them, right? That did not see the light of day, right, as well.

I'm probably familiar with close to 100 that have happened in the industry over the last 30 years. And exactly how many major new memories have we had over the last 30 years? Which, you know, okay, D-Ray, M.S. rain flash.

Okay. What else is there? D-Ray, M.S. rain flash.

Yeah. So I do think- Flash stacking comes for a little bit. Barely. Yeah.

Right. But anyway, right. So, you know, in that it has been, right, you know, disappointing that we haven't created, you know, the next physics, you know, that enable, you know, low cost, high performance, resilient, manufacturable memory, right, you know, for it. So I do think, and now, and over the last 30, 40 years, you know, memory was a terrible industry, right?

Because you would make money one year and then you'd lose money for the next four, right? So let's go do R&D in an industry that makes money one out of five years, right? You know, it's just always, you know, it's just been so terribly difficult because it's been such, you know, driven by such violent commoditization cycles. That has changed a little bit though, recently.

Over the last five years? Happy days, man. So all three big members of Amazon are the top 20 most valuable companies on the planet. Yeah.

That's a crazy change. Oh, it's just so, right? You know, who would have thought, right, associated with it? And anybody who looks at the AI workloads, I mean, it's always, you know, it's much harder to, you know, it's easy to predict the past, but it's a little bit harder to predict the future, right?

And, you know, in that you look at it and say, AI is a memory work site, right, you know, inside of it. So I think there is the combination of capital and technology means that we'll justify memory innovation. Now, there's two domains that I'm excited about, you know, for it. You know, one is, you know, this idea, you know, I do think that the shoreline ban with limitations of HBM is a fundamental issue.

Right. So with that, the idea, you know, and obviously, you know, we have a company D matrix that's doing this, but I think others will be, you know, whether it's your share of risk, you know, and their techniques. It's just that. Yeah.

Somehow you got to bring memory and, you know, you got to bring memory and compute together, right? You know, in the fundamental new structures, I do think that given the workload now has such capital being deployed against it in the AI space, that I do think will actually be able to break through some of those physics challenges as well as well. So I do believe and, you know, I just want to just, you know, I just funded a new memory company as well, that's still in stealth. But, you know, I'm quite excited about some of the innovations that will occur in new memories.

And you got to see new materials, you know, for memory people, a lot of people looking at things like pharaoh electrics, you know, finding non-capacitive, you know, high-density memories, finding memories that can, you know, be high performance, high-density and stackable, right? But, you know, people are looking at low-cost flash, you know, for that, I'm not particularly enthusiastic about that idea myself. I think, you know, speed and performance of the underlying cell are somewhat problematic with a flash or an M-RAM structure. You know, but I do think there's memory innovations are coming for the first time in 30 years.

But I do think that's going to be an exciting space. And, you know, we'll have a couple of companies there, I'm sure. Hopefully, we have a few. Yeah.

We'll be sectioned. Okay, you know, but we'll find some things to do here. But I think memory innovation for the first time in 30 years is not upon us. Yeah.

Yeah. It's at least about a couple of $10 billion of market capital. Yeah. It's really okay.

Sorry. Yeah. I think the memory industry is now increased by market cap by $2.5 trillion over the last four years. I think that's about right.

Of course, just invested there and not done any of our venture stuff. Yeah, exactly. Use your money. How long?

What's your view? Sorry, go ahead. Yeah. To expand a little bit on the stacking part, if you have to predict how quickly will our chips get taller?

Is it going to be like, you know, we're going to go to an incentive for a while, or will we have the 60 layer sandwich, you know, happily mixing logic? Yeah. That's right. I'm pulled down.

That's right. Actually, I was going down at them often. But how do you see this? Yeah.

I think there are some. The problem with stacking is now essentially the value of the stack right becomes greater as you go to higher dimensionality of the stack, which means the yield of any individual layer in the stacking process has to keep coming. Essentially, it can't go up linearly. Yeah.

Right. It has to go up exponentially and improving the manufacturing and defect characteristics in an exponential way. When you get to eight or 16 high stacks, gets to be so, so hard when you run the math on that, associated with it. Your main factor flows have to be so close to perfect.

Well, they could see designs that take into account a certain model. Like we've seen it for a long time. But remember that was the nature of D-ring. You always have to have block replacement until spare lines and so on like that.

So not in any way, right? Minimizing that, but assuming you're doing that, you still need the yields to be so good. You know, if I have a crack die, I don't care how much resilience you've built into it. You can't, you know, it's a bad die.

Right. So, you know, those kinds of, you know, so as a result of that, I'm not, you know, a crazy 1632 stack, you know, kind of guy. I think there's going to be, you know, nice three, four, five stacks, you know, that just end up sort of being the sweet spot. The midrisers.

Yeah. Right. You know, for like, and I expect for, you know, part of this manufacturing, part of this complexity, associated with it, you know, where's those yield sweet spots associated. So that's sort of where I think most things land.

But even there, right, you know, so let's say I now have, you know, I now have a great compute time, right? You know, how big does that time lead to be? I mean, everything's become reticle limited now, right? You know, can I go to more chiplets that better yield characteristics, you know, better thermal, you know, and then you're gonna say, I want a memory stack on that.

You know, is it a two stack memory, a four stack memory? You know, I don't see it becoming an eight stack memory, you know, so to me, two or four sort of, right? You know, but then power delivery needs to get integrated into that. So that becomes sort of another element.

And I want that to be mentioned, right, associated with it, because, you know, I want my power rail coming into this as well, right? And then I need fairly intelligent RDL layers to distribute signals across that platform. So to some degree, you know, my canonical thing is already about eight layer stick when you add up all of those pieces associated with it. And I think that's about all you can do, right?

At least as we understand manufacturing today. And then, of course, you're way to buy the optics to come in. So you're gonna have more three, five material structures, you're gonna be closer to this complex, because I want, you know, my IO getting, you know, essentially, you know, deeply integrated into that construct as well. You know, so my silicon centric pieces need to get complemented by some of these other three, five materials.

And how do I get optical connectivity built into this? How do I get power rails, you know, built into that? You know, that is a, you know, I'll say that is an engineering feat and a manufacturing nightmare to bring all those pieces. But that's where the physics is taking us over time.

So because of that, you know, throwing a 16 high stack and say, you know, it's like, that's crazy, you know, two or four high memory stack. You know, I think that's probably where the sweet spot will be. And didn't talk about the thermal dissipation and new materials needed for that. Yeah.

And obviously, there's going to be a lot of, you know, first thing, right, you just can't have some of the, you know, the physics hotspots that you have today. So you have to be more intelligent about how you use the heat and spread the heat. But you know, then you're going to be saying, okay, how do I both get power in and dissipate power out of those environments? And by the way, if I can, you know, get much more fission power in at the voltages, I want it with much better power responsiveness of my voltage regulation, you know, then I don't create as much thermal chimneys.

But I do have to think, okay, how do I cool these things? And you know, what role do new materials, diamonds and other things have been postulated as ways to do a better job of the thermal cooling? How do I then have better, you know, liquid cooling techniques and others, you know, sitting right on it, you know, with good thermal characteristics. So yeah, you know, we engineers are becoming plumbers.

Yeah. Everybody about let me see the chip scale. Yeah. Let me ask you about an operate from some people have said, Hey, let's get the memory off of the package and just go all optical between the compute and the memory.

And that'll give you more degrees of freedom and more ways of building things. What's your view on that as opposed to stacking that's not extreme? Yeah, you know, I think the more OEO, right, interfaces that you have, you know, optical to electrical to optical to you know, the more of those that you demand in your core compute complex, it means just hard, right? In that sense, and you're always going to have losses associated with it.

You know, people are working hard to come up with more clever techniques to decrease, you know, both the cost and the power, right? You know, how much of the power am I wasting as I go through my, you know, my Dax and your conversions. So I'm not a fan of those. Yeah, because I think it gives you access to large pools of memory, right?

But it's a lot of power getting there. Yeah. Right. And remember, my, you know, the essentially, you know, the FENTO JUULS per compute is now actually pretty amazingly good, right?

The FENTO JUULS per bit of communication is comparatively 1000 times worse, right? And if I'm spending that much to go get to large pools, so I don't like those approaches, you know, myself, I think it's somewhat defined the underlying physics of the solution. I mean, and then there's sort of the other approach to say, like, we'll kind of actually integrate compute and memory directly, right? Shift some of the compute into the memory.

It's some of the PIMA practice. Yeah, exactly. So what, what are your thoughts about that? I'm not a fan of those either, right?

Because I think workloads, you know, because now I have to take my workload and I have to like push these little pieces of compute into the memory, right? You know, so that, you know, I don't need as much bandwidth and memory coming into my real compute, right? And that puts constraints on workloads. And again, you know, I could be wrong in this particularly some of the design flexibility and the intelligence that we can now bring, you know, through the design aspects.

But most PIM ideas, you know, have been around for 25, 30 years. And I think 25, 30 years from that, I'll still be hanging around, right? In that sense, you know, give me great memory bandwidth, you know, that I can have, you know, high density, high performance and get it close to the real compute engines. I expect that's going to be a more generalizable solution across many, and evolving workloads over time.

Well, so we're still stuck with denser, denser tips, more compute, more power, better IO. I am a huge fan, right? I've got to, you know, moving optics for all IO functions, right? You know, I declared the death of copper about 25 years ago.

Eventually, I'll be right. You still right. One day soon, I'll be right. But I think it's that, you know, to me, all IO should go to optical, right?

You know, but that core compute memory complex, no, right? So when do you first step for going on optical is to scale up even more shows them scale up? Yeah. What's your views on that?

Yeah, you know, and I think we're now sort of be like the application of the workload driver, if you will. Well, I think the workload driver is the one in front of us, right? I mean, you know, today, you know, our scale up environment is we're putting tons of copper, right? You know, essentially, copper is becoming a waveguide that's getting shorter and shorter, right?

And all of a sudden, you know, to make my copper work at, you know, five meters is more expensive than making optical work, right? At 100 meters, right? You know, so, you know, I think the physics is driving us there today, you know, and you know, so if that's the case, Pat, why haven't we moved there already? Yeah.

It's a complex supply chain that, you know, has never been tested at scale. And I think now, you know, I think everybody is looking at, you know, some form of NPO, CPO, et cetera, you know, in the 28, 29 timeframe, because, you know, I can't scale up to large radics, you know, compute clusters from my scale up environment without making that move. So, you know, clearly NVIDIA has indicated that I think everybody else in the industry expects, you know, that's the conversion point. You know, in reality, I should have never built an NDL 72.

You know, it's an engineering marvel, it's the manufacturing nightmare, right? You know, it took the industry, you know, one extra year and a half to digest that beast, right? It really is amazing that, you know, we were able to get to scale and, you know, so on. But, you know, that really proved that had we had a more mature optical supply chain, we'd be much more advanced today, we'd be able to build better scale up environments today, you know, better, you know, power, better cost, better energy, but we didn't have the maturity of the optical supply chains to allow that.

And those things are not trivial to get resolved. There's a lot of work to happen, but I think 2829 is the... Not capacity. Good.

Is that the most yeses? We're currently gaining a lot of experience how impact soft-works are networking, right? I wouldn't. And I think many of that will translate nicely to use.

But all of a sudden, you know, it's like, okay, these things have to work, you know, at 10 million GPU scale. Wow, right? You know, it has to fit in packages. And, you know, there's just a lot of things there, you know, and many forms of optical don't like bad thermals.

Well, guess what? Bad thermals is the nature of the game for it. You know, so we have thermal problems to deal with, you know, package integration problems. We didn't have enough, you know, laser capacity in the industry.

So a bunch of things, but I don't see anything there fundamental. It's just a lot of hard work at this point to pull all those pieces together. In terms of the network architecture, there's also mean-a-switch, what's more circuit-switch networks, just because, you know, you can't scale these very fast optical networks indefinitely. Well, you know, I like Nick McEune, another champion for professor here as well, you know, I mean, you know, a little bit also, you know, you know, but to me, he describes it well, you know, we built, you know, networks to be able to handle any packet going anywhere, but no knowledge of where it might go.

When you think about AI, you know, it's almost exactly the opposite, right? You know, we're doing flows that are large and predictable. So in that sense, just the very description says, hmm, that looks more like a switch than like a packet. So I do think this confluence of moving to optical, right, and, you know, having a workload that is far more predictable, manageable, you know, large and flow, you know, to me, you know, I do see that, yeah, we will go all the way to bring optical into the compute complex, and we will all the way to OCS for some form of optical switching, you know, with more predictable flows, you know, I mean, that is the right architecture.

And then you sort of say, is that scale up or scale out into some degree of scale? It's sort of dissolves the boundary. Yeah, right, if my scale up is big enough, then yeah, you know, just saying, okay, what's the rate of my scale up environment? And when do I want to hook it up to other clusters that are really big?

And to me, that ends up being a pretty beautiful answer, you know, to the future. But I'm looking at somebody who's way more expert on this than myself. So now I look at it, I always found scale up was a scale up to some degree in artificial distinction. And you know, network comes from, right, as we're kind of using different protocols, when it's front to end when it's back in network, and so on.

So right now, practically speaking, it is a it is a big difference. If you take a step back and look at it from a system architecture perspective, it achieves the same goal, you know, with slightly different flavors, slightly different protocols, sit over time, I think will convert. Yeah, I mean, especially for the training workload. Yeah, although I mean, training and inferences, actually look fairly similar.

So the inference corpus lesson, yeah, now I'm going to, you know, right, does your brain stop learning? Yeah, right. You know, you know, by the way, it's another characteristic of workload evolution. I do think the, you know, the separation of training and inferencing is going to become, you know, less so going forward, not more so, right?

And that's a little bit of the argument against, you know, more and more heterogeneous architectures is I think continuous learning. You know, I think that'll be algorithmic domains that will say yes, continuous learning, you know, becomes the nature of the algorithm where yeah, I do want my inference environment to be updating my model weights because you know, that's giving me more and more refinement, you know, evolution across, you know, workloads moving into environments that are less formalized training, as well as smaller data sets and those environments. So I'm not sure that's a number of nodes you need in order to be useful. There's a smaller diameter for inference that's changing, but today that's true.

I still think there might be a role for heterogeneous coming from a different angle, which is that different types of AI workloads require very different underlying good, like some more compute limits, some more memory bandwidth, a smaller distribution workload also, like the optimal cart for that looks very different from, you know, top of the line, you know, 15 trillion dollar 50, 50 trillion weight LOM, right? That means a massive scale. I think that might drive your attention. I think there's some room for that.

But I also say, okay, now I'm talking about putting a million GPUs to work in my data center, right? And you say, well, you know, for that particular diffusion workload, you know, I should really put something optimized this way, right? So I'm not going to start, you know, putting, you know, 1000 of those in, right? And for that workload, I need, you know, 2000 of these in, and you know, for some, you know, all of a sudden, you know, you don't get some of the flexibility of scale, right?

That you would say, and you know, how much does it cost me to manage and upgrade and, you know, connect and, you know, fall to a domain, you know, all those types of things. Ages will do up all the other things. So that's not a lot of both. Yeah.

Yeah. Yeah. Thank you. Everything gets easy.

So good out of here. Yeah. So fleet management is done. So we talk about what computer memory networking power was the next one.

Yeah. Yeah. And you touched about it with what it will delivery inside the chip on the chip. But I think it was 800 World DC.

There's power delivery to the data center. There's an entire ecosystem all the way to power generation and the social political dimensions of that and so on and so forth. Yeah. Yeah.

So that is innovation coming in and very, just like its pure execution. Well, the, you know, if we start at, you know, first principles here, you know, our nation has done a terrible job with this energy capacity, right? You know, essentially, we went through 10 to 15 years, where essentially I was taking coal offline at the rate I was adding renewables and essentially the nation was flatlined in terms of overall energy capacity, not good, because in a AI digital age, energy capacity equals economic capacity. Right.

So essentially, my economic capacity as a nation flapped for 15 years if you buy that thesis, which I think is very, you know, provable over the last five years, I've, you know, we've seen this enormous influx and it just give me more capacity. But you know, I think, you know, even with that, we've gone to maybe increasing our national energy capacity, four percent per year, something like that going from, you know, essentially zero to one, right? Wow, four X or four percent. Right.

So to me, you know, that, you know, it's a bad situation. So we just need more energy capacity for it. So number one, and obviously, we have some companies in that space. We talked about nuclear operating as one of those with Alva.

And, you know, I think nuclear, right, you know, with base load is, you know, fabulous thing that we want to go build more of, you know, unfortunately, all of our renewables have deep dependencies in China, which is very unfortunate, you know, for us, you know, we're, you know, you know, gas turbine lead times are only eight years at this point. Right. So, you know, you know, I mean, we really have a difficult environment to scale up rapidly, but one, we just need more capacity and fundamentally energy capacity somewhat dampens how you fork we can get on AI, right? Because you know, why build a new data center and buy, you know, the million GPUs if I can't power them, right?

And I think you're going to see more and more defaults happening on many of those data center projects, because the energy will be there. So you're saying it'll be a significant head of it? I think it will be a head what? And I think you're already starting to see some of the first indicators of that, the Oracle, you know, comment, you know, I think that's the first of what you're going to see many now.

It's not like we're going to slow down as a result. But I think people will start saying, do I really have the energy, you know, to underwrite the capital commitments that I'm making, you know, for, you know, cement for data center racks and GPU purchases. So I do think it's, you know, gonna be, you know, an upper bound, but we need more energy capacity and we need to get more innovation into that and we need more of those supply chains in the US, nuclear, one of my favorites, you know, one was the last nuclear reactor that came online in the US, right? You know, it's 20 years ago, right?

You know, we just stopped, you know, a very capable industry. So that's one we need more, you know, we need to get much more efficient at delivery, you know, the power networks. As you said, you know, moving 800 volt DC data centers, I absolutely, you know, right, you know, and being able to eliminate so many conversion steps along the way because of the historical reasons and, you know, standardization reasons. Okay, we got to fix this, you know, 800 volt DC, the revenge of Edison is upon us.

Yeah, so I do think that ends up being we need better power conversion. You know, we have a couple of companies, one of those vertical, vertical, you know, vertical, GAN, you know, I think will be, you know, killer technology, you know, to be able to do 800 to 48 or 800 to 12 or even 800 to five and one conversion step, right, which just gives you a lot more energy. You know, we touched on those transformers. Yeah, you know, all of those are, you know, all of those are, you know, all of those are, you know, all of those are, you know, all of those are, you know, all of those say transformers, switches, et cetera, you know, rebuilding, you know, the power delivery network and I think that will, you know, have a lot of innovation, you know, sort of like, okay, power is cool again, right?

You know, you know, for it. And then you're going to need all of those in the cooling systems as well, right? You know, so there'll be a bunch of innovations in that, you know, so, you know, turbines and refrigerants and, you know, all of a sudden they get to be exciting new technologies again. So, you know, for all of us hardware guys, this, yeah, this is like Renaissance and Brannas, it's going to be 25 years, I had to go back and read about Cardinals, Cardinals, artificial season, oh, yeah, it's like, how does that first of all, if they're about dynamics, working as a business, I think after all here, so it's going to be, you know, exciting time that way, you know, but, you know, we also say, you know, we need to change the physics, right, because CMOS has to improve, you know, the essential power per terra, power per terra flop for the last five generations of NVIDIA chips, you know, flat line, not okay, right?

And that's where some of the things, you know, we have a superconducting company, Snowcat, right, which affords the opportunity to fundamentally change the physics, you know, a thousand times better power performance, so I do think some of those innovations are upon us in the near future. I've been a terrible time manager here for the past hour, so so many topics you have to cover, we've been covered, but we've got to cover at least one before we leave. Okay. Because now you're in charge.

Yes, so I thought, now VMs are back. How do you think about that? I'm a former CEO of VMware. Well, you know, I feel excluded from this conversation.

Not a lot of the other CEOs. But go ahead. One of those I was going to talk about networking as well. So, but I don't know what the time is.

You know, hey, I think the, you know, every innovation has always led to the next abstraction, right? And I think many of these ideas and, you know, I would like to say, you know, what's new in our world today? You know, data structures, algorithms, you know, and abstraction, you know, we bring it back to those, you know, foundationals and hey, you know, I think, you know, the virtual machine abstraction, you know, whether it's done from the infrastructure level or from the application level, you know, I think it's still a foundational abstraction model, you know, that deserves to always have a place in the compute hierarchy. So, you know, so what do you think?

One of us go back and run the M-ware? Yeah, that's what I tell a lot of times. I think I told you those, let's talk to go back to one of the M-ware again. Yeah, yeah.

And, you know, and I do think some of these abstractions, you know, but you also think about it. And like one of the things we did at the M-ware, we, you know, essentially, you know, managed every aspect of the computing hierarchy, right? You know, when you manage the workload, you manage the network, you manage the storage system, and how do you abstract those? And I think in the AI context, right, we think about, you know, these agents and worms, well, who's going to manage the agents, who's going to create the security profiles around all of the agents, who's going to manage the performance of the agents.

You know, essentially every fundamental element of virtualization and management needs to get recreated in this next computing hierarchy. There's going to be lots of wonderful companies that you'll sort of break through for how those things will be, you know, done. Do you want to go into the network, Jake? I love to stay with the VMs for a second.

The most interesting thing for me is that we're now building VMs so that they useable for agents as opposed to useable for humans. What we've seen in some other companies is that the changes a lot, the form factor that changes a a lot, how you market them, right? You have to make the agent to make the pickle for you for your VM offering. It changes a lot, how much complexity you can have, it changes how quickly the startup should be, humans a lot more patient than agents.

So it's, you have to rebuild VMware today. How would it look different? Yeah, and I do think, the aim of agents, right? Yeah, and I do think there's, does there need to be a VMware for agents for humans, right?

I see what I mean, yes. Right. You know, in the sense that, you know, I need humans. Let's stick to the agents that way more.

Yeah, but I think it's an important point because I do think that there needs to be, you know, and I'll use, you know, Dario's constitution, right, or guardrails, right, or being able to set policies. And I think then you really say, okay, every aspect of the virtual machine, as we know, we love it today, it is servicing agents, right, as opposed to servicing hardware and humans, right? So I think, you know, that becomes your fundamental design constraint on this side. You know, how do I make agents great, right?

How can I make them secure? How can I make them performant? You know, you know, you know, how do I abstract them? How can I migrate them?

All of those things, you know, have the new embodiments, you know, be motion for agents, right? It's like every one of these will have the new embodiment here for them. But then I also need to think about how can humans be setting the policies, describing the constitutions, you know, getting the, you know, dashboards, etc. And I think that's sort of the way the interesting thing is because you need to be both of those, right?

Not just one of those, but they're fundamental. You know, also the hard-dying stuff like we did for virtual machines was associated abstracting the hardware in a performant way for applications or operating systems here. It's, you know, abstracting the hardware and the operations for agents. It's a fundamental thing they were serving.

It was great discussing this. Thank you so much for coming in and we'll do it again. Well, two of my favorite people anytime. All right, that was awesome.

Thank you. Thanks for listening to this episode of the ACCD podcast. If you like this episode, be sure to like, comment, subscribe, leave us a rating, we'll review, and share with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify.

Follow us on X and A16Z and subscribe to our substack at a16z.substack.com. Thanks again for listening and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal business, tax, or investment advice, or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund.

Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com forward slash disclosures.