I'm back with Ryan or Pope who is the CEO of Maddox, which is a new AI chip company. Last time we were talking about what happens inside a data center. Now I understand what happens inside an AI chip. How does this chip actually work? For this closure, by the way, I am an usual investor in Maddox.
Hopefully, you have designed a good chip. Okay. Also, if you're listening to this on an audio platform, it's much more preferable to watch this Blackboard lecture on platform. We can see what's happening. So, to show you how to do our Spotify.
So, I'll start with the very smallest fundamental units of chip design and we'll build up into what an overall actual production chip, what are they components of that? At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like and or not. Then these are connected together by wires that have to be laid out physically as metal traces on a chip. The main function that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiplied cumulative just like of pairs of numbers. So, we're going to demonstrate what that calculation looks like by hand, and then infer what a circuit would look like for that.
It'll turn out to be easiest if I do multiplication, a cumulative something like a four-bit number with another four-bit number. Then the actual clearest primitive is actually multiply accumulate. So, there's a multiply these two terms, and then we're going to add in, so product of these two terms, and then we're going to add in an eight-bit number. Can I ask a clarifying question? Why is this the natural primitive for whatever computation happens at a computer?
Yeah. So, there's a few reasons for this. It's a little bit more efficient, but the reason it's natural for AI chips is that if you look at what's happening during a matrix multiply, what is matrix multiply in very short, it is there's a for loop over i and over j and over k of output i, k, plus equals to input i, j times other input j, k. So, multiply accumulate, happens at every single step of a matrix multiply.
It makes sense. Then the other observation is that the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe a specific to AI chips, but you're multiplying the precision numbers, but then when you accumulate errors accumulate quickly, and so you need more precision here. So, this is why we try to do a four-bit multiplication and an eight-bit addition. Let me make sure I understood that.
There's two ways to understand that. One is that the value will be larger than the inputs, and the other is that if it was a floating point number, it would be maybe that part is less intuitive to me, but it's maybe the same principle. It is really the same principle. I guess the separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers, so you've got a lot of roundings of accumulating, whereas in this case, there's only one multiplication in that chain, so there's not a lot of rounding errors of accumulating in the multiplication. Why are you summing up a whole bunch of numbers of two numbers?
Just, I mean, this summation happens, it's repeated. Ah, any errors accumulate, I see. Yeah, me so. So, how would we perform this calculation by hand? I mean, as a human, we would probably separate it as a two, but we can sort of do it all in one using long multiplication.
So, the multiplication term first, we're going to multiply this number, this four-bit number here by every single bit position in the other four-bit number. So, we write that out. First, one zero zero one, multiplied by this bit position. That is the number itself. Then, shifted across by one, we're multiplying by zero.
That gives us an all-0s number. Shifted across even one more to multiply by this one, we get one zero zero one. And then finally, for this last bit position, we get an all-0s number again. So, this sort of gives us a bunch of terms that we're going to have to add for the, for the multiplication. And then, while we're doing that summation of this, we might as well add in the actual accumulator term as well.
So, we just copy that directly across. So, this is the sum, it's a five-way sum that we're going to want to commute to compute. So, firstly, what logic gates did it take us to even get to this intermediate step? We needed to produce all 16 of these partial products. How do I produce one of these partial products?
So, let's take this, this number one, for example here. It is one. So, how do we produce this number by multiplying this number by this one over here? We can actually produce that by an end gate. This number is one, if both this bit is one and this bit is one.
If either of them is zero, then the multiplication of one times one of zero times anything is zero. So, to produce all of this stuff, we ended up consuming 16 end gates. Or in the general case, if I were doing a, like a P bit, multiply times a Q, it's multiply, then this will be like P times Q, many ends. Finally, I sum them. Actually, most of the work is going to happen in the summing.
So, let me describe the other logic gate that we use here. And it's almost the simplest logic gate that exists on a chip. It's almost the smallest. At the other extreme, typically the very largest logic gate that you'll use is something called a full adder. And what this does is, coming from software, you might think that a full adder is 32 bit numbers together.
In this case, it just adds three single bit numbers together. So, you can think of it as adding zero, one, and one together. Now, when I add these together, the result can be zero, one, two, or three. So, I can express that in binary using just two bits. So, it has inputted has three bits and as output it has two bits, which in this case, I wanted like the number two in binary is one zero.
So, this is also known as a three to two compressor, because it takes three bits of input and produces two bits of output. The two inputs are an X and a Y value, and then some carry that came in from like, sorry, the three inputs are all bits that are in the same sort of bit position, like three bits that are in a column here. Yeah. So, I have sort of drawn them vertically here and horizontally here to kind of match this vertical versus horizontal layout here, which is expressing that things that are in the same column are in the same like bit position, whereas things that are in the JSON columns, like this is a carryout where this was the sum.
So, if the input and the full adder are, we'll say like, 101, then the output will still be 10, it was 111, it was 000, it was like 010, it was like 010, it still be 01. So, yeah, it's just counting, essentially, the number of things I'm expressing that in binary. So, this circuit actually can sort of capture what we as humans naturally do when we're doing something along the column. So, I can show that sort of, I'll show sort of one iteration of using the full adder to sum. The way I sum here is going to be a little bit unnatural for humans.
We would sort of sum along the column and then remember the carry. But instead of remembering the carry, we'll actually just explicitly write it out. So, in this, we proceed from the rightmost column towards the left, on the rightmost column, we sum the one and the one, and that produces like a zero here and a carry of one. So, we sort of used a, this full adder circuit on this pair of bits and produce a pair of bits as output. Now, we can do the same thing with this column.
We've got a column of one, two, three, four numbers, and so maybe we'll like take the first three of them, run a full adder on them and that gives us a zero and a zero as output. So, like some of these is zero, zero. So, that's the full, full, full adder, apply to all of these bits. As I've used up bits, I'm going to sort of just across them out to indicate that I've handled them. Let's just keep going a little bit more.
So, we'll go here. I take these three numbers, I add them, that gives me a one and a zero. I've done with these three numbers. And now I take one, two, and I can even take these three numbers, for example, right now, and add them and that gives me a one and a zero and I've dealt with these numbers. So, I can sort of, like the way I should view this is that I have this whole grid of numbers that need to be added.
I'm going to just keep applying full adders to all the bits that are here, constantly removing three numbers from a column and then writing out two numbers as output. Keep going with this over and over and over again until I eventually get like some, just one single number coming out here, something like that. This is probably the wrong sum. So, this approach that I've described here, this is called a data multiplier. And this is sort of like the standard for how you do area of fish multiplies using full adders.
Let's try and quantify the circuit size of this. So, we've got a sense of like how big things are so we can compare to them later. How many full adders do I use? I started with how many numbers? I have the 16 partial products, which is a product of all of these terms with all of these terms plus the eight terms that I'm adding here.
So, I started off with 24 bits and then I produced eight bits on the output eventually. And in every step, I was sort of crossing off three numbers and writing two numbers out as an result. And so, every single use of a full adder eliminates one of the bits here. And so, how many full adders it must be to 24 minus the eight. So, there were 16 full adders in the circuit.
In general, this is during the general case as well. There will be p times q many full adders in the circuit. I'm going to show you an interesting logic about that. So, the input bits 24 is p times q plus p plus q. That's right.
And the output bits is just p plus q. And so, p times q plus p plus q minus p plus q equals p times q. So, I think this explains sort of, or at least hints at the second reason why we chose to do a multiplier accumulate. So, this is the first reason being that's actually what shows up in matrix multiplication. But second reason being it gave us this very slick p times q, very simple algebra.
So, we've sort of described like this whole procedure, every single atomic step that I took here becomes a logic gate and then sort of the wires connected together. Like, when I had these three inputs that I saw and introduced these two outputs, like, if I think of mapping this to a physical device, it would be a wire that runs sort of connecting all three of these things together into a logic gate that purchased this output. Okay. So, this is the main primitive at different bits widths that is inside an AI chip. So, we're going to build up from here to how would you use that to run all the other operations you want?
This might be the wrong time to ask this question. But, whenever in video reports that this chip can do X many Fp4 or half as many Fp8, it seems to imply that those circuits are fungible, that there's not as dedicated like Fp4 versus Fp8. By the way, you're mapping it out here, it seems like you would need, if it has to be mapped out in the logic, you would need a dedicated Fp4, multiply, accumulate, and a dedicated Fp8 accumulate. Basically, can you can you can you find them? Has drawn that they're actually not particularly fungible.
This is actually one of the main choices you have to make when designing a chip, which is how much of Fp4, how much of Fp8 do I have? And then sometimes I'll make a consideration from the point of view of like, what do I think is the customer requirement? Another way to take an angle on that is to say, what is the power budget for equalize the power budget between Fp4 and Fp8? But, so then when they report those numbers, and they just happen to be the case of like, it has two X many Fp4 as Fp8. They just happen to choose like give equivalent diareas to all the floating points and as a result that end up being, like, why is the ratio exactly?
Yeah, exactly. Yeah, exactly. Yeah, so part of it is, I mean, surely that one would be exactly equivalent to diarea. Because there's a data movement reason actually, and we'll maybe come back to this when we sort of look through how it goes into and out of memories. There's something really nice just from a software level of the fact that I can pack two four-bit numbers into the same storage as an 8-bit number.
And so when I store that to a memory or something like that, it's the sizing of the buses that I wire within the chip. It actually makes that work out really, really nicely. Actually, I'm going to think of it. It's not just two X, it would be the amount of area it takes that sounds like as quadratic. It's quadratic, in fact.
Yeah, with the bit length. So that's why a smaller precision is like even more favorable than the United States. This is a really big reason. In fact, Nvidia made a change historically up until B100, or B200. Every time you have the bit precision, you double the fault count.
That ratio is exactly like for the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. It should be like an even bigger speed up than you might otherwise think. Nvidia's products have started acknowledging that in B300 and beyond where the FB4 is three times faster than the FB8, though it should be 4X. Yeah, what I've shown here is the simplest case of integer multiply. When you're dealing with fighting point as you do in FB4 and FB8, there's this sort of other term which is the expiner that just complicates this.
So what can we see already from this? I think the big observation you've made is that there's this quadratic scaling with bit width, which is very effective and is the single reason why low precision arithmetic is so well for neural nets. But the other thing we're going to do now is we're going to compare the area spent on the multiplication itself with all of the circuitry that is around it. So we'll walk back in time a little bit and see how did GPUs prior to Tensor Core's work, which is the same way as the way that CPUs worked, in fact. So which is like, where do we stick this multiplied cumulate?
So generically, I'll describe a CUDA Core or a CPU. You'll have some register file, which stores some number of entries, maybe it's 8 entries of, I guess, 42-bit numbers or something like that, which is numbers. So this is the inside the CUDA Core. I'll have some register file of some depth and then I will have my multiplier cumulage, so I could multiply and accumulate. So I could.
And what it's going to do, it's going to take three arbitrary registers from this register file, perform the multiplier cumulate, and then write back to the register file. So it's going to make you write to this one, but it was able to read from this one, this one and another random one. So it'll take three inputs, like this. So this is the core data path of many processors. Most processors look like this.
You've got some set of registers and then you've got some set of logic in it, so ALUs. We want to analyze the cost of the data movement from the register file to the ALU and back. So ultimately, there's going to be some circuit that says, well, I don't always have to select this guy. I might select any of the registers in at any point in time. And so a first question is, how can I build a circuit?
The circuit I'm going to look for is a MUX. So in this case, it's going to have eight inputs, one from each entry of the register file. And it's going to have one output, which is actually producing myself. And then what is the cost of this thing? It's all we have to build it out of is AND and OR.
And so how do we build it? We do the dumbest thing possible. We form a mask saying, OK, when we want to read the third entry, we're going to end every single entry with either one or zero based on whether that's what we want to read, and then we're going to OR all the way together. OK, just to make sure I understand the basics. What the MUX is doing is selecting.
Just selecting, just selecting an input. Yeah, so like invisible to software is like you say, I want input number three. That means there's a MUX. And so what is the cost of this MUX? So an input MUX operating on P bits.
Well, I'm going to. So I have N rows as this eight rows and I've got like each row is P bits wide. Well, I have to and every single bit. So I get N times P many AND gates. Every single input, I have to say, I'm going to like mask it out or not.
And then I'm going to order them all together. And so there's going to be like N minus one times P many OR gates, which is saying, I've got all of these different things, all of them are zeros, but I need to sort of collapse them down into like from my eight options down into one option. And so every step I need to or like one row into an existing row. Yeah. It's actually kind of funny that you would sort of, you don't think of the level of hardware.
You sort of just think like, oh, I'll just select element three and something as simple as that is. I sort of like in and of itself, quite complicated circuit. Yeah, I mean, this is the first step of all of the hidden data movement costs. Yeah. Yeah.
And so like the thing like we're just going to like compare like I have to pay this cost. And I've got one max here. And then in fact, I have two more copies of that for each of the three inputs to my multibly operation. And so I have this cost, which is like three times n times p and gates over here, compared to this p times q, like sort of Gates in the actual circuit that is doing the thing I care about. And if you plug an actual numbers like this n being H, like I get like 24 times p gates over just in the data movement compared to like if q is four, like four times p gates just in the other multibly manner.
And so where's the three coming from? Three different inputs here. Go ahead. So the case, like really just what I'm hinting at here is that like all of this work, which scales like as the size of the register file and this is a very small register file, all of this work, just moving the data from the register file to the logic unit is many, many times more expensive than the logic unit. In the most recent cluster, Max report, semi-nalsis ranked almost 100 different GPU clouds.
Crusoe was one of only five that made the gold tier. Semi-nals is found that gold tier providers like Crusoe had a total cost of ownership, those five to 50% lower than silver tier ones, even when they had identical GPU pricing. This makes sense because total cost of ownership is downstream of a bunch of different things that don't necessarily show up in the sticker price, but that Crusoe has optimized things like how well you detect faults and how quickly you replace failed notes. For example, Crusoe was one of the first clouds to adopt any sentinel in media's own GPU monitoring and self-healing software for enhanced GPU, uptime, utilization, and reliability. This sets Crusoe, makes use of everything that NVIDIA has learned about why chips fail across all their different cleats and deployments so that Crusoe can catch faults earlier in the process.
And once they identify failure, Crusoe can swap into healthy mode in less than 10 minutes. Because they're not very metal, Crusoe doesn't have to spend time installing an operating system or configuring drivers. They can just spin up a new VM on an already running and pre-qualified host. If you want to learn more about this or the other reasons that Crusoe made gold tier, go to crucoe.ai slash dorkash. Maybe helpful to see what a mux looks like, maybe like a two bit or four bit mux.
Yeah, great. So we'll take some inputs, we'll have maybe like, we'll just do a two way mix. So we've got two different numbers. We've got these two inputs and then we have a, so these are the inputs, the main selector between and then we have a selector, which says, which can either be like, I want this one or, or it could be, I want the other one. So this is a one-hot encoding.
So this is what we all start with. And then the output we want to produce, like, let's focus on this case. So, so this is the actual input we got. Yeah. We just want to produce this guy as they're not.
And so, like, so very laboriously, what we do is we end this bit with all of these. And so that produces like, ending this bit with this row. And likewise, we end this bit with this row that produces all zeros. So this was the, there's four hands here, there's four hands here. And then finally, we just pour these two together.
And this goes to one, we all these two together. This goes to one, we all these two together. It gives us zero, we all these two together and gives us one. And so this is the four ores. So, like, this actually ends up looking a little bit like addition, in fact.
Like, we did exactly the same, same set of hands here. So we've added all of these things together, but then instead of collapsing it by using these full-adder circles, we just get a very simple collapsing with or gates. And that doesn't look like n times p. So, yeah, so this was with n equals two inputs. Right.
In the general case, we will have n, like, this is n rows. And then we'll have p bits per row. So that gives us the n times p many end case. So this is kind of described here, almost all of the cost, like seven, eight. So the cost is in the reading and writing the register file.
And only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve. This essentially was the state of play prior to the vault of generation of embedded GPUs. This is what this kind of thing is what was inside the CUDA cores. And this sort of problem statement is what motivated introduction of Tensor cores, which are more generically called Sittelika race.
So if we think about how we're going to solve this problem, like we were spending almost all of our circuit area on something that we just really don't care about and is hidden to a software programmer. And the thing that we actually care about is not much of the area. Well, make this one bigger somehow while keeping this at the same size. That's the goal. So the evolution was like we had baked this much into hardware in the stage.
Like this single line is a multiplied cumulate and this was the single thing was baked into hardware. The idea of a Sittelika ray is to sort of go two levels of loop up and bake this entire loop out here into hardware. And so the idea being that if we have a much bigger granularity fixed function piece of logic, maybe the taxes we pay are putting out a much more. It's not that you're suggesting that if you go up one step in the in the matrix multiply loop that there's some you can tilt the balance more towards compute than communication. That's right.
So there's two effects that we're going to take advantage of here. One is just that we can do more stuff before every trip through a register file. And then the other thing we're going to take advantage of is in fact, in some of this loop we can take advantage of, for example, some things staying fixed. So let's sort of visually we're going to look at this matrix multiplication. So this portion of the loop corresponds to a matrix vector multiplication.
In fact, so we'll take a matrix and multiply it by a vector. So how do we do this? We take every column gets multiplied by the vector and then some. So we're going to sum sort of long columns. So this is zero and three gets multiplied by the three and seven and gets summed.
And then the one and two gets multiplied by the three and seven and gets summed. So there is a multiplier cumulate associated with every single one of these entries in the matrix. So we'll just draw out these four multiplier cumulates. I just want to write it by this before it multi-accumulates. So if each entry in the column that corresponds to the output vector is a dot product and in this case will be like two multiplications and then the addition of this two multiplications that you're accumulating.
Yeah, so the addition so really there's only one addition product but like we like to start with the initialization of zero. Yeah. Yeah. So what we're going to aim for is to have. So we've got to we want to have quadratically more compute.
We do we have got sort of x times y as much compute as we had before. But we're going to want to somehow aim for having only x times as much like communication. And this is sort of the intention so that we get this advantage term going as y. So we've laid down the multiplications bringing in like we're going to want to bring in a vector of size two. And so that sort of already is in line with our columns target.
That's fine. However, we need to somehow manage the communication of this matrix, which which exceeds our budget of x. And so the idea is that in the in an AI context, this matrix is actually going to stay fixed for a long period of time. And so instead of like bringing it in from the outside, so we've got some register files sitting over here, we we we don't want to have like the amount of stuff coming out of this register file. This is the term that we want to go sort of as as x in some sense.
We don't want to bring this full matrix in from the register file every every cycle because we don't have enough that would cost too much in terms of wiring from the register file. And so instead we're going to store our key trick is that this matrix can be stored locally to the satellite array. And so where we will store these numbers zero, one, two, and three in just like a gauge called a register, like physically stores these numbers and we're going to reuse these numbers over and over again for a large number of different matrix. And so the optimization here is that like the nature of matrix multiplication is you can store this like square quadratic thing directly where the logic is happening.
And which is like higher dimension than the or has an extra dimension compared to the inputs which keeps swapping in and out. That's right. I mean, this is the nature of what a major multiplication is is that you do a lot of multiplication to get one value out to like a dot product is a result of a lot of applications. And so that optimization means that you're just like, you can stuff a lot of like multiplication in before you get some value out of it. That's right.
Yeah. So like just complete the picture here of concretely how that looks. I swapped the three and the two here, three and two. So just like this is zero and three is going to multiply by the three and seven. And so we're going to form a dot product sort of along columns here.
So somehow we're going to feed a three and a seven in here. These participate in sort of this feeds into this multiplication and also feeds into this multiplication likewise, the three feeds into here and also into here. And then we're going to sum sort of along here with like starting at the top of a column, we feed in zeros and then coming out the bottom, we get results coming out. So sort of just to visually see what we've got. There's a dot product that is performed along columns in a matrix and that sort of maps exactly to what is done spatially in the satellite gray here.
And so this is one dot product summed vertically and this is a second dot product also something particularly. And then what is the data that needs to go into and out of the register file? We have X amount of data that's coming out here on the output and we also have sort of this data coming from the input, X amount of data from the input. And so with respect to the input and output vectors, at least, we sort of met our goal of having the X as much data going in and out of the register file. This leaves open the question of like I said that the wait matrices, wait matrices stored locally in the satellite gray, how did it get there in the first place?
Is sort of like at some point you need to boot your ship and populate this data. And so where did that come from? The trick is just we just do it very slowly. So we very slowly trickle feed it in into the satellite array. The sort of the simplest strategy is that we sort of run this daisy chain that sends like feed a number into here and then on the next clock cycle, we'll move down to the next entry of the satellite array.
And so we can do that in every column in parallel and that gives us sort of this is also going to come from here and this is going to be another factor of some of the X units of coming in. You just want to be in the sentence. So like we're sort of like, we know that we're going to be bringing in numbers only rarely into the matrix. And so we just want to come up with any construction at all such that the amount of wiring that actually feeds into sort of crosses this boundary of this satellite array, like this boundary right here, we just want to keep that bounded to X and not go as X, Y. And so a particularly simple strategy is that we sort of bring in a number into the top row of the satellite array.
That's what we can do in one clock cycle. And then for like for why consecutive clock cycles are going to be bringing in the top row every time and then sort of shift all the other way down by one. And that keeps the sort of the wiring that needs to come from this expensive register file only down to a factor of X rather than X. I see. OK, so there's two questions in terms of communication and there's like communication time and then there's communication bandwidth.
Yes. And you're saying, since we're only going to be loading this in once, let's maximize, let's minimize bandwidth. That's right. Because bandwidth equals diarrhea. And let's just like load it in slowly over like smaller lanes because we're going to keep this value in there for a while.
Exactly. Interesting. So it's interesting to me that when we were talking last time about inference across many ships, the big high level thing we're trying to optimize for is increasing the amount of compute per memory bandwidth that is to say per communication. And here also we're trying to increase the amount of like actual multiplies or actual additions relative to transporting information from registers to the logic. So in both cases, we're trying to maximize compute relative communication.
Yeah, this turns out sort of all the way up and down the stack. This is sort of close to the bottom sort of like it's to the gates. There's sort of a version that's maybe even closer to the gates of just like even the precision of number format that you choose to use. We saw that same effect. There's like a square cube law or like squared versus linear term going on both in just purely the precision of this ALU, but then also in terms of the size of this, the matrix.
Yeah. Interesting. So so this unit is sort of the next big unit we had like the multiplication circuit. And then on top of that, we have a pretty large, just like gray I drew it is two by two. But in like, for example, old TPUs, they would describe as 128 by 128 of this circuit shown here.
And this circuit ends up being this is the most efficient known mechanism for circuit for implementing a matrix multiply. So we've talked about sort of, it seems obvious that you try to maximize compute relative to communication. What are non obvious tradeoffs that actually you are, you know, keep you up at night about what should we do? Actually, we do why. And it's not an obvious what the answer is.
Yeah. So I mean, I think most of the decisions in chip design are sizing decisions. And so already in what we've done so far, like, so AI chips all have this, this circuit, you know, they have a systolic array and then somewhere near a rigid file providing inputs and outputs. The two sort of like, even within this scope sizing questions that you have are how big should I make my systolic array and how big should I make the, the register file? The, and then the tradeoff for the size of the systolic array, actually, these two questions are coupled is one way to think about it is to say, I'm going to like have a budget for how much, what percentage of my chip area I want to spend on data movement.
So maybe I just say that I want this to be 10% and the systolic array to be 90%. And then sort of, I can size my register file bigger register files and more flexible. They allow me to run sort of more, I can get more application level performance out, but then they sort of take away from this area spent on the systolic array. I recently ran an essay contest where I asked people to write about what I considered to be some of the biggest open questions about the eye. The submission window closed last week.
So I used cursor to create a couple of different interfaces to help me review the entries. One interface anonymizes submissions and hides unnecessary information. It lets me group responses by question, add notes and record my source. The other interface helps me review entrants who also want to be considered for the researcher role that I'm hiring for. The UI puts the applicants essay right next to the resume and the personal website so that I can see everything at once.
Curcer's harness is really good at helping these models see and improve their U.S. I watched it render the interfaces in the built-in browser, take screenshots, click through sections and keep iterating. And this point, Curcer is where I do most of my work. Whether I'm reading and visualizing a bunch of research papers, recording up an interface to review applications, or making flashcards for my Blackboard lectures, Curcer just makes it very easy for an AI to look at whatever I'm looking at and help me understand it and work with me on it. So whatever you're working on, you should do it in Curcer.
Go to cursor.com slash work hash. Where does the clock cycle of a chip come in? What determines what that is? Yeah. And what is a clock cycle?
So I guess at baseline, it's sort of worth observing the chips. They're incredibly, incredibly powerful. Right? You've got a hundred billion transistors in the chip. A key thing that you need to do whenever you have massive parallelism is you need to synchronize between the different parallel units.
In software, typically you have these very expensive synchronization methods, like a mutex. So one thread will finish what it's doing. It will grab a lock, some sort of memory, and then notify the other thread that it's done. On chips, we take a very different approach and say that every nanosecond or so, all circuitry in the chip will kind of pause for a moment and then synchronize every so it actually synchronizes every single nanosecond or so. And so that is the clock cycle.
The entire chip, typically all in one fell swoop, goes in lockstep to the next operation that happens. And so what this looks like in circuitry is that you will have, it's typically drawn, so the clock is sort of mediated by registers, which are these storage devices that we've drawn elsewhere. And the way to think of it is that I have some storage, which is storing like a bit, which might be zero or one. And then I have some sort of cloud of logic, which maybe is like this historic array or this multiplier or something like that. And then I've got some, and that's going to produce some output.
So my inputs, I've got a bunch of inputs feeding to this cloud of logic. And then eventually later there's going to be some output register that this writes to. There is a global clock signal, which drives all of these registers. And it says, at a certain instant in time when the clock strikes, whatever value happens to be on this wire, that instant, that's what's going to get stored in there. And so the sort of the challenge here is like, I would like to have my clock speed run as fast as possible, because if I can run a two gigahertz, I can get twice as many operations per second, if I run it one gigahertz.
But what that ends up meaning is that I'm very sensitive to the delay from this cloud of logic, because any computation that is going to happen in here needs to sort of finish before the next core cycle. So a major point of sort of optimization on any chip then is to make this delay delay from here as short as possible. Interesting. And is there ever, because the constraint here seems to be that if you add too much logic, then you might risk missing the clock cycle. But if you don't add enough, then you're leaving potential compute on the table.
Is there ever a situation where you're like, you take a probabilistic chance that a computation finishes and or is it just like, no, either it's going to finish back like we're not. Yeah, in standard chip design, you imagine that such that, I mean, there is a probability, but it's like many, many standard deviations, like way standard deviations out such that for all the types of purposes, it is a reliable part. It will always meet the clock. There are some weird exceptions to that. There are clock domain crossings where you go from one clock to the other clock, and then you actually do have to reason about this probability.
But interesting. In the main path, you just like you imagine that such that you'll get there like 25% of the clock cycle in advance. So it's very unlike. And in this, in this, the clock, where the clock synchronize, I guess, where the registers are, this is not something you determine as a chip designer. This is sort of just like an artifact of, hey, I want whatever sequence of logic.
And then the software you use to convert your very log into the thing you test into TSMC that just determines like, Hey, in order to make this work, you got to register here and here to make sure that there's a there's no one step that is like too long such that it makes a whole clock like over the entire chip longer than it has to be. Yeah. So there's actually a huge part of the work of design. It actually is it is in turning them. So it is done in a combination of manually and automatically.
So I mean, like, just like to show the very sort of dumb version of like what you can do here, you can take this logic and split it in half. And so like, actually, instead of just one cloud of logic, I'm going to have two smaller clouds of logic, which do the same thing, but split them up by a register, right, feeding in like this. And this is like, if you split it like in the middle, you can hit twice the clock frequency. That's great. You get twice the performance at the cost of this extra register.
And so at the cost of some more storage. Instead of stepping back, why do we need to synchronize a whole chip? Like, if you imagine playing factory or something, there's no like global clock cycle, it's just shit is done and it's done. There's iron on the plate. You can take it if you want.
Yeah. So taking that analogy, the thing that you need to be mindful of is if I've got two different paths through some logic. So I have to do a computation like F here and then computation G here. And then they're going to come and meet for computation H somewhere here. And so there's going to be manufacturing variants here in some ships.
F will take a little longer, maybe in some ships G will take a little bit longer. And so if I got some signal that's propagating through here and the result from F and G have to sort of meet up at H, what can the thing that can go wrong is that F can get there early. And it meets like the previous value of G or the next value of G. And A is needs to know when to start. Exactly.
Like when when has this next iteration of. And so this explains why different ships made at the same process, know the same like TSMC technology can have different clock cycles. Like two ships made at three nanometer, right? Different clock cycles based on whether they were able to optimize, making sure that like there's no one critical path that is so long that it slows down the whole ships clock cycle. That's right.
This optimization that I showed here, this is just the sort of pipeline register insertion it's called. We've inserted in the middle of the pipeline register here. This is a sort of pure tradeoff between clock speed and area. Yeah. This is the easy case.
There is a harder case too, which is sort of drawn as a pipeline of logic here. But in other cases, you may have some, some calculation which actually feeds back in on itself. So it runs some function F and then writes back to itself like this. So for example, this might be this addition, like you've got some number that you're adding into every clock cycle. And so this could be like a plus where adding in some number every clock cycle.
So this like this little circuit, essentially it's just going to sum all the numbers that are presented on different clock cycles. And the challenge is if this plus takes too long, what can I do? If I like split it in, if I try and put a pipeline register like right in the middle of it, like here in the middle of it, this will end up changing the computation that's done. Instead of forming a running sum of everything that comes here, I will actually have two different running sums. I'll end up having a running sum of the even numbers and running some of the odd numbers.
So, sort of this constraint where I have a loop in my logic, which all chips have somewhere, this is actually the thing that is the hardest thing to address and then sets the clock cycle. I don't understand why it would be a problem. Or I'm not sure what it would mean to have a register there. Like is this sort of atomic operation, right? Yeah.
Well, so plus is not really atomic. Like I think as we just demonstrated, yeah, it took a whole lot of work to do an explanation. And so like you can take the early parts of that work and then stick around it and then do take the like parts of that work. Okay. Yeah.
And I guess it's then up to, so the TSM offers a PDK, which says that I say, here's the primitives of logic that we can grant you in the chip. And it's up to that to determine that no, no primitive is bigger than like the clock cycle. They're hoping a process node targets. But other than that, is there like what further optimizes? He can't just say like, Hey, here's all the primitives from TSMC.
And keep adding registers in between the primitives as much as needed until you get to your desired clock cycle. Yeah. As a logic designer, like the chip architects set the clock cycle. So just for one example, the primitives you get from TSMC are on the order of like handcates or full letters. They depends a lot on voltage and frequent and which library you choose and so on.
But generally they, you can typically have about like 10 or 20 or 30 of these in a inner clock cycle sequentially. So these primitives are very, very fast, like 10 pixels or something like that. And so as a logic designer, I mean, like in principle, if you literally just had like, like register and then and gate kind of in a loop like that, you could get in samely fast box speed, like more than four or five, six giga or something like that. But if you take this, this like really sort of like simple circuit and you look at the area you're spending here, like this is maybe like one, I mean, this is called one gate equivalent in size, so like, you know, the one in area and this thing is like, you know, eight in areas and like that.
And so like this is just, again, almost all of your cost is being this like synchronization or communication costs compared to the actual logic. And so, so this is because we've gone too far, you've made a clock speed really, really fast at the cost of spending almost all of your area on, right, on, on, on pipeline registers. Interesting. So what you're hinting at is a dynamic where you can have really fast clock speed, but you're not getting that much work done. Yeah.
Yeah. Um, and so you can have like a little agency, but, but a little bandwidth or throughput, rather. Yeah. The, it has your throughput in fact, because like the throughput of your chip, you can think of as the product of, um, how much I can get done per clock cycle, which is based on this area efficiency thing times how many clocks I get per second. This is, this is actually so similar to the thing we're just going to think last time about like batch size.
Um, where if you have a low batch size, then you can, anyone user can receive their, uh, next token really fast, but the total number of tokens that are processed and say an hour will be kind of lower than good otherwise be. Yeah. Exactly. You get less parallelism out if you, if you drive your clock speed up really high. Language models are starting to compete against the best human forecasters.
I sat down with two senior J-3 doors, Ron Minsky and Dan Pundegorvo and asked, at some point, this is a, I just do what J-3 does. There's a world that we should take seriously where, you know, we're going to build large language models or some other AI systems that are like strictly smarter than all humans on the planet and more capable at all cognitive tasks, trading in particular feels to me as like kind of AGI complete sort of like, NP complete because at the end of the day, trading involves figuring out what things are worth, which means making predictions about the future. J-3 isn't betting against AI. They just signed a $6 million computer deal, but Ron's view is that the edge keeps moving. I've never been more desperate to hire more engineers and more traders than I am today.
You know, you have the usual thing of like the other hard parts that we don't get know how to automate. Well, that ends up being where the competitive edge lies. You can find these open positions and watch the full interview at Jainstreet.com slash Thor Cash. OK, so I remember talking to an FPGA engineer at Jain Street Clark, who actually helped me prep for the previous interview we did together. And he was explaining why he was FPGAs.
And I imagine that for high frequency trading, throughput is less important than latency and so having very specific control of the clock cycle in a deterministic way is the most important thing. Maybe interesting to talk about how you, why you can't just achieve that with an ASIC where FPGA is the, why you're using FPGA to have deterministic clock cycles for like high frequency trading. Yeah, so I mean, firstly, like, let's consider the sort of business case for an FPGA versus an ASIC. FPGAs and ASICs use largely the same sort of conceptual model, which is that I have a series of gates built from ads or as those like very small primitives connected together with a fixed clock cycle and connections together with wires that are running in a fixed clock cycle.
So anything you can express in an FPGA, you can express in an ASIC too. And it'll be about an order of magnitude cheaper and like better energy efficiency on an ASIC than an FPGA. The trade off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs you $30 million because it requires an entire tape out. So sort of the business use case for an FPGA would be that I want something that has this very deterministic latency and fast run time and high parallelism. But, you know, I'm going to change it very quickly.
I change what I do every month or something like that. And so then I don't want to pay the tape. I got it every time. Now, how does an FPGA actually implement? It sort of like it emulates the ASIC programming model, but in a fixed case of how to work.
And so how does that work actually? So what it has at the base is it's got a it's got the two components we just talked about. It's got these registers as storage devices. And then it's got these these are called lots of tables, which actually provide all of the gates. So and then we're going to see even the sort of the third component, we then have like sort of a swarm of these registers and lots and all of these are available and then they're connected by sort of this big set of sort of muxes.
So in front of every single one of these, we've got something like one of these muxes, which selects one input from everywhere else sort of selecting from all of these two things, you know, we've got a whole bunch of different options feeding into into into into all of these things. So what this allows is essentially a when I program my FPGA, I can say that I'm going to take all these components and I'm going to sort of superimpose on top of this a particular wiring, which like goes through this lot and then feeds into this lot and then goes to this register and then feeds into this lot to something like that. So what I've drawn in orange is like how you like FPGA means field programmable get array. This is the orange is what has been programmed in the field, whereas the white is all of the wires that must exist in FPGA in order to actually to make the device in the first place.
What does it mean to be programming to a field? Like programmed in the field. So like the device is being deployed in a data setting in the field and then you can come and program. That field is like electric field field is like out there in the world. Yeah. And so if I see look at the how the field programming comes out of the first
lookup table and goes in a second one, how is it? Yeah. How? Like where the wires that happen? Yeah.
So I got a little bit like lazy and drawing all these every single device here as a mux sitting in front of it, which can select from all of the like nearby like circuits that are available. And so the actual configuration of the FPGA is like amounts to it is the mux control. So like we in this mux here, we have the data inputs and then we have like the control that selects. And so like there's a little storage device sitting next to every single one of these muxs saying this is where you're going to source your input from. Right. And so programming it consists of like configuring every single one of these muxs.
So that makes sense. What is happening inside the lookup table? Yeah. So the purpose of the lookup table, so it's going to also have a little bit of control feeding telling what to do as well. The purpose of the lookup table is to function, to be able to configurably take the role of an AND gate or gate or any of those different things.
So there's many ways you can consider doing that. The way it is done in sort of traditional FPGAs is to say it will support. So it will be a lookup table will be, it will have four bits of input, one bit of output. How many different functions are there for four bits to one bit? There are 16 different functions.
And so you can actually just tabulate this as like 16 different, like 16 different numbers. You've got a table of 01, 01, 01, 16 countries. And so what it does is this table is stored in this blue configuration bit. And then it views these four bits as binary, looks up the relevant row of the table and emits that bit. So there's a truth table view of lookup tables essentially.
OK, so the lookup table, if you think about an AND gate or gate or an OR gate, XOR gate, these are all like take as input. Those are like two input functions. Yeah. So sometimes we have like a three input function would be a three way XOR or a four way XOR. And in this case, how many, it just depends how big it is.
But typical size for lots is four input, which is sort of just a sweet spot between, there's another computer communication trade off like here. Like if it has two inputs, then you need to use more lots. Yeah. But basically the lookup table is like a truth table. It's a truth table.
Yeah. And with a truth table, you can program in any gate you want. That's right. And instead of looking at table, just think like a programmable gate. That's right.
And so I mean, one of the things you can do here is you can see why the rule of thumb that an FPGA is like an order of magnitude and right-spend, and then as it comes from, is to count how many gates would be inside this lookup table. Yeah. So we can view this lookup table essentially as one of these muxes. And so it is a mux with has to select between 16 different values. And so it is a mux with sort of n equals 16 options, a equals 1 bits.
And so what we saw way earlier is that this circuit costs like n times p many gates. And so it's like, so it costs like n times p equals 16 and gates and also 16 ores. They started being the mux. Yeah, exactly. The mux is the core.
The muxes that goes into the lookup table. So the lookup table itself, you can think of as being actually a big mux that like selects from all 16 rows down to one up. Yeah. OK. So that is the lookup table.
But the way you've drawn it here, there's like a mux and then a lookup table. It's muxes all the way down. So I mean, there's a second mux that is inside here. This mux is this. It's got it.
And then the other mux is just saying where it came from in this sort of mess of case. Right. And then the second muxes, OK, now you have one value. But that value is still a four-bit value. Yeah.
So I've selected four bits from the super right. And then I use those four bits to select which entry in the lookup table. Yeah. I'm going to use. Right.
OK. And it's just like, suppose in the first mux, there's like eight nearby. You're pulling from eight nearby registers as input. And so that's like a total of 32 bits going in. And then out of that, four bits come out.
Those four bits go into the second muxes, which is inside the lookup table. So actually, I would say in this case, these registers are single-bit registers. So if there are eight nearby registers on the lookup tables, then I have eight bits total coming in nearby, I select from eight down to four different values. So there's actually like four different muxes, one associated with each of these input, like little mux associated with each of these input bits. Each of them is selecting one out of eight.
And where are those incoming from? Nearby registers and other lots. And each register is one bit. Yes. Yeah.
And so I guess you, AMD or whoever makes these FUJs still has to be opinionated about what the register is connected with registers. And then you can program in the actual gates, but they add a wire in the connection, like the communication topology, right? Yeah. So there's the sort of like, you get flexibility in a local green thing. There's a sort of nearby neighborhood where you can select from.
But then more grossly, like more coarsely, longer distance connections they form in opinion. Right. Yeah. And the reason it's 10x slower is why. So if you look at the cost of building this lookup table, it's like 32 gates.
And then it can give me the equivalent of like, what's one interesting thing I can do here? I can do a four-way hand gate. And so that's like, I'm using 32 gates of lookup table to sort of implement like a four-way and means like, what is a four-way and I would do like, and, and, and, and, and, and. So to input like, this is a, a sucker that I could implement in an ACK directly using these three and gates, but using a lot, I can also implement it, but it's going to take like these 32 gates instead of. Right.
And so the overhead is really coming from the, like, the fact that the lookup table, the mux and the lookup table is, there's a more concise way to describe a truth table than listing out every single possible combination of inputs, which is just to like write out the gate. Yeah. Like place down the poly-silicon and then the limes. Yeah. Interesting.
One important point you made to me is that the reason they prefer APGAs to CPUs is because they get deterministic clock cycles. They know what a pack will come in and go out. Why is it not a guarantee in CPUs? So you can actually design a CPU that has deterministic latency as well. And in fact, like the, the processes that are inside a lot of AI ships actually also have deterministic latency to, because I advertise this TPUs have that in the core as well.
The challenge is getting sort of deterministic latency and high speed at the same time. And so where does the non-deterministic latency come from? Non-deterministic latency comes from specific design choices in a CPU. It's actually possible to remove those design choices and make a CPU that has deterministic latency. Those are not very attractive in the market.
And so people don't make those CPUs anymore. But actually in some sense, deterministic latency is maybe a sort of a simpler designing starting point. And then, like some chip designers have added things into it to be non-deterministic. To take a concrete example of that, the probably the most important example is on a CPU just like the CPU cache itself. So in a CPU, you have the CPU, this is the CPU die itself.
And then there's a memory off on the side. This is the DDR memory off on the side. And then you have a cache system here inside it is the cache that sort of remembers recent access to DR and stores them. And so when I'm running through my CPU instructions, every time I have an instruction that access memory, it first checks and cache was the data starting cache. And then if not, it goes, it fetches out DDR.
This is a huge optimization. The cache is like two orders of magnitude faster than DDR. If you never use the cache, basically all programs would run 100 times slower. So the presence of the cache is absolutely necessary for a CPU to run at reasonable speed. But whether or not you get a cache hit is dependent on the ambient environment of the CPU.
Like what other programs are running, what is running recently, what is the random number generated inside the cache system doing? And so that is a big source of non-deterministic in the runtime of a CPU. So this is sort of the memory system for a CPU. The big thing that you can do differently is instead of having the hardware say, I'm going to read memory and then decide the hardware decides whether or not it comes from cache or not. You can actually bake this into software.
So a different design philosophy is to, and you see this in maybe for example, TPUs. The TPU instead has, I mean, I'll draw the same diagram but I'll call it a scratch pad. And so the main difference is, so this would be like a TPU and then HBM, in this case rather than DDR, but it's still an off-chip memory. And instead of like the software saying first access, memory and then the hardware decides, you've got some instructions that go here. This is like one kind of instruction.
And then a totally different kind of instruction that goes to HBM. And so this style is generically known as scratch pad. Instead of cache, the key distinction being that you have like one kind of instruction that says read or write scratch pad and a totally different instruction that says read or write HBM. So it's scratch pad being the cache. Yeah, this thing here is the scratch pad.
So stepping way back, people will say computers have the, I'm going to go to John Moine, a von Neumann architecture where there's this serial processing of information. Maybe just because we've been talking about parallel accelerators, but I just don't, like the FHA is super parallel. The kinds of the TPUs are super parallel. Even CPUs are super parallel if you think about all the cores they have. And so is it actually, in what sense is modern hardware actually the von Neumann architecture?
Is it actually a fair way to describe modern hardware? I think it's a fair way to describe CPUs. Just the amount of parallelism, like I want a CPU, the amount of parallelism you get is about 100 cores times maybe like 16 way vector unit. So 106, about 1000 way parallelism on a CPU. Yeah, one question is, what is the, there is a die that is being used for the CPU?
And if there's fewer threads, it just doesn't matter of like transistor voltages or switching on and off. Is it just that there's literally one control flow, like a small part of the die where voltage is switching on and off? Or like, in what, how you actually occupy the die area of a CPU if there's a, I suppose? If the CPU cores, like what do you, I think is happening there? The cores are just much bigger and more complicated.
So I mean, like, so I guess we should compare like a CPU core which takes up one, one hundred of the die to like, I mean, to a lot, like a lot is just only these 16 gates. So like it's clear why there's so many more lots in an FHA than cores in a CPU. And then sort of maybe the, like, why are there more CUDA cores, for example, than, than CPU cores, I think would be like, why, what's the difference between a CPU and a GPU or something like that? Would be a big difference. Inside the CPU, you have, so one big user, so the top unit, uses of area in a sub-side CPU of the cache.
Very little is actually the ALUs. Like mostly it's like these register files rather than the logic units. And then both of these things have equivalents in a GPU. And so that's not a big difference. But the thing that does not have any equivalent to a GPU is the sort of this branch predictor.
So there is a whole big area in the CPU, which is sort of just a whole bunch of predictors that are saying, when will my next branch be and where's the branch target for that? And so stripping a lot of that out as well as sort of making these register files tighter in a sense is driving a lot of where the GPU gains. And what is the branch predictor to like execute both branches at once or what does it do? So the issue is that when I go to series of instructions, like instructions, instructions, instructions, instructions, instructions, if I have a branch like here, if this instruction is branch, the actual processing step of processing instruction takes a really long amount of time.
It takes like maybe five nanoseconds or something like that. So like the time to actually notice that I've got a branch and then evaluate the Boolean, whether it's true, and then update the program counter to the new target and then read from the instruction memory for that. That could take like actually five nanoseconds to finish. And so in reality, this may finish somewhere down here. I don't want to, but I want to run a clock speed that is much faster than what five nanoseconds allows.
Like five nanoseconds is 200 megabits, clock speed. I would like to run a one or two gigahertz or something like that. And so I need to run other instructions while the branch is being evaluated. And so I really just want to keep running the following instructions that happen after me. But that might have been wrong.
Like if the branch ended up being taken, then I need to know that instead of it, if I do any of these instructions, I actually need to jump to wherever the target is and run these instructions instead. And so the purpose of the branch predictor is like genuinely to predict based on it before you even get to the instruction to be like five cycles earlier to predict there was going to be a branch that's going to happen. So if I think about how the brain works versus what you're describing here, at high level, the differences might be that while you can do structured sparsity in these accelerators and then save yourself some area that you would have otherwise had to dedicate to these gates, in the brain, there's unstructured sparsity, you know, any neuron can connect any other neuron and not in like ways where they'd have to call them aligned or whatever.
Then there's a fact that memory and computer co-located. I guess you could say in a way that memory can be co-located on this guy. This is exactly the co-location in some sense of the main view. That's right. Yeah.
Yeah. So maybe that actually isn't a big difference. And the other, maybe a big difference is that the clock cycle on the brain is much lower than on computers. And partly that's a preserve energy because the faster the clock cycle, the bigger the voltage needs to be in order to identify for the signal to settle and to identify what state of transition is. I don't know if you have other head level things about like how any commentary on what the brain might be doing versus how these chips work.
Yeah. I mean, so let's take the clock speed one first, actually. The clock speed is quite high on a chip because that drives higher throughput. When we compare a GPU running some workload, it's running batch size 1000 or something like that. Whereas the brain is not running batch size 1000.
It's only one of me. And so you could sort of imagine saying, well, take a GPU and instead of running it, a gigahertz running a megahertz or something like that. And that would start to look maybe a little bit more like a sort of equivalent thing that you're talking about in the brain. There is, in the way that silicon works, there are, that does not give you a thousand extra advantage in energy efficiency. So what it ends up looking like is you can, like, you sort of just end up running this circuit once to the stabilization and then it'll sit idle for a long period of time.
It doesn't consume a lot of energy while it's sitting idle because most of the energy is consumed in sort of toggling bits from zero to one and back. So actually, let's talk about the energy consumption of a circuit like this. The way to think of a bit being stored is you've actually deposited some charge in a capacitor somewhere in the chip implicitly. So it becomes charged when it is bit becomes a one and then it becomes discharged when it next goes to a zero. And that cycle of like charging the capacitor and then dumping that charge out to ground, that is where the energy is consumed.
This is called the dynamic or switching power. This is most of the energy consumption of a chip. There's some other energy consumption just coming from the fact that insulated is not perfect insulators, but we'll just describe that. Most of the energy consumption actually comes from just the charging and discharging of like, toggling from zero to one and back to zero. Right.
So if you run a chip much slower and you only clock at once every thousand cycles or something, you will have a thousand times fewer transitions. It'll be about a thousand times less energy consumption, but, but, but not a substantial advantage in energy efficiency. Okay. So you described how a TPU works at a high level. What is the difference at high level between how a GPU and a TPU work?
Yeah. So I mean, I think that's sort of a high level organization principle that is different. And then there's sort of inside the course, what are different? But we'll look sort of outside the, like at the high level. So we'll take a GPU and what does like sort of the top level box structure look like.
If you think of this as the whole chip in each case, the organization of the GPU is mostly a bunch of almost identical units, which are these, these are the SMs. And then they've got an L2 memory in the middle and then a bunch more of these SMs on the bottom. And so there's sort of this fairly regular grid of course. And then like if we look at a TPU in comparison, you end up with much coarser-grained units of logic. And so you end up with something like some large number of maybe, like, maybe just a few matrix units.
These are the big like system like arrays. And then in the middle, you've got some vector unit and then you've got your matrix units at the bottom. So now sort of like matrix units with a vector unit in the middle, sort of this is the whole TPU chip. You can sort of think of scaling this thing down into a really tiny unit with a smaller matrix unit, a smaller vector unit. And that is sort of what an SM is.
So sort of at a very high level point of view, the GPU has a lot of tiny, tiny TPUs sort of tiled across the whole whole chain. So like you're suggesting the 10-star core within a stream SM is now like an MXU. Yeah, it's very, very simple. Yeah. I see. And so if you had more like, more lack of structure having a bunch of tiny TPUs makes a lot of sense.
Whereas if you kind of just have like huge matrix multiplications, you're like, why don't we just, why don't we avoid the cost of having the individual SMs with their own registers and work schedulers and things like that. Why don't we just like make a huge thing and like, amortize those costs across the whole thing. And I mean, I think this turns up in how large you can grow things. We've sort of seen this theme like, especially with the Sysolik array where largest Sysolik array amortizes the register file costs better. This sort of design allows you to have large Sysolik arrays, whereas the sort of GPU design constraints you're having small units or everything.
There is a trade-off however. The, there ends up being because of this sort of coarse-grained separation of things there. You need to move a lot of data from the vector unit to the matrix units. And so like, you need to move a lot of data through through a sort of like two lines of perimeter here. Whereas if you sort of look at the column thing here, you've got vector units everywhere and you need to move data through this line, through this line, through this line, through this line, through this line.
So the amount of data you can move between a vector unit and matrix unit is actually much higher in, in a GPU than in a GPU because, because like, it's like instead of having to like move all the data through these two lines, you're moving all these data through like 16 lines or something of, of wiring instead of a GPU. Right. But also you might have to move across less area, which I mean is also taking like a few seconds. So data ends up moving like, if you can operate entirely within an SM, the data movement is much smaller, but then the moment you want to operate across SM, it becomes sort of more complicated and expensive. So you don't have to comment, but one might expect that a thing that Maddox might try to do is to get the GPU like smaller structure of systemically surrounded by SRAM, but also at the same time, make it so that like the things you need in an SM to support the CUDA architecture, but take a bunch of space, you might discard.
Yeah. We've talked publicly about something which we call a splitable systolic array, which is sort of in some sense, you can think of as like big systolic arrays that can be small systolic arrays as well. Okay. Things are good to close on. Right. Thank you so much. Thanks for sharing.