Still got Swedish Osmophia, Greyhound in my head from the pump up. Nice. Nice. It is funny how all like GPU companies, like I was watching a bunch of Nvidia Kinoats and AMD Kino's D.R.D.
for this and everyone is so like techno, neon lighting. It's like crypto before crypto. Who got the truth? Is it you, is it you, is it you, who got the truth now?
Is it you, is it you, is it you, is it you? Sitting down, stay straight, another story on the way, who got the truth. Welcome to season 10, episode six of acquired. The podcast about great technology companies and the stories and playbooks behind them.
I'm Ben Gilbert and I'm the co-founder and managing director of Seattle based Pioneer Square Labs and our venture fund PSL Ventures. And I'm David Rosenthal and I'm an angel investor based in San Francisco. And we are your hosts. When I was a kid, David, I used to stare into backyard bonfires and wonder if that fire flickering was doing so in a random way or if I knew about every input in the world, all the air, exactly the physical construction of the wood, all the variables in the environment, if it was actually predictable.
And I don't think I knew the term at the time, but modelable. If I could know what the flame could look like, if I knew all those inputs. And we now know, of course, it is indeed predictable, but the data and compute required to actually know that is extremely difficult. But that is what NVIDIA is doing today.
Ben, I love that engine. That's great. I still like, where has been going with this? And this was, as I was watching Jensen, showing the omniverse vision for NVIDIA.
And realizing NVIDIA has really built all the building blocks, the hardware, the software for developers to use that hardware, all the user facing software now and services to simulate everything in our physical world with our unbelievably efficient and powerful GPU architecture. And these building blocks, listeners, aren't just for gamers anymore. They are making it possible to recreate the real world in a digital twin to do things like predict airflow over a wing or simulate cell interaction to quickly discover new drugs without everyone's touching a petri dish or even model and predict how climate change will play out precisely. And there is so much to unpack here, especially in how NVIDIA went from making commodity graphics cards to now owning the whole stack in industries from gaming to enterprise data centers to scientific computing, and now even basically off the shelf self-driving car architecture for manufacturers.
And at the scale that they're operating at, these improvements that they're making are literally unfathomable to the human mind. And just to illustrate, if you are training one single speech recognition machine learning model these days, one, there's one model. The number of math operations, like ads or multiplies to accomplish it, is actually greater than the number of grains of sand on the earth. I know exactly what part of the research you got that from, because I read the same thing and I was like, you've got to be freaking kidding me.
Isn't that nuts? I mean, there's just nothing better in all of the research that you and I both did. I don't think to better illustrate just the unbelievable scale of data and compute required to accomplish the stuff that they're accomplishing and how unfathomably small all of this is the fact that that happens on one graphics card. Yeah, so great.
All right listeners, this is a great time to think one of our favorite companies here at acquired, Century. That's S-E-N-T-R-Y like someone's standing guard. Yes, Century helps developers debug errors and latency issues, pretty much any software problem and fix them before users get mad. As their homepage puts it, they are considered not bad by over 4 million software developers.
Today we are talking about the way Century works with another company in the acquired universe, Enthropic. Enthropic used to have some older infrastructure monitoring in place, but at their massive scale and complexity, they instead adopted Century to help them fix issues faster. Yep, crashes can be a massive problem in AI. If you're running a huge compute job like training a model in OneNote fails, it can affect hundreds or thousands of servers.
Century helped them detect bad hardware so they could quickly reject it before causing a cascading problem. Century also enabled them to debug massive issues in hours instead of days so they could get back to their training runs. And today, Enthropic relies on Century to track exceptions, assign errors, and analyze failures in real time across all of the primary languages used by Enthropic's research teams, including Python, Rust, and C++. According to the Enthropic team, Century gives our developers one place that will have all the information they need to debug an issue.
And speaking of AI, Century now has an AI debugger called CIR. CIR is an AI agent that taps into all the issue contexts from Century and your codebase to not just guess but root cause gnarly issues and propose merge ready fixes specific to your application. We're pumped to be working with Century. They have an incredible customer list, including not only Enthropic but cursor, Vercell, Linear, and more.
If you want to fix your broken code fast, like over 150,000 other organizations that use Century from IndiHobbias to some of the biggest companies in the world, you can check out sentry.io slash acquired. That's S-E-N-T-R-Y dot I-O slash acquired and just tell them that Ben and David sent you. Yes, and they are offering two months free to all acquired listeners. Yes, thank you, S-E-T-R-Y.
And after you finish this episode, come join the Slack, acquired.fm slash Slack and talk about it with us. All right, David, without further ado, take us in. And as always listeners, this is not investment advice. David and I may hold positions in securities discussed.
And please do your own research. That's good. I was going to make sure that you said that this time because we're going to talk a lot about investing in investors in NVIDIA stock over the years. It has been a wild, wild journey.
So last, we left our plucky heroes, Jensen Huang, and NVIDIA, in the end of our NVIDIA, the GPU company years, and then roughly, you know, 2005, 2006, they had cheated death, not once, but twice. The first time in the super overcrowded graphics card market when they were first getting started. And then once they sort of, you know, jumped out of that frying pan into the fire of Intel, now getting for them, coming to commoditize them like all the other, you know, PCI chips that plugged into the Intel motherboard back in the day. And they bravely find them off.
They team up with Microsoft. They make the GPU programmable. This is amazing to come out with programmable shaders with the GeForce 3. They power the Xbox.
They create the CG programming language with Microsoft. And so here we are. It's now 2004, 2005. And it's a pretty impressive company.
Public company stock is high flying after the tech bubble crash. They've conquered the graphics card market. Of course, there's ATI out there as well, which will come up again. But there's three pretty important things that I think the company built in the first 10 years.
So one, we talked about this a lot of the last time, these six month ship cycles for their chips. We talked about that, but we didn't actually say the rate at which they ship these things. I actually wrote down like a little less. So in the fall of 1999, they shipped the first GeForce card, the GeForce 256.
In the spring of 2000, GeForce 2, in the fall of 2000, GeForce 2 Ultra, spring of 2001, GeForce 3, that's the big one with the programmable shaders. Then six months later, the GeForce 3, Ti 500. I mean, the normal cycle, I think we said was two years, maybe 18 months from most other competitors who just got largely left in the dust. Well, I was just thinking, you know, the competitors are gone at this point, but I'm thinking about Intel.
How often did Intel ship new products, let alone fundamentally new architecture? You know, there was the 286 and the 386 and the Pentium and it got to Pentium. Dude, I feel like the Intel product cycle is approximately the same as a new body style of cars. Yes, exactly.
Every five, six years, there seems to be a meaningful new architecture change. And Intel is the driver of Moore's law, right? Like these guys ship and bring on new architectures at Warp Speed. And they've continued that through today.
Two, one thing that we missed last time that is super important and becomes a big foundation of everything in video becomes today that we're going to talk about. They wrote their own drivers for their graphics cards. And we wrote a big thank you for this and many other things to a great listener, very kind listening to Jeremy who reached out to us in Slack and pointed us to all a whole bunch of stuff, including the Asianometry YouTube channel. So good.
I probably watched like 25 Asianometry videos this week. So, so good. Huge shout out to them. But all the other graphics cards, companies at the time, and most peripheral companies, they let the further downstream partners write the drivers for what they were doing.
And video is the first one that said, no, no, we want to control this. We want to make sure consumers are using video cards, have a good experience on whatever systems they're on. And that meant A, that they couldn't share quality, but B, they start to build up in the company, this like base of really nitty gritty low level software developers in this chip company. Not a lot of other chip companies that have capabilities like this.
No, and what they're doing here is taking on a bigger fixed cost base. I mean, it's very expensive to employ all the people who are writing the drivers for all the different operating systems, all the different OEMs, all the different boards that has to be compatible with. But they viewed it as it's kind of an Apple-esque view of the world. We want the control or as much control as we can get over making sure that people using our products have a great user experience.
So they were sort of willing to take the short term pain of that expense for the long term benefit of that improved user experience with their products. That they're users, high-end gamers that want the best experience, they're going to go out, they're going to spend the time three, four, $500 on an NVIDIA top of the line graphics card. They're going to drop it into the PC that they built. They wanted to work.
I remember messing around with drivers back in the day and things not working. This is super important. So all this is focusing on, of course, they have the third advantage in the company is programmable shaders, which ATI copies as well, but they've innovated, they've done all this. So all of this at this time, it's all in service of the gaming market.
And one seed to plant here, David, when you say the programmable shaders developers, the notion of a NVIDIA developer did not exist until this moment. It was people who wrote software that would run on the operating system, and then from there, maybe it would get that compute load would get offloaded to whatever the graphics card was. But it wasn't like you were developing for the GPU, for the graphics card, with a language and a library that was specific to that card. So for the very first time now, they start to build a real direct relationship with developers so that they can actually start saying, look, if you develop for our specific hardware, they're advantageous for you.
And really, our specific gaming cards, like everything we're talking about, these developers, they're game developers, all of this stuff, it's all in service of the gaming market. So again, they're a public company, they have this great deal with Microsoft, they bring out CG together, they're powering the Xbox. You know, all street loads of them, they go from sub-a-billion-dollar market cap company after the tech crash up to five to six billion dollars, kind of by 2004 to 2005. Stock keeps going on a tear.
By mid 2007, the stock reaches just under $20 billion market cap. You know, this is great. And this is all the stories, like this is pure play gaming. These guys have built such a great advantage in a developer ecosystem in a large and growing market, clearly, which is video games.
Which on its own, that would be a great way of to surf. I mean, I think what's the gaming market today? 180 billion or something? And when we talk to Trip Hawkins, who sort of like helped invent it or no one push now, it was zero then.
And so NVIDIA is sort of like on a wave that's at an amazing inflection point, they can totally just ride this gaming thing and be an important company. It's not running out of steam. I mean, like, how could you not be not satisfied, but like more than satisfied with this as a founder? You're like, yes, I am the leading company in this major market, this huge wave that I don't see ending anytime soon.
You know, 99.9% of founders who are themselves as a class, like, you know, very ambitious, are going to be satisfied with that. But not Jensen. But not Jensen. So while all this is happening, he starts thinking about, well, what's the next chapter?
You know, I mean, this market, I want to keep growing. I want a video to be just a gaming company. So we ended last time with the little, almost a surely apocryphal story of a Stanford researcher, you know, sends the email to Jensen and is like, ah, you know, thanks to you. My son told me to go buy off the shelf, you know, GeForce cards at the local fries, electronics, and I stuffed them into my PC at work.
And you know, I ran my models on this. He's a, I think it was a quantum chemistry researcher supposedly. It was 10 times faster than the supercomputer I was using in the lab. And so thank you.
I can get my life's work done in my lifetime. And Jensen loves that. It comes out at every GTC. So that story, if you're a skeptical listener, my big two questions.
First is a practical one. You know, we just said everything's about gaming here. And here's like a researcher, like a scientific researcher doing, you know, chemistry modeling, using GeForce cards for that. What's he writing this in?
Well, it turns out, programmable shaders, right? Yeah, they were shoe-horning CG, which was built for graphics. They were translating everything that they were doing into graphical terms, even if it was not a graphical problem they were trying to solve. And writing it in CG, this is not for the thing to part so to speak.
Right. So everything is sort of metaphorical. He's a quantum chemistry researcher, and he's basically telling the hardware. Okay.
So imagine this data that I'm giving you is actually a triangle. And imagine that this way that I want to transform the data is actually like applying a little bit of lighting to the triangle. And I want you to output something that you think is the right color pixel. And then I will translate it back into the result that I need for my quantum chemistry.
Like you can see why that's suboptimal. Yeah. So he thinks this is an interesting market. He wants NVIDIA to serve it.
If you really want to do that right, it is a massive undertaking. It was 10 plus years to get to the company to this point. You know, what CG was is like a small sliver of the stack of what you would need to build for developers to use GPUs in a general purpose way, like we're talking about. You know, it's kind of like they worked with Microsoft to make CG.
It's like the difference between working on CG and like Microsoft building the whole dotnet framework for developing on Windows, you know, or today, even better Apple, right? Like everything Apple gives to iOS and Mac developers to develop on Mac. Right. Yeah.
The analogy is not perfect, but it's like instead of just Apple saying, okay, objective C is the way that you write code for our platforms. Good luck. They're like, okay, well, we need UI frameworks. So how about AppKit and CocoTouch and how about all these other SDKs and frameworks like ARKit and like StoreKit and like HomeKit.
It's basically that you need the whole sort of abstraction stack on top of the programming language to actually make it very accessible to write software for domains and disciplines that you know are going to be really popular using that hardware. Exactly. So when Jensen commits himself and the company to pursuing this, he's biting off a lot. Now we talked about they've been writing their own drivers.
So they have actually a lot of very low level, I mean, low level, like bad, I mean, low level, like infrastructure, like close, very difficult systems oriented programming talent within the company. So that kind of enables them to start here. But like, still, this is the thing. So then the second question, if you're a discerning investor, particularly in a video that you want to ask at this point in time, is like, okay, Jensen, you're committing the company to a big undertaking.
What's the business case for that? Show me the market. I mean, Don Valentine at this point would be sitting there listening to Jensen and being like, show me the market. And not only is it show me the market, but it's how long will the market take to get here?
And it's how long is it going to take us and how many dollars and resources it's going to take us to actually get to something that's useful for that market when it materializes? Because while CUDA development began in 2006, that was not a useful, usable platform for six plus years at NVIDIA. Yeah, this is closer to on the order of the Microsoft development environment or the Apple development environment than what NVIDIA was doing before, which was like, hey, we made some APIs and worked with Microsoft so that you can program for my thing. Right.
I'm going to flash way forward just to illustrate the insane undertaking of this. I searched LinkedIn for people who work at NVIDIA today and have the word CUDA in their title. There are 1,100 employees dedicated specifically to the CUDA platform. I'm surprised it's not 11,000.
Yeah. Okay. So like, where's the market for this? Yes, Ben, you asked the third question, which is, okay, the intersection of what does this take to do this and when is the market going to get there and time and cost?
And on that, but even just put that aside, is there a market for this is the first order question. And the answer to that is probably no at this point in time. And what they're aiming at is scientific computing, right? It's researchers who are in science specific domains who right now need super computers or access to a super computer to run some calculation that they think is going to take.
Weeks or months and wouldn't it be nice if they could do it cheaper or faster? Is that kind of the market they're looking at? Yeah, they're attacking like the craze market, like crazy computers, like that kind of stuff. You know, great company, right?
But like, that's no Nvidia today. Right. And they were dominating the market. Yeah, it's scientific research computing.
It's drug discovery. It's probably a lot of this work they're thinking, oh, maybe we can get into more professional like Hollywood and architecture and other professional graphics domains. Yeah, yeah, yeah, sure. But you know, you sum all that stuff up and like, maybe you get to a couple billion dollar market, maybe like total market and not enough to justify the time and the cost of what you're going to have to build out to go after this to any rational person.
So here we come. Denson and Nvidia, like they are doing this. He is committed. He's doing the cooling 2006, 2007, 2008.
They are pouring a lot of resources into building what will become good that will get to in a second. I already is good at this point in time. And I think Jensen's psychology here is sort of twofold. One is he is enamored with this market.
He loves the idea that they can develop hardware to accelerate specific use cases in computing that he finds sort of fanciful and he likes the idea of making it more possible to do more things for humanity with computers. But the other part of it is certainly a business model realization where he has spent the last, gosh, at this point, 13, 14 years being commoditized in all these different ways. And I think he sees a path here to durable differentiation where he's like, whoa, to own the platform. You know, it's kind of the Apple thing again, to own the platform and to build hardware that's differentiated by not only software, but relationships with developers that use that custom software.
Like then I can build a really sort of like a company that can throw its weight around in the industry. 100%. Jensen, I don't know if he used it at the time because he probably wouldn't have gotten pillied, but maybe he did. I don't think he cared.
He certainly has used it since the way he thought about this was it wasn't just like, if we build it, they will come, which is what was going on. The phrase he uses is, if you don't build it, they can't come. So it's not even like, yeah, I'm pretty sure if we build it, they will come. It's one step removed from that.
It's like, well, if we don't build it, they can't even possibly come. I don't know if they will come, but they can't come if we don't build it. So Wall Street is mostly willing to ignore this in 2006, 2007, 2008. The company's still growing really nicely.
They're this great market cap run leading up to right before a financial crisis. But then, you know, you mentioned last time, I think it's announced in 2006, maybe in closes in 2007, AMD acquires ATI. Yeah. And ATI was a very legit competitor, the only standing legit competitor to NVIDIA through its whole life.
But now AMD acquired it. And they acquired it for what? Six, seven billion dollars, something like that. Something like that.
So it was a lot of money. And then they put a lot of resources. Like they weren't just acquiring this to get some talent. Like they're like, no, no, this is going to be a big back line for us.
We're putting a lot of weight behind this. We haven't done the research into AMD the way we have into NVIDIA, but the AMD Radeon, which used to be the ATI Radeon, that is how you think about AMD as a company is that they make these GPUs mostly for the gaming use case. Yep. Before the acquisition, I think the first PCI built in like an high school beginning college, I think I had a Radeon card in it.
I think I was probably in the minority. I think NVIDIA was bigger, but for whatever reason, I liked ATI at that point in time. So like, they were legit. Well, so here's NVIDIA now focusing on this whole other thing.
And you're still in the gaming market, which like we said, is like massive rising tide. Your competitor now has all these resources and AMD that's fully dedicated to going after it. Mid-2000, NVIDIA, we have saw an earnings. Like this is natural.
They took their eye off the ball. Of course they did. And the stock gets hammered. Because anything that CUDA empowers is not yet a revenue driver and they've totally taken their eye off of gaming.
Yes. So we said the high was around the $20 billion market cap. It drops 80% 8 to 0. This isn't just a financial crisis.
It's almost queen, I think, you know, for me thinking back on the financial crisis now and like people freaking out the Dow, you know, dropping 5% in a day. Like, oh, that's a Thursday these days, you know, is literally the Thursday that we are recording. Yes, for a company stock to drop 80% a technology company stock, even during the financial crisis. They're not just in the penalty box.
They're like getting kicked to the curb. Right. Are they done? The headlines at this point are is NVIDIA's run over.
If you're most CEOs at this point in time, you're probably calling up Goldman or, you know, Alan company or Frank Quattro and your shop in this thing because how are you going to recover? But not Jensen. But not Jensen, obviously. So instead he goes and builds CUDA and continues to build CUDA.
And this is just a context. Like we get excited about a lot of stuff unacquired. But I think CUDA is like one of the greatest business stories of the last 10 years, 20 years, more. I don't know what do you think then.
I mean, I'd say it's one of the boldest bets we've ever covered, but so we're programmable shaders and so was NVIDIA's original attempt to make a more efficient quadrilateral focused graphics. Those were big bets. I think this is a bet on another scale. This is a bet that we don't cover that often on a choir.
Those were big bets relative to the company's size at the time, but this bet is like an iPhone size bet. That's exactly what this is. It's an iPhone size bet. It is a bet the company when you are already a several billion dollar company.
Yes. An attempt to create something that if they are successful and this market materializes, this will be a generational company. Yeah. So what is good?
It is NVIDIA's compute unified device architecture. It is, as we've referred to, you know, thus far throughout the episode, a full and I mean full development framework for doing any kind of computation that you would want on GPUs. Yeah. And in particular, it's interesting because I've heard Jensen reference it as a programming language.
I've heard him reference it as a computing platform. It is all of these things. It's an API. It is an extension of C or C++.
So there's a way that it's sort of a language, but importantly, it's got all these frameworks and libraries that live on top of it. And it enables super high level application development, you know, really high abstraction layer development for hundreds of industries at this point to communicate down to Kuda, which communicates down to the GPU and everything else that they have done at this point. This is what's so brilliant. So right after we released the same day that we released part one, the first NVIDIA episode we did a couple weeks ago, Ben Thompson had this amazing interview with Jensen on Stretakery.
And Jensen in this interview, I think, points what Kuda is and how important it is better than I've seen anywhere else. So this is Jensen speaking to Ben. We've been advancing Kuda in the ecosystem for 15 years and counting. We optimized across the full stack iterating between GPU acceleration libraries, systems and applications continuously, all while expanding the reach of our platform by adding new application domains that we accelerate.
We start with amazing chips, but for each field of science, industry and application, we create a full stack. We have over 150 SDKs that serve industries from gaming and design to life in our sciences, quantum computing, AI, cybersecurity, 5G and robotics. And then he talks about what it took to make this. This is like the point we were trying to like hammer home here.
He says, you have to internalize that this is a brand new programming model and everything that's associated with being a program processor company or a computing platform company had to be created. So we had to create a compiler team. We had to think about SDKs. We had to think about libraries.
We had to reach out to developers and to vandalize our architecture and help people realize the benefits of it. And we even had to help them market this vision so that there would be demand for their software that they write on our platform and on and on and on. It's crazy. It's amazing.
And when he says that it's a whole new programming that he says maybe paradigm or way of programming, it is literally true because most programming languages up to this point and most computing platforms primarily contemplated serial execution of programs. And what Kuda did was it said, you know what, the way that our GPUs work and the way that they're going to work going forward is tons and tons of cores, all executing things at the same time, parallel programming, parallel architecture. Today, there's over 10,000 cores on their most recent consumer graphics card. So insanely or dare I say embarrassingly parallel and Kuda is designed for parallel execution from the very beginning.
That's the catchphrase in the industry of embarrassingly parallel. And it's actually kind of a technical term. I don't know why it's embarrassing. It's basically the notion that this software is so parallelizable, which means that all of the computations that need to be run are independent.
They don't depend on a previous results in order to start executing. It's sort of like it would be embarrassing for you to execute these instructions in order instead of finding a way to do it parallel. It's not that it's parallel that's embarrassing. It's embarrassing if you were to do it the old way on CPUs, seriously.
I think that's the implication. Got it. Got it. This is so obvious that it's embarrassingly parallel.
OK, now it makes sense. Now here's the Kuda cross. We're going to spend a few minutes talking about how brilliant this was. Everything we just described this whole undertaking, like building the pyramids of Egypt or something here.
It is entirely free in video to this day. Now this may be changing. We'll talk about this at the end of the episode has never charted a dollar for Kuda, but I can download it, learn it, use it, you know, blah, blah, blah, all of this work. Stand on the shoulders of everything in videos done, but then what is the but it is close source and proprietary exclusively to Nvidia's hardware.
That's right. You do any of this work. You cannot deploy it on anything, but in video chips. And that's not even just like, Oh, in video putting their like terms of service that you can't deploy this on, you know, in each of their whatever.
Like, literally doesn't work. It's full stack. It's like, if you were to develop an iOS app and then try and deploy it on Windows, like it wouldn't work. It is integrated with the hardware.
So OpenCL is sort of the main competitor at this point. And they do actually let OpenCL applications run on their chips, but nothing in Kuda is available to right elsewhere. It's so great. Okay.
So now you can see this is just like Apple and it's Apple business model. Apple gives away all of this amazing platform ecosystem that they built to developers and then they make money by selling their hardware for very, very healthy. Gross margins, but this is why Jensen is so brilliant because back when they started down this journey in 2006, even before that, when they started and then all through it, there was no iOS. There was no iPhone.
Like it wasn't obvious that this was a great model. In fact, most people thought this was a dumb model that like Apple lost and the Mac was stupid and niche and like Windows and Intel is what won the Open ecosystem. Well, but Windows and Intel did have proprietary development environments and, you know, full stack dev tools. Oh, yeah.
There's a lot of nuance here. It's not like they were like open source per se, but it could run on any hardware. Well, except that it couldn't. It could only run on the Intel IBM Microsoft Alliance world.
It wasn't running on power PCs. It wasn't running on anything Apple made. That's true. It's funny.
In some ways, Nvidia is like Apple. In other ways, they're like the Microsoft Intel IBM Alliance, except fully integrated with each other instead of being three separate companies. Yeah, that's maybe a good way to put it. It is sort of somewhere in between.
There is nuance here. Remember when Clay Christensen was bashing on Apple in the early days of the iPhone being like, yeah, oh, yeah, open is going to win. Android is going to win. Apple is doomed.
So you know, clothes never works. You got to be modular. You can't be integrated. And like Clay was amazing.
It was one of the greatest strategic. But I think that's just representative to me of like everybody thought that like the Apple model sucked. Yeah, I mean, it sucks unless you're at scale. And at the time there was very little to believe that Nvidia was going to have the scale required to justify this investment or that there was a market to let them achieve the scale to justify this.
That's the thing. Even if you were to say, OK, Jensen, I believe you and I agree with you that this is a good model. If you can pull it off at the time, you could be done down a diner, whoever looking around and maybe done was still looking around because they probably still held the stock. But you're like, well, where's the market that's going to enable the scale you need to run this playbook?
OK, listeners, now is a great time to tell you about a new friend of the show. We are very excited about WorkOS. Yes. WorkOS is the enterprise ready platform used by OpenAI, cursor, perplexity for cellplaid and literally hundreds of other winning companies.
So what are all these companies using WorkOS for? Imagine you're a fast growing startup. You've got product market fit and you're getting inbound interest from big enterprise customers. Very exciting.
But then they send you their security questionnaire. Yep. And it's like 47 pages long with requirements that kind of sound like alphabet soup. Do you support SAML 2.0?
Can you integrate with our octa? Do you have skim provisioning as CIM? What about RBAC? And you're thinking, I have no idea what these acronyms even mean, let alone how to implement them.
So here's the thing. These are not nice to have. These are deal blockers without SSO, without skim, without RBAC, without audit logs. You simply cannot close enterprise deals, period.
But none of these features make your core product better. They don't make your beer taste better to use our favorite analogy here on the choir. So if you're building like a design tool, spending six months building SAML authentication doesn't make your design tool more powerful. So this is where WorkOS comes in.
They've built Stripe for enterprise features. WorkOS turns enterprise authentication requirements into drop in APIs, abstracting away as much unnecessary complexity as possible. So instead of your team spending months reading SAML specs, you can implement enterprise SSO in minutes. WorkOS handles user provisioning, permissions, audit logs, all the checkbox items that enterprise IT requires.
So whether you are a seed stage company trying to land your first enterprise customer or already big and expanding globally, WorkOS is the fastest path to becoming enterprise ready. Just visit WorkOS.com or just message their Slack support. They have real engineers in there who answer questions fast. And when you get in touch, just tell them that it's on you.
All right. So you're going to take us to 2011, 12, or we hop them back in here. If only the world where it's like fiction and we're actually like a truly straight line, it's never a straight line. We will get there.
And that is what saves NVIDIA and makes this whole thing work, but they have some misadventures in between. So a sax getting hammered. It's 2008 and I'm just completely speculating on my own. But they're in the penalty box.
They're committed to continuing to invest in CUDA and making general purpose computing on GPU a thing. I do wonder if they felt like, well, we got to do something to appease shareholders here. You know, we got to show that we're trying to be commercial here. So it's 2008.
What's going on in 2008? You know, in the tech world, it's mobile. So in 2008, they launched the Tegra chip and platform with an NVIDIA. This may not be what saved the company.
This is not what saved the company. This is more a clown car style. I think that's maybe that's too rough on NVIDIA. But what was Tegra?
People might recognize that name. It was a full on system on a chip for smartphones, competing directly with Qualcomm, with Samsung, like it was a processor, like an ARM based CPU, plus all the other stuff you would need for a system on a chip to power Android headsets. This is like a wild departure for leverages, none of NVIDIA's core scale sets, except maybe graphics being part of smartphones. But like, come on, there's ever a use case for integrated graphics.
It's smart phones. Right. Low power, smaller footprints. Yeah.
Totally. Do you know, this is one of my favorite parts about the whole research. Do you know what the first product was that shipped using a Tegra chip? No, it was the Microsoft zoom HD media player.
That just tells you pretty much everything you need to know. It did though, the Tegra system. It is all around sort of to this day empowered the original Tesla Model S touch screen. So like before any of the autopilot autonomous driving stuff, they were the processor powering just the infotainment, the touchscreen infotainment in the Model S.
And I think that actually starts to help NVIDIA get into the automotive market. The Tegra platform still to this day is the main processor of the Nintendo Switch. Oh, they repurposed it for that. Yeah, for that.
And they, I think they still have their NVIDIA shield, proprietary gaming device stuff that I don't know that anybody buys those. Oh, this makes so much sense because they basically have walked away from every console since the PlayStation 3. Yeah. And so it's interesting that they have this thriving gaming division that doesn't power any of the consoles except the Nintendo Switch.
And I always wondered like, why did they take on the Switch business? Because they kind of already had it done. It's not for the graphics cards. It was as somewhere to put the Tegra stuff.
Fascinating. Yeah, quick aside, it's funny how these GPU companies have not been good at transitioning to mobile. There's like a funny naming thing, but do you know what happened to? So there's the ATI Radeon, which became the AMD Radeon desktop series.
They tried to make mobile GPUs. It didn't go great and they ended up spinning that out and selling all that IP to another company. Do you know the company? Oh, I do not.
Was it Apple? It is Qualcomm. And today is Qualcomm's mobile GPU division and Qualcomm's good at mobile and so is a natural home for it. Do you know what that line of mobile GPU processors is called?
No, it is the Arduino ARDNO processors. And do you know why it's called the Arduino or Arduino? No, that sounds super familiar, but no. The letters are rearranged from Radeon.
Ah, that's great. Yeah, that's great. So you're saying Nvidia's mobile graphics efforts didn't quite pan out. No, we didn't talk about this as much in the Sony episode, but my impression of the whole Android value chain ecosystem is that there's no profits to be made anywhere.
And Google keeps it that way on purpose. Ironically, they make a lot of money now on the Play Store. Ah, yeah, the Play Store and ads. Right.
I do think the primary way that they monetize it is not having to pay other people to acquire the search traffic. Right. But I mean, for like partners, like if you are making everything from chips all the way up through hardware in the Android ecosystem, I don't think you're making it like maybe if you were the scale player, but like these things are designed to sell for dirt cheap as in products like there's no margin to be out here. Yeah.
Yeah. Also, before I continue, you just did the sidebar on the M.D. mobile graphics chip. I see your sidebar.
I'm going to raise you one more side. We have to include, you know, because the NCS guys told us about this. So when Nvidia is going after mobile, they buy a mobile based band company called I Sarah British company called I Sarah in 2011. You know where I'm going with this.
Oh, yes. This is so good. You need to come back to later because they're investing in mobile integrity is going to be a big blah, blah, blah. And then a few years later, when they end up pretty much shutting down the whole thing, they shut down what they bought from I Sarah.
They lay everyone off. The I Sarah founders who made a lot of money when Nvidia bought them. They go off and they found a company called Graphcore that we're going to talk about a little bit at the end of the episode is, you know, maybe one of the primary sort of Nvidia. Bear cases, you'd be a bear cases and video killers out there.
They've never raised about 700 million in metric capital and mobile in some ways. It's kind of like Bezos and Jett.com. Yes, if Jett had been successful, I think that's sort of the Graphcore to Nvidia analogy. Yes.
Well, I'm sure he's still out. If anybody's going to be really successful in competing with the video, although I think the market now is probably ironically big enough that large Nvidia can be the way all and there can be plenty of big other companies to. So anyway, OK, back to the story. So Nvidia is bumping along through all of this in the early late 2000s, early 2010s, you know, some years growth is like.
10% maybe it's flat and others like this company is completely on sideways in 2011. They whiff on earnings again, stock goes through another 50% drawdown. It's clear. I was going to say it.
I don't even know if you can say it about Jensen. Like here we are. The company is screwed again. Like everybody else will give it up, but obviously not them.
So what happens? Basically a miracle happens. I don't know if there's any other way that you can describe this except like a miracle. So maybe this is actually not a great strategy case study of Jensen because it required a miracle.
Well, Jensen would say it was intentional that they did know the market timing and that the strategy was right. And the investment was paying off and that they were doing this the whole time. Yeah, sure. Even the Ben Thompson interview, I think he said Ben basically lays out like how did all these implausible things happen exactly the right time and his responses.
Oh, yes, we planned it all. It was so intentional Jensen did not play on Alex net or see it coming because nobody saw Alex net coming. So in 2009, a Princeton computer science professor and also undergrad alum of Princeton, just like yours truly one place named Fei-Fei Lee, their specialty is artificial intelligence and computer science starts working on an image classifying project that she calls ImageNet. Now, the inspiration for this was actually a way old project from like the 80s at Princeton called word net that was like classifying words.
This is classifying image net. Her idea is to create a database of millions of labeled images, like images that they have a correct label applied to them. Like this is a dog or this is a strawberry or something like that. And that with that database, then artificial intelligence image recognition algorithms could run against that database and see how they do.
So like, oh, look at this image of you and I were looking at like that's a strawberry. But you don't give the answer to the algorithm and the algorithm figures out if it thinks it's a strawberry or a dog. Whatever. So, she and her collaborators start working on this.
It's super cool. They build the database. They use a mechanical Turk, Amazon mechanical Turk to build it. And then one of them, I'm not exactly sure who, if it was Fei-Fei or somebody else has the idea of like, well, you know, we've got this database.
We want people to use it. Well, let's make a competition. This is like a very standard thing in computer science academia of like, let's have a competition, an algorithm competition. So we'll do this annually and anyone, any team can submit their algorithms against the ImageNet database and they'll compete like who can get the lowest error rate, like the most number of images percentage of the images correct.
And it's great. So it brings her great renown becomes popular in the AI research community. She gets poached away by Stanford. The next year, I guess that's okay.
I went there too. So that's fine. And she's still there. I couldn't resist.
I couldn't resist. I was just like a kindred spirit to me. Do you know? I know you do know, but most listeners do not know what her endowed tenure chair is at Stanford today.
I do. She is the Sequoia chair. Yes. The Sequoia Capital Professor of Computer Science at Stanford.
So cool. Why does she become the Sequoia Capital chair and what does all this have to do with Nvidia? Well, in the 2012 competition. A team from the University of Toronto submits an algorithm that wins the competition.
And it doesn't just win it by like a little bit. It wins it by a lot. So the way they measure this is the 100% of the images in the database. What percentage of them did you get wrong?
So it wins it by over 10%. I think it had a 15% error rate or something in the next like all the best previous ones. I've been like 25 points something percent. Yes.
This is like someone breaking the four bit mile. Actually, in some ways, it's more impressive than the four bit mile thing because they just didn't brute force their way all the way there. They like try a completely different approach. Yes.
Boom showed that we could get way more accurate than anyone else ever thought. So what was that approach? Well, they called the team, which was composed of Alex Krasevsky. I was the primary lead of the team.
He was a PhD student and collaboration with Ilya Sutsker and Jeff Hinton. Jeff Hinton was the PhD advisor of Alex. They call it Alex net. What is it?
It is a convolutional neural network, which is a branch of artificial intelligence called deep learning. Now deep learning is new for this use case, but then you weren't exactly right. It had been around for a long time, a very long time and deep learning neural networks. This was not a new idea.
The algorithms had existed for many decades, I think, but they were really, really, really computationally intensive. They required to train the models to do a deep neural network. You need a lot of compute, like on the order of, you know, the grains of sand that exist on Earth. It was completely impossible with a traditional computer architecture that you could make these work in any practical applications.
And people were forecasting to like, when with Moore's law, will we be able to do this? And it still seems like the far future because not only did Moore's law need to happen, but you also needed the NVIDIA approach of massively parallelizable architecture. Where suddenly you could get all these incredible performance gains, not just because you're putting, you know, more transistors in the given space, but because you're able to run programs in parallel now. Yes.
So Alex and that took these old ideas and implemented them on GPUs. And to be very specific, he implemented them in CUDA on NVIDIA GPUs. We cannot oversee the importance of this moment, not just for NVIDIA, but for like computer science, for technology, for business, for the world, for staring at the screens of our phones all day every day. This was the big bang moment for artificial intelligence and NVIDIA and CUDA were right there.
Yep. There's funny. There's another example within the next couple of years, 2012, 2013, where NVIDIA had been thinking about this notion of general purpose computing for their architecture for a long time. In fact, they even thought about should we relaunch our GPUs as GP, GP, general purpose, graphics processing units.
And of course, they decided not to do that, but just built CUDA. Which is code word for like, we've been searching for years for our market for this thing. We can't find the market. So we'll just say, you can do it for anything.
Right. And so deep learning, generating a lot of buzz, a lot from this AlexNet competition. And so in 2013, Brian Katanzaro, who's a research scientist at NVIDIA, published a paper with some other researchers at Stanford, which included Andrew Ng, where they were able to take this unsupervised learning approach that had been done inside the Google Brain team, where they had sort of the Google Brain team had sort of published their work on this and it had a thousand nodes. And you know, this is a big part of the sort of early neural network, hype cycle of people trying cool stuff.
And this team was able to do it with just three nodes. So totally different models, super paralyzed, lots of compute for a super short period of time in a really high performance computing way or HPC as it would sort of become known. And this ends up being the very core of what becomes KuDNN, which is the library for deep neural networks that's actually baked into CUDA that makes it easy for data scientists and research scientists everywhere who aren't hardware engineers or software engineers to just pretty easily write high performance deep neural networks on NVIDIA hardware. So this AlexNet thing, plus then Brian and Andrew Ng's paper, it just collapses all these sort of previously thought to be impossible lines to cross and just makes it way easier and way more performant and way less energy intensive for other teams to do in the future.
Yeah, specifically to do deep learning. So I think at this point, like everybody knows that this is pretty important, but it's not that much of a leap to say if you can train a computer to recognize images on its own, that you can then train a computer to see on its own, to drive a car on its own, to play chess, to play go, to make your photos look really awesome when you take them on the latest. iPhone, even if you don't have everything right to eventually let you describe a scene and then have a transformer model paint that scene for you in a way that is unbelievable that human didn't make it. Yep.
And then most importantly, for the market that Jensen and NVIDIA are looking for, you can use the same branch of AI to predict what type of content you might like to see next show up in your feed of content and what type of ad might work really, really, really well on you. So basically all of these people we were just talking about, I bet a lot of you recognize their names. They get scooped up by Google, Fayettely goes to Google. Brian went to Baidu and he's back at NVIDIA now doing applied AI.
Brian went to Baidu, definitely goes to Facebook. So all the other markets like even throw out, say you don't believe in self-driving cars, you don't think it's going to happen or any of this other stuff. Like, it doesn't matter. The market of advertising, of digital advertising that this enables is a freaking multi-trilling market.
And it's funny because that feels like that's the killer use case, but that's just the easiest use case. That's the most obvious, well-label data set that these models don't have. They have to be amazingly good because they're not generating unique output. They're just assisting in making something more efficient.
But then like flash forward 10 more years, and now we're in these crazy transformer models with hundreds of millions or billions of parameters. Things that we thought only humans could do are now being done by machines and it's like it's happening faster than ever. So I think to your point, David, it's like, oh, there was this big cash cow enabled by neural networks and deep learning in advertising. Sure, but that was just the easiest stuff.
Right, but that was necessary though. This was finally the market that enabled the building of scale and the building of technology to do this. And in the Ben Thompson Jensen interview, Ben actually says this when he's sort of realizing this, talking to Jensen, he says, this has been talking the way value accrues on the internet in a world of zero marginal costs, where there's just an explosion in abundance of content that value accrues to those who help you navigate the content. He's talking about aggregation theory.
And then he says, what I'm hearing from you, Jensen, is that yes, the value accrues to people that help you navigate the content. But someone has to make the chips and the software so that they can do that effectively. And it's like it sort of used to be with Windows with the consumer facing layer and Intel was the other piece of the wind telemonophily. This is Google and Facebook and a whole list of other companies on the consumer side and they're all dependent on Nvidia.
And that sounds like a pretty good place to be. And indeed, it was a pretty good place to be. Amazing place to be. Oh my gosh, the thing is like the market did not realize this for years.
And I mean, I didn't realize this and I probably didn't realize this. We were the class of people working in tech as venture capitalists that should have. Ooh, do you know the Mark Andreessen quote? Ooh, no.
Oh, this is awesome. Okay, so it's a couple years later. So it's like getting more obvious, but it's 2016 and Mark Andreessen gave an interview. He said, we've been investing in a lot of companies applying deep learning to many areas.
And every single one effectively comes in building on Nvidia's platforms. It's like when people were all building on Windows in the 90s or all building on the iPhone in the late 2000s. And then he says, for fun, our firm has an internal game of what public companies we'd invest in if we were hedge fund. We'd put in all of our money to Nvidia.
This is like a, it was paradigm, right? That called all of their capital and one of their funds and put it into Bitcoin when it was like $3,000 a coin or something like that. Yes. We also have been doing this.
So literally Nvidia stock in 20, like recent, like this is now known, 2012, 13, 14, 15. It doesn't trade above like five bucks a share. And Nvidia today, as we record this, as I think about 220 share, the high in the past year has been well over 300. Like, if you realized what was going on and getting a lot of those years, it was not that hard to realize what was going on.
Wow, like it was huge. It's funny. So there was even and we'll get to what happened in 2017 and 2018 with crypto and a little bit, but there was a massive stock run up to like $65 a share in 2018. And even as late as I think the very beginning of 2019, you could have gotten it.
I tweeted this and we'll put the graph on the screen on the YouTube version here. You could have gotten it and that crash for 34 bucks a share. 2019. If you zoom out on that graph, which is the next tweet here, you can see that like in retrospect that little crash, this looks like nothing.
You don't even pay attention to it. And the crazy run up that they had to 350 or whatever. They're all the time I was. Yeah, it's wild.
A few more wild things about this. It's not until 2016. And again, Alex, that happens in 2012. It's not until 2016 that Nvidia gets back to the $20 billion market cap peak that they were in 2007 when they were just a gaming company.
That's all the 10 years. I really hadn't thought about it the way that you're describing it, but the breakthrough happened in 2010, 2011, 2012. Lots of people had the opportunity, especially because frickin Jensen's talking about it on stage. He's talking about earnings calls at this point.
He's not keeping this a secret. No, he's like trying to tell us all that this is the future. And people are still skeptical. Everyone's not rushing to buy the stock.
We're watching this freaking magic happen using their hardware, using their software on top of it. And like even semiconductor analysts who are like students of listening to Jensen talk and following the space very closely. So I think he sounds like a crazy person when he's up there, espousing that the future is neural networks and we're going to go all in and we're not pivoting the business. But from the amount of attention that he's giving in earnings calls to this versus the gaming.
I mean, everyone's just like, are you off your rocker? Well, I think people just lost trust and interest, you know, after like there were so many years of like they were so early with CUDA and early taking that, they didn't even know that this like, they didn't know Alex was going to happen. Right. Jensen felt like the GPU platform could enable things that the CPU paradigm could not.
And he would like, had this faith that something would happen. But you know, this was going to happen. And so for years, he was just saying that like we're building it. They will come, you know, and to be more specific, it was that, well, look, the GPU has accelerated the graphics workload.
So we've taken the graphics workload off of the CPU. The CPU is great. It's your primary workhorse for all sorts of flexible stuff. But we know graphics needs to happen in its own separate environment and have all these fancy fans on it and get super cool.
And it needs these matrix transforms. The math that needs to be done is matrix multiplication. And there was starting to be this belief that like, Oh, well, because the, you know, professor, the apocryphal professor told me that he was able to use these program, the matrix transforms to work for him. You know, maybe this matrix math is really useful for other stuff.
And sure it was for scientific computing. And then honestly, like it fell so hard into Nvidia's lap that the thing that made deep learning work was massively parallelize matrix math. And they're like, Nvidia is just like staring down their GPUs. Like, I think we have exactly what you are looking for.
Yes. There's a, that same interview with Brian K says about when all this happening, he says the deep learning happened to be the most important of all applications that need high throughput computation under statement of the century. And so once Nvidia saw that, it was basically instant. The whole company just latched onto it.
There's so many things to lodge in for, you know, he was paying a vision for the future, but he was paying very close attention and companies paying very close attention to anything that was happening. And then when they saw that this was happening, they were not asleep at the switch. Yeah. 100%.
It's interesting thinking about the fact that in some ways, it feels like an accident of history and some ways, it feels so intentional that graphics is an embarrassingly parallel problem because every pixel on a screen is unique. I mean, they don't have a core to drive every pixel on the screen. There's only 10,000 cores on the most recent Nvidia graphics cards, but there's not, which is crazy, right? But there's way more pixels on a screen.
So, you know, they're not all doing every single pixel at the same time, every clock iteration, but it worked out so well that neural networks also can be done entirely in parallel like that, where every single computation that is done is independent of all the other computations that need to be done. So they also can be done on this super parallel set of cores. It's just, you got to wonder like, when you kind of reduce all this stuff to just math, it is interesting that these are two very large applications of the same type of math. In the search space of the world of what other problems can we solve with parallel matrix multiplication?
There may be more, there may even be bigger markets out there. Totally. Well, I think they probably will be a big part of Jensen's vision that he paints for Nvidia now, which we'll get to in a sec is this is just the beginning. You know, there's robotics, there's autonomous vehicles, there's the Amiverse.
It's all coming. It's funny. We just joked about how like nobody saw this before the run up in 2016, 2017. There were all these years where like the Mark Andreessen knew, you know, whether he made money in his personal account or not.
I love to ask him. But then in 2018, another class of problems that are embarrassingly parallelizable is of course cryptocurrency mining. And so a lot of people were going out and buying consumer Nvidia, you know, graphics cards and using them to set up crypto mining rigs in 2016 and 2017. And then when the crypto winter hit in 2018 and the end of the ICO craze and all that, the mining rig demand, it fell off and this had become so big for Nvidia that they were actually actually declined.
Right. Yeah. So a couple of interesting things here. Let's talk about technically why.
So the way crypto mining works is effectively guess and check. You're effectively brute forcing an encryption scheme. And when you're mining, you know, you're trying to discover the answer to something that is hard to discover. So you're guessing if that's not the right thing, you're incrementing your guessing again.
And that's a vast oversimplification and not technically exactly right. But that's the right way to think about it. And if you were going to guess and check at a math problem and you had to do that on the order of a few million times in order to discover the right answer, you could very unlikely discover the right answer on the first time, but you know, that probabilistically is only going to happen to you once if ever. And so well, the cool thing about these chips is that, A, they have a crap ton of cores.
So the problem like this is massively parallelizable because instead of guessing and checking with one thing, you can guess and check with 10,000 at the same time and then 10,000 more and then 10,000 more. And the other thing is matrix math. So yet again, there's this third application beyond gaming, beyond neural networks. There's now this third application in the same decade for the two things that these chips are uniquely good at.
And so it's interesting that like you could build hardware that's better for crypto mining or better for AI. And both of those things have been built by NVIDIA and our competitors now. But the sort of like general purpose GPU happened to be pretty darn good at both of those things. Well, at least way, way, way better than a CPU.