AI is becoming more powerful, but the resources needed to build it are increasingly concentrated. Does it have to stay that way? MTS host, Sophia Du, heads to the open source AI summit in San Francisco. To ask researchers and founders across the AI stack, whether open source can create a more distributed future for AI. Lucas Kaiser, co-author of the Landmark, attention as all you need paper, argues that today's concentration may be a property of our current technology, not an inevitable feature of AI.
Transformers thrive on enormous amounts of data and compute, but they're less than a decade old, and the next breakthrough could change those economics. From open models and access to compute, to companies owning their own intelligence. Sophia explores where AI power is actually concentrated, what openness can change, and what it still leaves unsolved. Joining me is Lucas Kaiser, an AI researcher and the co-author of attention as all you need, the paper that introduced the transformer that completely changed modern AI. Where is modern AI and where is power currently concentrating?
There was a very important moment sometime around the end of last year, start of this year, where the current agents started to work. The nature of this AI is that you need to train it on very expensive data centers and on a lot of data. So there's big companies that do this and then they charge for it, right? That concentrates it in the sense to get the, like, to me, even the current models are not smart enough. So there still could be much smarter, right?
There may be some research breakthroughs that are needed to make them smarter with less data, but since research breakthroughs, you know, sometimes they come, sometimes they don't. They're not like a business proposition. But the big companies are like, okay, but you can do things without research, which is to go bigger, bigger, bigger. Now that is very concentrated. You go bigger, you need billions of dollars, you need to scrape data from every corner of the internet.
So the current state of the technology is a bit concentrating, but we should remember that it's just the current state. Short transformers are great, but they're not even 10 years old. I'm sure there will be other things. Maybe we'll make it possible to train with less data, more diverse models. Like, transformers are really good if you train them on the whole internet and then they're reasonably smart.
But if you try to say, you know, you just learn about this, then they're just stupid, right? Yeah. It doesn't work, but it doesn't work because maybe we need a research breakthrough that will make it work. But the transformer needs to be everything. And that's a problem, but it's a research problem that hopefully gets off.
With the transformer, architecturally, you need a lot of resources. And the more resources you have as a company, as a model, the more powerful you can be. But do you think there is a type of algorithmic breakthrough or some type of research breakthrough that will allow smaller players to really compete? Well, I mean, we know it exists. We are the proof, right? Humans are not experts in everything.
But in the field, there are experts that are very good, sometimes better than our super-futurables. So it exists. I don't think we have found it technically. It may even have attention later somewhere, right? It may be because it's not just the model. It's also the loss and the data and how it's trained.
There's a lot of factors, so we don't really know exactly where to look at the thinking research. But on the other hand, I think more and more people are looking. I hope. I think the labs have a little bit, no, Open A1I drink was a very research app. Now it's a research-lab bit, but it's also a very big company, like that needs to do products and stuff.
So it has less focus on researching. But this gives the chance to open source movements, that's a chance for academia to universities. I recently started doing things on my own also, and it's funny. So I bought a 59 TRTX view. It has more power than the eight HGPU machines we used as a team to do the Transformers research.
So you can buy and have one GPU that has all the power we use to design transformer. So you know, so you can do things with it. Like you can't train a big alarm, but you can research and experiment on things. And I think many people should. I believe there is still much more in machine learning to research them.
My last question for you is what makes you feel optimistic that the future of AI is going in a direction that will be truly open and broadly empowering for everyone? If you look at the world, humans are the most amazing computers in the world, right? Our brains are still unmatched by the models. And we are not generalists. And there's not like one big brain of humanity. I think fundamentally, it might be that given a fixed amount of data,
the best way to learn from it is to have a lot of distributed models. Strong on its own, but even stronger when there. And if you look at some basic research, there are actually reasons to believe it. I was just talking about examples and things like that. So yes, transformers are great when you feed them all of the internet.
But we know there is an algorithm that's even better and it can learn from much smaller data and be amazing at the things it's learning. We are the proof. We have not found it yet. How to do this in computers very well. But we will. I mean, progress in machine learning has not stopped in 2017.
Of course, because it's so good, everyone has focused on the big data. But now that it's so expensive, maybe we will focus back more on more fundamental research. And I'm absolutely convinced it can work and it just needs research and effort. I think many people sometimes look at the current state of AI and get a little pessimistic. It's like, you know, it's big data centers. Everything's so big and then you get a subscription to something you cannot do much about.
And to me, it's like, okay, but that's just the current state. You know, we're just this year next year. It's a step in the technology. But it's a step towards something much more fun where you can have everyone can have their own model. Maybe we'll learn from much smaller data. The models will be experts in different domains.
Like we are and maybe, you know, maybe one will be making jokes this way and another another way. It's like, currently when you ask a big language model, the joke is always the same. I don't think about that. This will change because it's an outcome of a technology and the technology will change when the research catches up. And also for researchers, there's like just a huge opportunity to catch up with this.
So, you know, like, just don't worry about, okay, it got concentrated because that was a convenient thing to do. And it showed that great things can be done. That's good. But there is, there is a thing ahead. We just need to just need some research breakthroughs.
Thanks for listening to this episode of the A6T podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating, we're a review, and share with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X, A6TZ, and subscribe to our substack at a6tz.substack.com. Thanks again for listening, and I'll see you in the next episode.
This information is for educational purposes only and is not a recommendation to buy, hold, or sell any investment or financial product. This podcast has been produced by a third party and may include pay promotional advertisements, other company references, and individuals unaffiliated with A6TZ. Such advertisements, companies, and individuals are not endorsed by AH Capital Management LLC, A6TZ, or any of its affiliates. Information is from sources deemed reliable on the data publication, but A6TZ does not guarantee its accuracy.