Skip to content
LLM Landmark: Unveiling a Colossal 3 Trillion Token Open-Source Dataset episode artwork

Feb 12, 2024 9 min

LLM Landmark: Unveiling a Colossal 3 Trillion Token Open-Source Dataset

from The Daily AI

In this episode, we celebrate a landmark achievement in language model development as the world's largest open-source LLM dataset, encompassing a staggering 3 trillion tokens, is revealed, promising advancements in natural language understanding. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: ⁠⁠https://AIBox.ai/⁠⁠ AI Facebook Community Learn more about AI in Music Learn more about AI Models

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

LLM Landmark: Unveiling a Colossal 3 Trillion Token Open-Source Dataset

0:00 9:15
of matches

Transcript Auto-generated, may contain errors

The headline here is that Allen Institute for artificial intelligence has pretty much unveiled what they're calling DOMA. So this is the largest open source language model data set. This is amazing. This is so heartwarming to me because this is what I think AI should be all about.

This is what's really going to help advance AI forward, whether it's this or future iterations of this. They're setting the example here and I'm here for it. So the Alice Institute for artificial intelligence is the AI to recently unveiled this massive open source data set, which is tailored for training on official language models. So it has a three trillion token size and DOMA really, I think because of that has emerged as the largest open source data set in the domain period.

So the creation of DOMA is part of AI to's OLMO initiative, which is really just geared towards establishing an open, transparent language model, which is definitely a very stark contrast to someone like OpenAI, who has a very black box when it comes to the data they put in there. For a number of reasons, right? They say it's because they're like, well, you know, it's our copyright, it's our intellectual property. But really, I think a big problem is people would be mad if they knew where all the data came from, right?

If they're going and scraping all of your Reddit posts and then using those to train and then charging you for them, people would be upset about that. So I think that's another reason why OpenAI is very non-transparent. So in creating this current data set, the Allen Institute really kind of stuck to five core principles. Number one, openness, representativeness, size, reproducibility, and risk mitigation.

Those are kind of five areas. And I want to go over what those mean kind of according to them. So openness, AI to responded to the researchers communities, pressing concerns about restricted access to pre-training core, but and associated language models, essentially they envisioned DOMA as a transparent data set that is open for thorough examination, right? What an awful concept.

You can literally go see what's inside of it, examine the contents. If there's something problematic or something illegal or something copyrighted, it should be able to be removed, theoretically. And so I think this is really interesting and this is the first step towards that. So as far as the whole representativeness, they aimed for DOMA's content to stand toe to toe with data sets employed by both private and open language models, a hem, open AI or close AI.

The third thing is size. So recognizing the potential benefits of large data sets, AI to incorporated findings from research that underscore the correlation between larger training, data sets and heightened model performance. The fourth thing is reproducibility. So AI to is really focused on community engagement, ensuring that any tools created during DOMA's creation are open for others to replicate or modify.

So honestly, they're doing a huge service to the entire community with this kind of open source, open concept approach. And I think a lot of people really appreciate that. So the fifth thing is risk mitigation, designing DOMA with a strong emphasis on privacy. The Institute ensured that its web crawl content could not be traced to actual individuals.

This is interesting, right? Because on the one hand, they're saying we have a very open data set where you can find, you know, everything that's in it, but they stop one step short of actually saying, you know, who said what in the data set, which is interesting, right? So like if they went and scraped Reddit, for example, they tell you maybe what subreddit stuff appeared on, but they wouldn't tell you what the user's names are now. That's interesting because perhaps that's for privacy reasons.

But it does. Yeah, I guess there's some tricky aspects of that when it comes to like attributing certain content to someone. And the reason I say that is like, if they ever were to do some play that I've heard, this isn't just like some moonshot. Like I've heard people flow this idea where essentially you compensate people when their data is used.

But really what that would look like is more like you're compensating like Reddit as a whole and non individual contributor that actually wrote a post that's getting used or, you know, so I'm not sure how that plays into it, but it's probably for legal reasons and whatnot. They got to keep the privacy of the individual sort of written something private beyond just like the website it came from. So in any case, this is, I guess, just another reason that if you are a content creator, you need like your own platform, your website to host a lot of your stuff and then it's probably a lot easier to claim it when these ais inevitably suck up all your data and train models off of it. So I think developing Dolma wasn't a very easy task, right?

This is the biggest open source data set ever. And so I would not expect this to be easy. The team grappled with some really critical decisions surrounding data sources, like I just mentioned. And then of course, pre-processing methods, language inclusion and strategies for personal data exclusion.

And essentially by employing a bunch of different practices that are pretty standard for data filtration and duplication, they were, and this is also coupled with, you know, measures to kind of weed out potentially harmful content. They essentially produce this data set with a really rich like conglomeration of everything on the web. So that was English literature from product Gutenberg, scholarly papers from PES 2 O Wikipedia entries and code snippets from GitHub. Unlike many private databases that really kind of shroud their components in mystery, Dolma really prioritized transparency here.

So even among its open data set peers, Dolma, I think kind of stands out just due to its sheer size and also its unique licensing terms. So I'm sure a lot of people are interested, right? What is the what is the licensing terms here for this? So licensing under the Allen Institute's impact terms, it's classified as a medium risk artifact.

So for those looking to tap into DOMA's potential, it comes with conditions centered on transparency, risk reduction and ethical usage. The license explicitly restricts its use in military operations, surveillance activities, or for generating disinformation, which I mean, sure, I'm sure everyone's like, chat, if you should put that in their terms, like, he's a for generating disinformation, I don't know. It's kind of funny. Like, it just like, it goes without saying, but also, you know, people do it anyways.

So like the people that are generating disinformation probably don't really care about the terms. But in any case, this licensing approach kind of strikes a balance between accessibility and the potential hazards of circulating such an extensive data set or so they think, but I mean, I'm not 100% sure. I'm not 100% sure what that means, right? Because like, let's say you download this data set, use it for reasons you shouldn't.

And like, let's say North Korea creates an AI that creates disinformation based off this data set. What are they going to do? Like, yell out North Korean telemedic to get back the data like too late. So I don't know.

I'm not saying that I actually get to get the thing they created this, but I just, I don't. So I think maybe in a lot of cases like this, they put stuff in the terms of service. Just so when something bad happens, they can have will be say like, it's against our terms of service. We told you not to, but whatever.

It's not, it's not a deal. It's the internet. All the information's out there. Anyways, I'm sure North Korea could build a giant web crawler scraper bot or probably already has an insect of all this info anyways.

But in any case, I think prospective users are going to be required to share their contact information and articulate their intended data set usage upon approval. They can download DOMA from the hugging face platform and for a deeper dive into the data set. A comprehensive data sheet is also going to be available. Additionally, a GitHub repository complete with code and instructions on DOMA's generation and exploration has been provided.

So AI2 has really big plans for DOMA. I think they're intending to augment it with more data sources and languages in the future, which is going to be interesting, right? Multiple languages beyond just English. And with this whole endeavor, the Institute aspires to really champion openness in AI and research and kind of pave the way for superior language technology advancements.

Overall, I think, you know, like a big tip of the hat to the Institute for doing this. I'm really excited for more projects like this to come forward. If we have a ton of open source models, a ton of open source data sets, this is like, this is amazing because this is how the community can build really creative, impressive tools, not get stifled by the costs of API calls or other things that may be able to afford. And so I think this is going to be a really interesting, exciting place to watch in the future.

Thank you for listening to today's podcast episode, breaking down how AI is impacting your industry. Today's episode is sponsored by AI Box, a no-code AI app builder and marketplace, which just launched a crowdfunding campaign. If you are interested in investing in a new AI startup, you can go to republic.com slash the i-box. I'll leave a link in the show notes as well to learn more.

The minimum investment is $150 and the maximum investment is $100,000. Until next time, have a fantastic day.

Frequently asked questions

How long is this episode of The Daily AI?

This episode is 9 minutes long.

When was this The Daily AI episode published?

This episode was published on February 12, 2024.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Daily AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.