breaking news out of open AI, a brand new model, or really an update of a very popular model has just been released. And Sam Altman just tweeted out and said, good new model out. And he's referencing an update that open AI just tweeted out where they said, GPT4O got an update. A little party emoji.
They said, the model's creative writing ability has leveled up more natural, engaging, and tailored writing to improve relevance and readability. It's also better at working with uploaded files, providing deeper insights, and more thorough responses. Okay, I'm going to unpack everything going on here because this is absolutely fascinating. There's a lot of interesting things people have criticized, but also people are sort of applauding with this.
Before I do that, I will say, if you haven't already joined the AI box waitlist over on AIbox.ai for my AI tool that is coming out shortly, I would love you to join. And if you want to get daily AI news straight into your inbox with just pure signal, check out the bottom of AIbox.ai. There's a little newsletter thing you can fill out. You can get on the waitlist, get the newsletter, and we'll be sharing every single day of my top three AI stories.
I try to make this short, concise, useful. So check it out. I hope this is really valuable to you. Link's in the description.
All right. So over on, over with everything happening over on OpenAI, what I think is really fascinating here is they haven't really announced if, you know, per se, they went and did like a retrain of this model, and people are betting that they haven't. This is just some essentially almost like software updates or tweaks they're able to do to make this a little bit better, maybe even better prompting as we feel like it's happening with O1 preview. So this is really interesting.
A lot of the comments on this update are people, you know, being disappointed. This isn't GPT-5. There's someone that said, can we get pin chats folders in the chat history sidebar? Thanks.
Absolutely. This is something that I've been begging them for for like, for like two years now. So I think it's really interesting. Someone said, LOL, everyone's waiting for O1.
OpenAI, another update for 4.0. Okay. So it gets deeper into what is so interesting that's going on here. So there was LMArena.ai that just tweeted.
So by the way, this is formerly LMSys.org, which is kind of like this benchmark platform that everybody uses. It's now called Chatbot Arena. Chatbot Arena is kind of the name. But anyways, it's LMArena.ai is the website.
You can see that all these websites are getting more popular. They're buying better domain names than LMSys.org now. LMArena.ai makes a lot more sense. Anyways, this is what they said in a recent tweet about this whole announcement and what came out and if it's actually better.
They said, over the past week, oh, and by the way, I'll just explain how this chatbot arena works. It's essentially users can go on there and they're given like two different chat responses and they pick which one they like better. And so it's the chatbot arena. It's anonymous.
You don't know which chatbot is giving you the response. There's hundreds of chatbots on there. You can talk with it and just pick which responses you like better. It pits the best models against each other and whoever gets the highest score by the end essentially rises in the rankings.
I love this because it's not just like voting on, oh, I like OpenAI better than I like Google. It's anonymous. So it's essentially just kind of like a blind test on what one has the best responses. Okay.
So this is what they said. Over the last week, the latest OpenAI, ChatGPT4020241120, competed anonymously as, quote, anonymous chatbots. It's just called an anonymous chatbot. It gathered 8,000 plus community votes.
The result, OpenAI has now reclaimed the number one spot, surpassing Gemini EXP1114 with impressive 1,361 score. Okay, what this means is just a week ago, Gemini came out with a model that shot to the top of the rankings on the chatbot arena. And essentially everyone was like, oh my gosh, OpenAI and Profic, but it's funny because you can probably assume the first day that Gemini hit that top chart, OpenAI put in this new model. Sometimes I wonder if OpenAI is just sitting on the sidelines with models ready to go in case something big happens.
They're like, all right, just push this. Or maybe it's not even the best thing they got. They're like, oh, this is significantly better. They don't want to get bumped down from the number one spot on this, right?
They never want to be seen as number two. They always want to be number one. So they were number two for about a whole week while this new chatbot was competing anonymously. And then this is what Ellen Marina tweeted about.
They said, latest GPT40 shows remarkable improvements. We see a leap in creative writing. It went from a score of 1356 to 1402. So up, you know, almost 40.
And they said, as well as technical domain, coding, they went from number two to number one on creative writing, number two to number one on coding, number two to number one on math. They went from number four to number three. So they sell some more to do on math on hard problems, which a lot of people call logic reasoning, I believe. They went from number two to number one.
So they said, huge congrats OpenAI, more analysis below. So this is fascinating because I love when we have these models come out and beyond just like looking at Reddit or something where people are complaining about which ones is better or worse for different use cases, the chatbot arena is a pretty, in my opinion, it's a pretty good take on this. Now, this isn't without criticism. Some people are saying OpenAI is playing chatbot arena and they're not making their models better.
They're just figuring out how to work, game system a little bit, right? Okay. So I want to break you down into this analogy. Um, but first things first, I mean, I do want to give a kudo to OpenAI for continuing to push.
Some people have complained that yeah, they're focusing on GPT-401. Oh, and preview is still this beta thing. Everyone wants it to actually be live and available. Um, so Ella Marina posted essentially a bunch of screenshots of how much better, um, the GPT-401-120 is.
And it's pretty impressive when you are comparing this thing to other AI models. Uh, for example, you know, it's, it's the only model that has cracked above, um, above 1400. Uh, and you can, you can see like on the very far end, you know, below, below 1300, you have models like the llama 3.1-405b, which is essentially that is very, very best model. So that is very, very best model.
Lama 3.1-405b, 400, yeah, whatever, 405 billion parameters. That is somewhere around, you know, 1250, 1260. And we're already having this one up in the 1400, you know, the past 1400. Really, really impressive.
Now, are they just gaining, uh, you know, the benchmark? What's going on here? Um, I mean, side note, funny, they had a typo when they first released this, uh, when they said it was called 1111, but it's 2011, whatever, they fixed it. So in any case, um, some people are looking at all these benchmarks are saying they're shocked that Claude 3.6 is number eight in the arena.
Um, so sometimes it's shocking, right? I know a lot of people say they love Claude. It's got a better tone and style. They use it for everything.
And so, yeah, to me, it's kind of funny. Um, but this is, and then, you know, in response to that, someone said, I'm starting to think that Claude Rose or Pagefields, they swear up and down as best model ever, but the benchmark say otherwise. So, you know, there's some fighting words in the comments here. Um, this is, uh, this is what I think is interesting though.
John Howell said, it appears they released it anonymously together, data fine-tuned and over fit on these leaderboard evaluations. Don't trust this leaderboard too much as it's being gained by OpenAI and others. So someone thinks it's being, someone thinks it's being gained, which I think is interesting. Um, and that was a AI analytics consultant that said that, um, not, I don't think anyone too famous, but, um, this is interesting.
This is a big point that I've seen in a bunch of these different comments. Someone said, I'm still puzzled how O1 mini continues to be O1 preview at coding. In my experience, I get, I'll use O1 mini at coding on the web. And when it gets stuck on something, there's usually a good chance O1 preview can solve the issue.
Okay. This is what people talk about other models because they're all in the same screenshot. So bottom line, uh, 4.0 is getting better, but at the same time, most people are still waiting for, you know, O1 preview to come out. They're saying, you know, they're complaining that maybe it's not as good.
I mean, that's the reason it hasn't come out fully because it's just not as good at everything. Um, because yeah, in benchmarks, people really, really are loving the, you know, the GPT for O1 and it's ranking higher than O1 preview, which is at like spot number three, um, just under Google Gemini. So Google Gemini's newest model is beating like this O1 preview, this brand new model that's supposed to come out and crush everything. Um, so yeah, that's, that's pretty, pretty crazy.
And somehow the O1 mini, meaning like the smaller condensed faster model is doing better or is scoring higher on like for coding than O1 preview. So there's a lot of things that are a little bit perplexing here. Overall, you gotta give a big kudos out to the whole team over at OpenAI, just everything they have been able to put out, um, and all of the work that they've done that, you know, the benchmark speaks for itself. This thing is really accelerating.
So if you haven't tried it already, I would encourage you to go over to chat GPT and try out the new model. Um, try out some of the updates, you know, help you, uh, creative article about Egypt, um, which I just did to test it out. And you can see how much, I don't know, like how interesting it is. Okay.
So I just had read a great article about Egypt. I'm just testing its alleged creativity. I'll give you the first blurb and you can, you can decide where we're at. Said discovering the wonders of Egypt, a timeless journey through history and culture.
Egypt, a land of timeless allure, stands a bridge between ancient civilization and modern life. Renown for its iconic landmarks, vibrant culture, profound historical significance. Egypt continues to captivate travelers and historians alike. Okay.
It sounds a little bit less strange words, but somehow timeless allure sounds kind of funny to me. Um, even, I don't know, some things are sound a little too cliche, vibrant culture. I don't know. That's just me.
But overall, I am noticing some of the words that they're saying. It seems a little bit less chat GPT. I haven't heard it say the word delve. Delve is a clear giveaway.
Uh, it says let's delve into this, that it's a chat GPT. So sometimes I wonder if they just like manually turned off some of those words or if they, you know, put some extra prompts in here to try to make this thing sound a little bit more natural. So overall, you know, creativity is up. How the, how the sounds is up.
The tone is up a little bit. Um, up to you to decide if this really competes with, um, Anthropix Claude. But overall, I'm really excited for others are like, Oh, they got stagnant, which by the way, I don't think that's the case of Google right now. They're really cooking with a lot of these new models, but we can claim that some of the bigger companies are stagnant.
OpenAI, you have to admit, is just doing things so scrappy. Like a startup, like throwing out anonymous, kind of ranking out rank Google. Uh, you gotta love to see it. So they definitely have a lot of fight left in them.
Absolutely fascinating. Again, if you're interested in getting the waitlist for AI box, go over to AI box.ai and check out my newsletter that we send out every single day of our top three AI stories at the bottom of the AI box website. And you can, um, yeah, get those straight to your inbox. And, uh, hope you guys so much for tuning to the podcast today.
I will catch you next time.