Skip to content
Beyond the Giants: Unveiling Open Source Alternatives to GPT-4V episode artwork

Jan 26, 2024 13 min

Beyond the Giants: Unveiling Open Source Alternatives to GPT-4V

from The Daily AI

In this episode, we venture into the world of lesser-known open source alternatives to GPT-4V, exploring the advantages and unique capabilities these models bring to the table for those seeking alternatives in the language model landscape. Invest in AI Box: https://Republic.com/ai-box Get on the AI Box Waitlist: ⁠⁠https://AIBox.ai/⁠⁠ AI Facebook Community Learn more about AI in Music Learn more about AI Models

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

Beyond the Giants: Unveiling Open Source Alternatives to GPT-4V

0:00 13:24
of matches

Transcript Auto-generated, may contain errors

OpenAI's GPT-4v has been essentially called a game changer in the realm of AI, I tend to agree with this. It's pretty impressive as far as all of its multimodal capabilities, which really kind of help it to understand both text and images. And I think this versatility opens up a whole bunch of different applications, but I think it also prevents a bunch of different challenges. So today I wanted to dive into some of the strengths and vulnerabilities of this next generation of AI and explore some of the open source projects that are also in the same arena so I can keep you guys up to date on what else is out there.

And I think one other thing I don't know if I mentioned there in the intro was, of course, the fact that GPT-4 now, you can upload images. So I could take an image of a street that has a bunch of parking things on and say, am I safe to park here right now? If you let me know, you could go take a picture of a menu and say like, hey, what on the menu looks good for someone that's lactose intolerant and vegan and is on the cheaper end of the menu, right? Or whatever, right?

And then it can give you an idea. So it can read, it can understand context, very smart, very interesting. So yeah, let's dive into all of the capabilities. So, okay, I've literally just like backspaced what I've recorded five times.

It's a serious tongue twister. I'll try to get it right this time. If not, we're just moving forward. But multi-modal, ah, multi-modal models.

Jeez, multi-modal models. Jeez, it's so hard to say. In any case, chat GPT or GPT-4, right? They have a bunch of advantages.

So they can, obviously, they're quite effortless in handling tasks that other traditional text or image-based models struggle with. So an example of this would be GPT-4. It can give you instructions that are a lot clearer with a visual component, right? So if you take a picture of a bike and you're like, hey, how do I repair this bike?

What pieces are missing? Or what tools do I need to repair this bike? You send it a picture, it can do this. So of course, these models are not just limited to just image recognition.

They can also like extrapolate and understand image content to a certain extent. And I think this really kind of opens the door to more nuanced applications. You know, suggesting recipes based on the ingredients found in your fridge might be like a good example. You take a picture of your open fridge.

Hey, here's my ingredients. What should I make? So there are some interesting, there's some interesting red flags that some people have raised or concerns that some people have had. I'll cover them.

Some I think are warranted. Some I don't think are as important. But OpenAI initially kind of hesitated to release GPT-4V. That's what it's called, GPT-4V.

I think the V is for visual. And they did that, like they kind of hesitated because of concerns that it could be employed to identify individuals and images without their consent, which is, you know, kind of a concern that seems fairly validated. So even after it's released, GPT-4V has a couple issues. Some people are saying that it has the inability to recognize hate symbols or, you know, showing a propensity to exhibit biases against certain genders, demographics, and body types.

It's all the same stuff that, you know, ChatGPT had when it first came out and they kind of put up guardrails for it, but different people are complaining. I think, you know, yeah. Anyways, so the whole critique is coming directly from inside of OpenAI itself, which is interesting, right? They're the ones critiquing themselves, but of course they launch it because you have to test it.

You have to find out how you can break the thing. And I'm sure they've done a ton of internal stuff. But yeah, in any case, so I think despite all of the controversy that it has raised, of course, this is an incredible tool. It's incredibly powerful.

So I'm very happy that it has been released. And also there's a bunch of, you know, both commercial and independent developers that are actively pursuing the development of other open source multimodal models that aim to achieve some of our challenges. So some of these are less complex, but I want to take a close look at some of them. So there is LLAVA 1.5.

So recently a collaboration between researchers at the University of Wisconsin, Madison, Microsoft Research and Columbia University created this. Essentially it's an evolution of the original LLAVA, which combines a, and so now it just kind of combines a visual coder and Vicuna, which is an open source chatbot to make sense of images and texts. I think what sets it apart is its scalability. So making it feasible for use on consumer level hardware is kind of why it's special.

And I think it will get some adoption because of that. The other one I want to talk about is QuenVL and that's Google's offering. So Alibaba's QuenVL and Google's models like PA, LIX and PalmE are making waves in the multimodal space. So QuenVL in particular, it's licensed to companies with substantial user bases, providing a substantial option for large scale applications.

Google, who of course is a major player in AI, is also actively pursuing some of these multimodal models. So I think this is going to be interesting between those two. The third thing I want to talk about is Adept. So Adept is a startup specializing in AI models for web navigation and it has introduced a GPT-4V like kind of multimodal text and image model with a unique twist.

It understands, you know, quote unquote knowledge worker data, like charts and graphs, and this makes it good at kind of handling complex data and manipulation tasks. So let's talk about some of the challenges and limitations. So LLAVA 1.5 for, you know, with all this promise is not without some limitations. While it excels at zero shot objects, object detection, and also explaining images, it's not great when faced with complex or crowded images.

So I think, um, it's, you know, its text recognition capabilities are definitely falling short of GPT-4V, which can be, you know, definitely a blessing in disguise considering GPT-4V's vulnerabilities to malicious text-based manipulations. So, I mean, I don't know, some people say it's a blessing in disguise. But in my opinion, if the technology is worse, it's not a blessing in disguise. You know, you can put your guardrails or whatever else you want on it, but have the technology to be better and not just say like, whatever.

Yeah, I think it's a silly argument. But in any case, FUYU8B, this is what Adept has created. And that is kind of their open source project for this. But it's essentially an open source multi-modal model.

And it is different in its approach. So it's not designed for commercial use. And it's, you know, yeah, maybe they'll do that in the future. And its training data is restricted under specific terms.

So instead, Adept is looking to engage with the developer community. They're trying to get feedback and do bug reports. So essentially this new software out of them has the ability to understand unstructured data, which I think does make it very valuable for knowledge workers. However, it doesn't come with all capabilities prepackaged.

Instead, Adept fine-tunes more advanced versions for internal use. And I think this raises questions about like potential misuse and also the need for moderation mechanisms and a bunch of other stuff. Well, yeah, I mean, that's really a lot of people's concerns about this technology. In any case, I think, you know, if we're kind of looking at the road ahead, there are a lot of developers that are pushing boundaries in what open source models can do.

Some people are saying, you know, the risk and limitations of these models are evident. I think as we kind of move forward on this, it's important that, you know, people obviously develop things responsibly. But I think that this is a very exciting time where, of course, there's these really powerful tools out of OpenAI that are closed sourcing and got to pay for it. But there are a lot of people pursuing this in a more open source manner.

And I like to hope that, you know, I'm not hoping for the demise of ChatGPT and OpenAI because I do love the technology a lot of the time. But I do hope that there are a lot of other competitors, open source players in the space that do get used and incorporated. I would hate for, you know, the AI space to get sucked up into the vacuum of just the biggest player in the space. So definitely is exciting to see these other projects.

And it's, yeah, it's pretty, it's pretty cool. So one thing I did want to say is LLAVA in testing it. You know, someone uploaded a picture of a dog. It said, you know, detect a dog on the image, return coordinates for this.

And actually, like, it sent back the coordinates of where the dog was located, which is kind of an interesting concept to me, right? It's like computer vision, but it's just going to be like the coordinates on the image. So you'll be able to select it. That was kind of cool.

Another thing that LLAVA's, you know, ChatBot and their image detector could do, there's like a picture of a guy. LLAVA's chatbot was able to say, um, the quote, the unusual aspect of this image is that a man is ironing clothes while standing on the back of a moving car. This is not a typical scene, is ironing clothes. It's usually done indoors in a stationary position and with proper safety measures.

Anyways, so I thought that was pretty good. Like, obviously, this thing's very intelligent. That's the kind of response I expect to get out of something like, um, something like ChatGBT. And what LLAVA has when you ask a question like that is it has the ability to upvote, downvote, flag, or, you know, regenerate or clear.

So, um, you are able to, I guess, like stuff. Another thing that someone tested out on LLAVA, um, they took a picture of, like a pile of coins. Is that how much money do I have? And it said, you have a total of four coins, which is a total worth of two euros.

Um, in addition, they took a screenshot of just like text from, like blog posts and said, um, read the text from this image. And it, you know, was able to type out all the text from the image, which I think was very interesting. You could think of a lot of, um, people might have concerns about like bots essentially doing captchas with this technology. Like this would be super easy to write a script to get all captchas down in the future, which you've seen captchas are definitely getting a lot more complicated now.

Uh, all sorts of like rotate this animal until it's in the correct position or whatever. So that'll all get, uh, that'll all get a little bit more, uh, you know, there's gonna have to be a lot of work done for captchas. One thing that a lot of people were, uh, criticizing, if you go to LLAVA and someone, someone did a screenshot, essentially uploaded a picture of a woman, um, and they said, what advice would you give this person? And it says, as a responsible AI, I cannot provide personalized advice without knowing the specific context of the person's situation.

However, in general, it's important for individuals to remain or to maintain a healthy lifestyle, which includes a balanced diet, regular exercise, proper sleep. What it's, you know, what it's saying here is a picture of a larger woman. And so essentially the research was trying to get it to say like, you know, what advice would you give this person? It's trying to say like she needs to lose weight, right?

But I mean, essentially here, I'm not even that mad about its response. So a lot of people were triggered by this and were criticizing this because it talks, it says the word maintaining a healthy weight at the end of this. Um, yeah, so that's what people are triggered about. I'm really not mad about this response here.

Um, it doesn't seem to go off on any unkind things. It's, it's very like, I don't know the context, but talk to a healthcare professional in general, like healthy lifestyle, balanced diet, regular exercise, proper sleep. Like those are all things that I don't think are very controversial. I know some people will probably criticize me, but whatever.

That is what it is. Um, so yeah, anyways, that was LLAVA. I think like in my opinion, that's probably dark for hell is making it respond in that way. And also I personally think that does a great job.

It'll be interesting to see which of these AI models comes out ahead. There's going to be controversies with all of them. So it'll be a very interesting topic to continue following into the future. Thank you for listening to today's podcast episode, breaking down how AI is impacting your industry.

Today's episode is sponsored by AI Box, a no-code AI app builder and marketplace, which just launched a crowdfunding campaign. If you are interested in investing in a new AI startup, you can go to republic.com slash AI-fox. I'll leave a link in the show notes as well to learn more. The minimum investment is $150 and the maximum investment is $100,000.

Until next time, have a fantastic day. Manufacturers today are under pressure. Tight margins, labor shortages, supply chain surprises. Visit Epicor.com to learn how Epicor makes it easier to stay ahead.

Our industry-specific ERP and built-in AI give you real-time production disability, smarter inventory planning, automated workloads, and financial insights you can act on. It's everything you need to run a more efficient, more resilient operation. Built by a company that's partnered with manufacturers for more than 50 years. Discover what easier really feels like at Epicor.com.

Frequently asked questions

How long is this episode of The Daily AI?

This episode is 13 minutes long.

When was this The Daily AI episode published?

This episode was published on January 26, 2024.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Daily AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.