Skip to content
S4E15 Computer Vision In Action episode artwork

Dec 6, 2024 52 min

S4E15 Computer Vision In Action

from Invisible Machines podcast by UX Magazine Hosted by Invisible Machines

Despite being a massive piece of the bigger AI puzzle, computer vision doesn’t always get the recognition it deserves. Robb and Josh welcome Dr. Jason Corso, Professor of Robotics at University of Michigan for an In Action episode that explores the complexity and power of machines that can see. Jason is the Co-Founder and Chief Science Officer of Voxel51, which is backed by Bessemer Venture Partners, and in this episode he demonstrates open-source tools available for working with massive datasets that include images and videos. The trio also discuss the border implication of adding the context that vision can afford to multi-agent systems. Explore Dr. Jason Corso’s Voxel51 platform: https://voxel51.com/ FiftyOne Open Source: Voxel51's open source toolkit for building high-quality datasets and computer vision models. FiftyOne Teams: Voxel51’s commercial offering based on open source FiftyOne. People can try FiftyOne at try.fiftyone.ai. Voxel51 on Github: https://github.com/voxel51/fiftyone A course on Coursera that Voxel51 created called "Hands-on Data Centric Visual AI" — https://www.coursera.org/learn/hands-on-data-centric-visual-ai Learn about orchestrating AI agents for your team: AI agent landing page with the whitepaper: https://onereach.ai/ai-agents/?utm_source=soundcloud+&utm_medium=social&utm_campaign=computer_vision_in_action_episode&utm_content=1 Learn about the only tools for orchestrating AI agents that are named a leader by all major analyst firmsAI agent landing page (dev): https://aiagents.onereach.ai/?utm_source=soundcloud&utm_medium=social&utm_campaign=computer_vision_in_action_episode&utm_content=1 #AIPodcast #TechPodcast #InvisibleMachines #AI #ArtificialIntelligence #AIAgents #AgenticAI #ComputerVision #MultiAgentSystems #DataScience #TechInnovation #AIinAction #GenerativeAI #NoCode #Automation #DigitalTransformation #AITech

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

S4E15 Computer Vision In Action

0:00 52:53
of matches

Transcript Auto-generated, may contain errors

Welcome to Invisible Machines, a podcast where we explore the expanding world of AI agents and conversational technologies. I'm Josh Tyson, co-author of Age of Invisible Machines, a best-selling book all about the agentic approach to automation and the pursuit of organizational AGI. I'm joined each week by Rob Wilson, CEO and co-founder of OneReach.ai. Rob and I wrote Age of Invisible Machines over countless hours of conversation, much like the ones we have each week on this podcast. Rob has spent decades researching and deploying conversational technologies, and Invisible Machines is where we share big ideas with fellow AI insiders.

We're continuing these conversations in partnership with UX Magazine because attention to experience design is paramount to the proper deployment of AI agents or invisible machines, if you will. Today, we're bringing you an in-action episode exploring the world of computer vision with Jason Corso, professor of robotics at University of Michigan. Jason is also the co-founder and chief science officer of Voxel 51, a leading open-source software framework for the development of computer vision data sets and models. Though often associated with things like self-driving cars, As you'll hear us discuss, computer vision is deeply connected to the burgeoning world of AI agents. We don't hear as much about computer vision as we do about, say, LLMs, but giving machines the ability to see things opens up a vast trove of contextual information that can push AI agents into new and exciting territory. As with all of our in-action episodes, this is a visual demonstration. Jason shares his screen as he walks us through some computer vision models and data sets, and we recommend watching it on the Invisible Machines YouTube channel. As I've already hinted, computer vision is going to become an increasingly critical component

to multi-agent systems, so let your brain feast on this in-action episode with Jason Corso. Invisible Machines is produced for UX Magazine in partnership with OneReach.ai. Named a leader by every major analyst group, OneReach.ai's generative Studio X platform was built specifically to facilitate the no-code orchestration of AI agents. GSX is trusted by global giants like PepsiCo, PwC, Deloitte, and DHL, but anyone can experience the level of agent orchestration we talk about on this podcast through interactive demos at OneReach.ai. Now, on with the show. All right, well, Jason, delighted to have you on the show. Thank you for joining us today.

Great to be here, Josh. Rob, a pleasure is always seeing you. So Jason, one of the things that we were sort of interested in talking about before we get into the demo portion of this episode is, you know, with LLMs kind of bursting on the scene, I feel like natural language understanding is now kind of a well-explored topic. There's a lot of people discussing language comprehension and it's sort of top of mind for a lot of businesses and organizations, but you work in AI vision and vision understanding is something that maybe doesn't get as much attention as it should, because it feels like it will be a very big piece of sort of the AI systems that come to kind of... you know, be present in every aspect of our lives in the near future. Would you want to talk a little bit about video understanding and like some of the key challenges there that you've been working around? Absolutely. Yeah, I totally agree, right? Like, I mean, we're humans, we communicate, we connect with people through language. So it's like, it's natural that NLP or natural language processing or NLU, natural language understanding through the excitement of the recent LM boom, you know, it kind of came first. But, you know, like as humans also, we, at least those of us who have the privilege of sight.

We use our vision quite a lot, right? Like there's a vast portion of our brain that's dedicated to vision. You know, like when I teach computer vision, which is kind of the name of the discipline AI is formally called computer vision. We call it visual AI just to connect it to like the action you take. But ultimately computer vision is the problem of like turning visual data into insights that then you might want to go act on. But when I first, when I teach computer vision, I normally like day one, we'll normally say something like, you know, how many of you walk into a wall on the way to the classroom? And like, well, no one said, obviously no one walks into a wall, right? Why? Because you're so good at seeing, we don't actually.

we don't think about how good we are at seeing and yet how complex the visual world is, right? Like I'm looking at you through this video recording right now. Half of my other view is a beautiful fall foliage. The other half is a whiteboard, right? Like way tons of complex scenery right in front of me here, but I can focus on you. And I don't have to think about that. But if I wanted to think about what was on the whiteboard with a student or whatever, right? Like that would also be possible. And I think, you know, the feel of computer vision, which is, you know, it's actually older than you think. I believe it was first, a problem in computer vision was first investigated at MIT, I believe in like the mid fifties.

Um, you know, and although we've seen massive growth in the last five, 10 years, um, you know, it's, it's definitely, definitely a well-established field. Uh, what we like to say at, at box 51 is that, um, you know, this is 2024 is the year of visual AI because, you know, like where, you know, where language got the last few years, right? Like now everyone's realizing like, if we connect visual data to language data, to make other inferences, whether or not it's like a, I don't know, like a thinking cap that you can put on to help you cook a meal that you've never cooked before. It can kind of point, point to you what to do or, or it's like an objective. Well, I like the NFL. If I can objective ref for the NFL, wouldn't that be nice? We don't have to think about flags being thrown all the time or whatever, right? So, you know, there's just a huge amount of opportunity in the field of computer vision. And we're not alone, you know, in this conversation, right? There's something like, I can get you the stat if you want, but like 85% of all the bits that go through network routers on the internet are actual video bits. You know, like think about like your iPhone or your Android phone, like whereas five years ago, it used to take actual photos. Nowadays, like when it takes a picture, it's actually taking a short video and then you get to scroll through it. Right.

pick the right frame, right? So just a huge amount of opportunity with the visual world. And, you know, I think, you know, it's high time to the, you know, it's exciting that we're seeing so many startups coming online and so much growth in the domain. Yeah, and there's two areas that, like, one thing is language. And it's not that vision is void of the language problem. Like, a lot of it is, like, converting image to language. So it kind of encapsulates both paradigms. And then also the complexity, you know, language has this, like, typical built-in benefit oftentimes of being somewhat focused.

I sort of think of it as like trying to do voice detection in a room with 100 people speaking at the same time, right? Because that's kind of what visual is, right? It's like, which person are we going to focus on? And it adds just this layer of challenge that normally in text or voice... interactions is just, you know, not as common in the environment to have 100 people talking at the same time. Yeah, I mean, great example, right? Like, although I can say I've been at some gatherings or parties where, you know, I'm talking to someone, and obviously, like, in a room with a lot of people, we humans are really good at source signal separation, which is kind of like a technical term for that problem. But like, if someone over there or whatever is saying something I'm interested in, I honestly do have a hard time maintaining my focus. But envision, that problem is called context, right? Like, how can you do scene understanding?

when, you know, the goal is to, whatever, navigate an autonomous vehicle down the road or, you know, help a surgeon figure out what's the right tool to pick up now or that they just missed a suture, you know, when the visual field is indeed rather complex and that, like, attention that you have to play through that messy context is one of the hardest problems in vision, indeed. Yeah, and there's overlap, too, right? Like, context is hard with language, too, and vision can provide extra context. Like, one of the reasons humans, I think, are good at making, like, figuring out what someone actually needs is we can track their body language, we can see what their facial expression is like, so there's opportunity there, too. Far away from, like, lip reading.

Oh, yeah, good question. You know, I actually, when I was, so I've been a faculty member for way too long now, since about 2007, I think. And I was originally thinking about when you're a faculty member, you just get started, right? Like you got to make an identity for yourself and kind of choose a direction that in which you believe you can succeed and there's not too many other people in there. But if you're alone, then no one's going to care about what you do, right? So it's kind of complex. But anyway, lip reading was a problem that I was interested in exploring. And this was like 15 years ago or more, right? So at that time, I already felt like we were mature enough to, as a field, it would be hard to be seen to make a revolutionary advance. Looking back, maybe that wasn't the best decision because I don't know of a tool that you could use still for lip reading.

Um, but, but, you know, anyway, I think I've, I've done it right, but I was, uh, but problem selection even, right, is, is interesting. Even like, like, I have a slide, Josh, uh, to, like, exactly to get at what you were just saying about, like, the complementarity between vision and language, right? Like, so I've been thinking about language. almost since I began. So instead of lip reading, I chose vision and language, right? That was my direction since I've been faculty. Like I won a grant called the National Science Foundation Career Award that was all about image to text, like image captioning in like something like 2008. But I have a slide in which I show some examples of like challenging visual phenomena that are hard to, like that have as complex lingual phenomena. Like in one of them is like a donut hole, right? Like if you, I don't know if you, if you know what like Tim Hortons is or like, right? Like you get like pop-ums or donut bowls as they're often called.

But like the donut hole is actually not the hole in the donut, right? It's the actual thing that results from the hole you cut out of the donut. And like, I mean, unless you've seen that, no one's going to, I mean, it's hard to describe. It's messy to describe, right? So. Right. Yeah. If you want to buy a donut hole and you don't, you've never seen one, you think you're just buying air. Exactly. It's a concept, a delicious concept.

But even like in other things, we've done a lot of work in instructional video understanding. That's a common problem in the field right now that I've played a role in kind of emphasizing or creating. There are some things that clearly require multiple modalities. Like a typical step of a recipe might be something like add oil to coat the bottom of a pan. Right. But like, unless you actually, like you read the text or you listen to the text and then you see the action of the pouring, like coating the bottom of the pan can mean a lot of different things. Right. If I'm like, if I have a big hole in the oil bottle, I might just, I might fill it with an inch deep, but we're like, that's not what that means. But I only know that because I've, you know, I've cooked things in way too much oil sometimes. And I've, or I've seen the example of it, right? Like that multiple modality is really key. And I think that's really where like vision is like coming out is, is, is like increasingly in the main stage because like, you know, it's back to the language bits, right? Like you could only do so much by reading textbooks, right? Like as a kid, I was often in.

I don't know, ridiculed, politely ridiculed by my family as being like book smart, but like, you know, the common sense part of it that you get from experiencing things in the physical world or looking at them or doing them and so on. I think, you know, LMs are great, but without like looking at things, without touching things, like the actual experience of going through the process, that's why I think visual AI, embodied AI, that's where the field is going, right? Obviously powered and fueled by LMs, but like, but I think that's the clear next horizon. Yeah. It's almost like watching a kid learn to ride a bike while holding a book and trying to read how to ride a bike. Yeah. Like someone needs to show them. Yeah. Yeah. And even in that example, I mean, I've been riding a bike forever now, but like if you overthink as a kid, how you're, when you're riding that bike, what you're supposed to be doing, you generally won't, you'll generally fail. Right. Or I think that the grownup analogy to that is like your golf swing, right? Like I've been playing golf like 30 years. I'm not very good, but I, you know, I hit like bogey golf, but like if I overthink my shots.

I mean, generally the ball is going to go in the wrong direction because I just wasn't working through the trust, if you will, in the motor experience. Well, I feel like singularity and AGI come up a lot just when people are interacting with large language models because they can speak so convincingly and they seem to know so much. But when you fold in the ability to see things and contextualize that way, that seems like that will be a huge leap forward for systems and models. Yeah, I mean, I totally agree. I think the connection of these capabilities to actual tasks or actual goals, I think it's critical to drive...

maybe not even like the next generation of systems, but like foundationally, right? Like, I mean, humans, I mean, I'm not an expert in human learning, but I have three kids and I've watched them learn, right? Like, I mean, my seven-year-old, I mean, she reads, but she certainly hasn't read all the world's knowledge. Yeah, she's very smart, very capable, like making himself food, like can interact with people she's never met before. She knows like the social norms of that interaction. Like all of those, all of those things are like core to like a sentient being, right? Like, I don't even think we're looking at those problems in AGI or AI, like in contemporary AI. I think we realize.

those those rationals among us realize that there's a huge a huge horizon of growth we have to do as a discipline to understand like right like i mean just very concretely right the human brain runs on something like 20 watts of power um i just about a year ago or 10 months ago i bought a 8 gpu box for my lab h100 box for my lab that i believe needs something like 8 kilowatts of power to run And, like, the number of nodes or the amount of data that can process through that is still a tiny fraction of, like, the actual neurons we have in the human brain. Like, do we need such big models? Like, I think these are, like, the questions that are being asked are, or will be being asked increasingly are, I think, partially along these lines, right? Like, how do we think more, you know, with more principles in mind, with more goals in mind, like, in terms of, like, grounding the exciting work that everyone's doing into a little bit more goal-oriented aspects so that we might drive more energy-efficient, more representation-efficient, more compute-efficient type models.

I think vision's a good driver of that, right? Like, I don't know if you've ever seen the experiment, but if you take... If you take any image that you can snap, like even the image of this screen right now or like this foliate outside my window here, like, and you look at neighboring pixels and just take the difference of their intensity value, like vertically or horizontally or so on, it follows an inverse exponential distribution, which means like it goes like this, right? Like, and like the peak is basically when there's no difference between neighboring pixels or like one intensity value difference. Whereas like the tail of that distribution is when there's any big difference, which signifies like an edge or some other content change, which is just to say like in the visual world, there's a huge amount of redundancy, both in like one frame spatially, but also frame to frame in video.

And we humans know how to deal with that well. Like that's how, as we've grown, we've learned about energy efficient processing and so on for what to pull out, what's informative, what's interesting. And there are methods and vision that look at those approaches, but like VLMs or LLMs and so on, right? They don't necessarily, they're not sort of looking naturally to exploit those statistics of natural images or like other regularities because there's no need to be power efficient right now, right? Like we'll throw as much compute as we need as a discipline. We're in this mindset of like more data, more compute, more power. Let's see what we can do.

Uh, and then I think when we realize that when that's going to plateau, maybe it has for the LMs already, I don't know. But, uh, then I think we'll begin asking a new set of questions, which are a little bit more like, like I was just saying, right? Like efficiency in the models or like, you know, potential power or representation or goal orientedness. And I think that will drive the next phase of growth. I'm a sort of a card carrying member of computer vision since my doctorate, but like, um, we often look at problems that we're, we're trying, we try to be too general in how we solve them, right? Like I want like the most generic, you know, image segmentation method possible. Well, I mean, however you segment images depends on the goal that you're like, what are you using the segmentation for? Or like the object detector like that, right? Like even.

like to map it a little bit more concretely, right? Like in autonomous vehicles, right? Like you can, one can either like try to detect the road surface, you know, all the facades of all the buildings next to you, all the occluders or like other actors in the scene, like the pedestrian, poplites, like you can kind of build that full reconstruction. Or you can shoot radar into the scene and look for empty space. And hopefully the empty space aligns with where you were planning to go on the map. There are many different ways to solve a problem. And if we don't look at the right goals, I think sometimes we make it harder than it should be. So when people have said like, oh, once a car can see well enough to drive, like we've taught machines how to see. I think what you're saying is not necessarily because they can see for the purpose of driving for that specific explicit purpose. But the idea that now we've taught machines to see.

for any context is not true. It doesn't mean it's, I'm not saying that that couldn't be the case, but the sure fact that it can see well enough to drive does not necessarily mean it can see well enough to do surgery. Yeah, 100%, right? I mean, like how much of the U.S. mail gets sorted by automatic address reading machines now? Right. I think the number is like 99%, some ridiculously high number. It's not a new number. It's not a new system. I'm pretty sure it was deployed. There's a document analysis center at the State University of New York at Buffalo that built the prototype of that system in like 2004, 2005, I believe, maybe even earlier.

right like this is this has been around for a while but does that mean computers can read right like not even right like that system was built for a specific task a specific problem um as far as i understand like it looks at the zip code part first and digit recognition is easier than letter recognition and then it was able to use the digits for the zip code to reduce the search space to understand what street it was and then when it knew the zip code in the street it was able to look at what possible house numbers were there right like or business address numbers right like that's a great engineering solution to a well-defined problem general seeing is is not well defined right so we're gonna interesting because you think you think about like Tesla and their approach to self-driving cars is like a car, essentially a car that could drive on any road in the world, you know, just fine. But then, and obviously that's not been easy to achieve, but then you see Waymo making it work because instead of having a car that can drive on any road in the world, they've got cars that can drive on any road in Austin or in San Francisco. And I believe they put some sort of like sensor system in, right, where they're actually providing extra context to make the whole process safer and easier. So that sort of...

connects with what you're saying, right? Like finding a more, like sort of a narrower use case in a way. It doesn't need to be able to drive anywhere, but if you can drive all around San Francisco, that's a really useful thing. I mean, indeed, I agree that we can add value using the advances we have in technology on the trajectory towards sort of like generally useful systems. I'm not familiar, too familiar with the Waymo solution. I believe they do actually rely heavily on maps and potentially even like infrastructure modifications, which are fine, right? It's not like either we change the vehicles or we change the environment or, you know, or we change the drivers and I don't think the humans are going to change that much, right? So like...

or, you know, around the vehicles. It is kind of interesting, right? Like, but there's still, even in North America where, you know, we're a first world country, pretty affluent generally, like the conditions vary drastically, right? Like Ann Arbor, where the University of Michigan, where I am right now, right? Like we actually, in the city of Ann Arbor, have dirt roads in the city limits. It is not a big city. And I think it's just some like local political, civil political thing that why there still are some dirt roads, but like, I don't know, there are channels to drive on. I have some, my kids have friends that live on some of these places or some of these roads and like, interesting, is it?

you know, like would a Waymo or Tesla be able to navigate this road? No lines, a lot of ruts and divots and cracks and so on, right? Like, would I, would I trust that? Or then take another example, like, you know, I was vacationing in Sicily a few summers ago, um, and, uh, right, beautiful place, but a lot of, we had to drive through a lot of back roads to actually get to the places where we were trying to get, go to. And like, it was hard for me to drive as an experienced driver. Um, so I think, you know, you can't boil the ocean as one of my early pictures today, right? Like we just have to make progress. And I think we are. Um, and it's just like, what, what, what frustrates me is just like,

kind of what you're alluding to, I guess, like the overreach of some of the commentary we see in the media. You know, we're not solving the vision problem yet. It's a hard problem, but we've made a lot of progress. And let's point, you know, give credit where credit's due. There's probably some like scenario here where we have lots of models. At some point we accumulate enough and some orchestration system that sits in like a meta layer that kind of picks different models for us depending on what we're trying to see and do. And that can give us this illusion that, you know, we've solved it, but it's really just this like cumulative mass of effective models in specific cases.

And I imagine that's probably similar to how our brain works, which is kind of where your tool comes into it. Like it's just lots of people building lots of models. Yeah, I think that's a fair way to look at it, right? Like, you know, there is a well-known theorem in machine learning called the no free lunch theorem. which basically says, you know, there's no one model that will be best in all scenarios. You know, you can always construct a scenario in which model one will fail to some other model that we may not know about, but, you know, whatever. So, but yet, I mean, given that said, right, there is on any given month, at the end of any given month, right, like there's a battery, like a closet of state-of-the-art models, model architectures that are generally well used or, you know, appreciated.

to perform generally pretty well these days, you know, like for different visual problems, like classification or like assigning one label or one or a set of labels to one image or a video or like detection, like finding an object that's balanced spatial boundaries in a frame and so on, right? There will be different models for those. But like, I mean, totally these days, even like a high schooler could go and like, you know, be given a problem for, I don't know, botany classification, right? Like go online, find an open source data set for different botany types and classes, pick any model out of a model zoo, like, you know, Torch Vision or our tool at Voxel51. We have a model zoo with like widely supported models.

put those two things together and, uh, and get decent performance, you know? And I think that the challenge is what, when is decent good enough to make, you know, social value or actually bring actual value to the world in a product. I mean, that's right. High school is not going to get there, but like, it's not hard to get 80% performance or 85% performance, you know, on your little open source toy data set that you used. Right. Um, and I think the challenge though, in practice, like the practical work of, of computer vision and visual AI is, is what's next, right? Like, you know, if I want to go deploy something, that's going to cook, cook it. Like, wouldn't it be great on if you guys have kids or anything like that, but like,

Wouldn't it be great to have something that cooks your meals and then cleans up after dinner? Like, that would be a godsend to most families. And, right, like, but if it breaks a plate every single night, you're not going to want it, right? Or if it, like, burns your food every other night, you're not going to want it, right? So, like, so I think of this as, like, how do we get visual AI to, like, five nines performance, right? Like, five nines comes from the old, like, network uptime or fault-tolerant community. But, like, basically, like, how do we get from 80% to 99.999%, right? Ultimately, that's what's needed to deploy anything. Like, a failure once a year. Like, if I drop a plate, I think we probably break.

I would say three to four glasses or plates in my house on accident a year. So that's like one failure every 80 days. I'm fine with that, right? So that's something like 99.99% performance for an automated system. However, like if you don't have the right tools to actually do the work, like to actually like figure out how to go from 80% to five nines, what do you do? Like the answer typically though is like,

just go get more data, right? Like your model, I mean, I shouldn't say that. Like the answer is like, what do I do next? Well, either my model's wrong or I don't have enough data. And clearly like the easier thing to do is probably just to like go and like look for more open source data or like basically pay someone to get some more data for you because changing a model is a lot of work, right? Like, you know, you have to prove it out and so on, right? So, but I mean, to me, I think that's like, that's a bad habit, right? Like if all you're doing is looking at the aggregate scores for your model and, you know, like the only solution you have or tool you have in your toolbox is to go get more data or go like get more annotations on your data.

I think that's a recipe for disaster. We need that mixology. Yeah, exactly. I read an article in the Wall Street Journal, I think it was, where somewhere between 70% and 90% of all generative AI projects are stuck in the okay performance phase. They never reach production. And I think, frankly, in my view, that's...

That is, that is the reason, right? Like, you know, like there's just no, there hasn't classically been a good set of solutions or like best practices around going from, okay, this is my performance. What do I do next? And like, in our view, in my view, like the real problem is understanding your data, right? Like, you know, that decent performance is about like the mean performance, you know, plus minus a couple Sigma, right? Like, so you get like typical performance and that's the 80%, but like what really matters is in like after that, like the outside of that space where you have like these corner cases and like the other nuances and that requires that you.

really get sort of get human with your data like actually pull it out visualize it query your data and like really understand what the what the challenges are with the data so you know is it a protocol problem is it not enough data for this type of corner case yeah is you know is it a light is it like a quality problem right is the lighting wrong in certain cases or is it like a glare in your view right and I think when you have the right tooling to do that and that's ultimately what what we build at Voxel 51 in the software that we call 51 is like exposing, basically, like making you be a scientist with your actual data. It's kind of like 51 is like a microscope for ML. Yeah, and taking the grind out of it, too. You want to show it to us? I'd love to see it. Yeah, let's take a look. I'm super curious. It's almost like a day in the life. I think a lot of people overthink it. I think people would be surprised. Yeah, sure. Our software is called 51, and 51 is open source, so you can go and pip install 51. If it wasn't wearing a 51 t-shirt, I'd be wearing the pip install 51 t-shirt, which we give away. I use it, so.

Right on. Yeah. Um, so pip install 51, or actually if you want to just play with it, you can just go to try.51.ai. This is a demo. That's just a similar site and you'll pull up this shared version of it. This is like the enterprise version of it that we sell as a product, but open source 51 is available. Um, you know, so why don't we, why don't we do that? So like, let's imagine that you are, let's see, let's imagine you are working in a, like you're an urban planner, right? And you need to get data for your team around like patterns of life for your city to understand.

like the next phase of road renovations and or park renovations, right? Like you need to know where people walk, where people go, maybe even where vehicles walk and go, right? So you, so let's, this is like a typical like problem we hear about like for smart cities of the future and so on. So I have a data set. It reminds me of like that open source project where they open source satellite images. So now you can.

You don't even need to put cameras in places. Exactly. And in fact, there's a great company called EarthCam, I believe that's their name, that exposes those publicly available webcams and so on, not satellite cameras, available for general viewing as well. Yeah, totally. So what you're seeing here in Voxel 51, software 51, right? So this is our general browser view. I would think of this kind of like a Google Drive for ML, if you will, right? Like you can share things, you can export things, clone your data sets. You have sort of different views, but let's just focus on the visual data, right?

The main part are all of the images in this one data set. And this is an image data set. And here you can sort images, videos, meshes, 3D meshes, endpoint clouds as well. And you can kind of drill down and like, you kind of browse and just get a sense for what's there in the data. But on the left side here are all of the attributes that we have for the data. So, you know, one thing we might do is get a sense for like, you know, imagine like you did this, you trained a model, and for some reason it's not doing as well as you wanted for the like people counting part, right? Ultimately for the urban planning case, you want to do some people counting. So let's see, we should probably try to figure out why we are, why we're failing here, right? So let's see, let's look at.

Let's look at the people classes only. So we can kind of drill into that here. It's a pretty big data set. So let's load the metadata for us. So let's go down and only select people classes. Oh, wait. See, now I'm already scared. As a user of this data set, I'm already alert because look, I didn't realize this. There's people and pedestrian classes.

So to me, I don't know, I already don't know what is the difference between those two classes. And from a machine learning perspective, that could be something like, you know, class leakage or, you know, like confusion in the protocol. So already I'm realizing, wait, I need to go revisit my annotation protocol so I can go and like understand what the difference here is. But what we want to do, like say there's a lot of false positives perhaps in this, right? So let's see, maybe we go down to our model case here.

And we can look at cases where there are false positives. That's FP. This is available per dataset and model run once you've processed the video through our evaluation system, right? So you click on false positives. So now that's changing the view. So it only shows images that the system believes have false positives. So let me see here. Is there one in here that we can go in where we can kind of drill into something that might be interesting? Well, kind of any of these probably are interesting, but let's like drill in. Let's drill into this one here.

So, all right, let's turn off the ground truth because that's kind of irrelevant. So this is the, that cyan was ground truth and the blue here is like the output of our best model that would say the current best model. So, I mean, in some sense, look how poorly the system's doing, right? Like they're, they misses all of these pedestrians. It thinks that that is a pedestrian. That is not a pedestrian. That is a garbage pail, I believe. It actually looks like a golf cart or like a mobile golf thing, but also look at all these things down here, right? So this would be a false positive. These are false negatives that we just didn't realize. But then if we go back, I don't know what we'll find if we turn off the best model and go back to the ground truth.

uh we look at like we missed all of these people back here as part of the annotation right so like so you're already finding problems in the data this there's a missed person down here as well for example like this took us three minutes to get there right like if you don't really understand like have the ability to visualize to kind of query in this way interactively and then make a plan based on your findings to do something else like in this case i want to go back and rework my protocol i want to have some notion of crowd of people and i want to have some notion of the ability like i can take detect my crowd of people and then have a different model that will regress the number of people in that crowd rather than trying to detect every single individual person. I also want to go and change my protocol so that more people that are farther away from the camera for my current problem are actual positive cases and detected. But we can even go a little deeper. And so what I'll do here is change it from whole image view to patches view on the ground truth.

So we can get a sense for other potential misses, perhaps, that we may have missed when actually doing the annotation for the protocol. And so some of these are definitely people, right? Like clearly person, person, person, person, right, person. But then I start to get a little bit worried, right? Like I need some more context for this, so I can't really tell. In this image, there really are two people. There are two people, two people, two people. So I'm able to see with just like this one click that if I'm going to be using this data for more than just person detection, my actual goal, like just to map it back to what we were saying like 15 minutes ago, right? Like my actual goal is person counting.

I really need to set up my protocol and my entire ML lifecycle for this visual AI problem to be really careful about how I annotate, right? This is actually a pair of people, not just one. You have to play capture all day. You definitely do, exactly. And the only way to do that is to look at your data. So there's some other ways of looking at the data as well. So let's look at an autonomous driving case. So this one is an open source data set from UC Berkeley called Berkeley Deep Drive. It has a lot of data around...

around visual scenes that you might imagine are used in an autonomous driving scenario it has detections it has classifications about what the scene is it also has what are called segmentations right semantic segmentation so we call them polylines in 51 but basically this blue label means it's a drivable area right so that's kind of where the car can go um and so on so you know, but look, there's a lot of data. I think there's 100,000 samples in this data set, maybe, sorry, 10,000 or 100,000 in this one that's loaded here, but we can scroll forever, right? This is like an infinite scroll. So what do we do, right? If I don't know where my model's failing, I'm not sure. So what I pulled up here on the right side is what we call an embeddings view. So I can just show you, like, generally when you load the app, you just get what we saw with the VisDrone example. But if you want to add other views to the app, like a Plotly representation, some other visual dashboard, we have a model evaluation panel.

other histograms over like which different views are there or embeddings and all that's extensible right so this is an open source toolkit you can just plug in your own stuff there and then we select an embeddings key and then we can split the view just like any id uh for ml or for anything really uh so then we get these clusters right so in this view on the right side here each sample is a corresponds to one image in the whole data set right so it's let me make this a little bit bigger So you can kind of see each sample is one image. Well, I'll show you how that's true in a case. So let's see. I can start to look at some of these groups together. I can lasso them and start to play. These clusters, these two clusters that came out of the data, they were just born out of the modeling problem that was given. So this model was designed to classify the content of the scene itself. And we noticed that, look, all of these images here that were in this cluster are night images. So all of a sudden, it wasn't trained to classify night versus day, but it was a natural separation of the content, which I think is really curious and really interesting.

But then you can even get to this and see something like, oh, wait, here's an outlier, right? So pretty far from this cluster. So probably a night scene. But what kind of night scene? Well, it looks to me like these are night scenes that might have glare in them. Oh, can I not visualize it? There we go. So like all of those have this really bright headlight, which does kind of change the type of processing that you need to do, right? So if you're in the world of Tesla where you want to use vision only, like cameras only, you're going to need to focus on situations where the visual glare will matter, right, to you importantly. Or you can imagine, let's see, what is over here in this cluster?

Oh, look, that's so cool. Like, so look, this is like a rain cluster over here, right? So each of these images seem to have like a lot of wetness on your, on your dashboard, right? You need to know how to do that. So like as an ML engineer and like a day in the life of when I encounter something like this, I'm like, oh wait, this is an opportunity. These, these are going to be challenging for me. And it doesn't matter like whether or not, like I can color these based on like, like a weather label or like the scene label or even the time of day. But if we just use like weather, it'll, it'll kind of like show me like, I know it's raining here, right? That's what, that's what we're seeing here. I can just turn these off and kind of get that sense that there's rain is scattered about, but this is a specific type of rain. I can kind of validate that.

What am I going to do next? What is my action from this as an ML scientist? I would go and basically use these data as query images to my large, like billion large data leak that's living out there in the cloud from all the miles we've ever driven and pull a huge amount of these samples down and then work with them in a new working set so that I can really see how to attack the problem of rain on the windshield, for example. And that kind of goes back to what we were saying, that idea that the query isn't language. You're not saying, show me all the images where it's raining. You're showing it examples of images that are raining and then saying, find other images that look like this.

And it's the language of querying is the image. And for those who, you know, that wasn't obvious. That's the right angle, right? Like I may not know what query or what question I want to ask of my data yet. The only way to get there is to do that type of analysis, either direct visualization or running like processing through embeddings on it. And then ultimately you get to understand where your corner cases are or other things. I thought just before, one last thing to show you guys. So 51 supports images, video as native types. It supported those since it was released in August 2020. Video is super interesting to me, especially as we think about the relationship between objects and different images. Yeah.

Yeah, totally. But also supports images, sorry, 3D meshes and 3D point clouds. So here's a point cloud example that you might also be interested in, like as an autonomous vehicle, like sort of company that allows you to have like, you know, the ego vehicles in this dead zone in the middle here, but you can have the same type of annotations and, you know, you can navigate around your space and detections and all that stuff for, you know, in the point cloud case. And the system supports not only these, this type of like, you know, partial depth point clouds, but also full 3D meshes as well to work with. We try to support all visual data. It is interesting to think that like...

Again, we find one of the biggest issues when you talked about failures. One of the biggest issues we find in failures is that people try to automate the way they do things, not the way they should do things. And that's just harder because you probably do it in a weird way. And so you're trying to teach a machine to do something in a weird way, in a very specific, weird way, instead of just letting it do it the right way. And I think when we think about machine seeing, you can't just look at it like having machines see how humans see because there's LiDAR, there's other infrared, there's things that machines can see that we can't. And utilizing that...

can make a whole difference, right? It's that don't anthropomorphize this technology. Yeah, 100%. I agree that the right engineering solution is to use the simplest problem that most effectively meets your performance goals. If that means radar, that means radar. If that means visual data, visual data, right?

for example, in-cabin imagery for vehicles. I mean, I don't think LIDAR or radar is that useful in cabin. Fine, put a camera in cabin, right, for drowsiness, for safety for the children, things like that. But for other problems, like why wouldn't we want to use the best solution for the task? But I think what you're saying is true on many different levels, right? Like, indeed, humans, like, I've talked to, I don't know, probably...

2,000, maybe more, 5,000 different machine learning engineers or scientists over the last five years building this software. As an open source tool, we have a lot of interaction with a lot of enterprise customers as well. Everyone wants the best out of the system in some sense. But one thing we've learned is each person has their way of working. Sometimes, depending on the type of problem they're working on, they want to work in a different way. And I think that natural flexibility is really critical because if I can work the way I want to work, then I will work faster and more effective. I'll do better work.

We have built 51 to be as flexible and adaptable to different workflows and different work styles as possible. That's why it's pluggable, extensible on the front end, in the back end. It's not like a SaaS solution where you go and create a web account and you can't ever see the code. It's open source. You can download it and work with it and things like that. It really is like Greenfield, at least my perspective, because it's not just the visual components. You've got a transcription potential aspect to it, which is audio. And then you have audio that isn't language-based.

you know glass breaking and door closing and there's just there's so much left to do there's so there's so much left to do and and in that exciting sort of explorative way you know to just try to capture um the information that exists in video uh and and utilize it to train machines it's like a Yeah, it's exciting and complicated. I totally agree. And I feel like I would be remiss if I didn't show you a video inside of 51. So let me just quickly do that as well. Since it is like, I mean, I've worked with video for a long time as an academic and now at Voxel. So let me just bring this back up and just show you a video, just to show you that we do support video. We have our own video player. In the video player, you can even like zoom in while the video is playing to a specific object and everything. Like pretty powerful analysis capabilities when it comes to the work that you might do with a video machine learning system. But I won't show too much there, but just to show how we support it. Yeah, and it's like,

To me, it's obvious, but to not everybody, you know, it might, to everybody might not be obvious, but the... Video is just a bunch of images. It's just a bunch of images and you analyze each one. And what's happening there is it's just so fast. Each image is being analyzed and classified and objects are being recognized in such a rapid pace that it looks like a box is moving, but it's really just recognizing it image after image. And it does. Yeah. So now we get into like, there's so much information in the still image. There's like a ton more in a video image because you have this temporal aspect to it, which is exciting.

So do you want a thought experiment? I don't know how much more time we have here, but I can give you a quick thought experiment. We'll wrap it up with that. Yeah. All right. So yes, the visual world is complex and it is significantly more complex than you want to believe it is, right? So imagine you have an image that is just 10 by 10 pixels, right? Like a very tiny image, right? Like a typical image we have, like HD is 1280 by 960, something like that, right? So in this 100 pixel image, now we can make it just as simple or more simple, right? Like imagine every pixel can only be black or white, on or off, right? So it's the binary image of 10 by 10.

Would you agree with me when I say that image cannot really describe that much? Maybe the digits in the characters of most alphabets, probably not Asian characters or Chinese characters, but it's not going to be able to describe like faces very well or actions or logos, right? Well, in that space of images, there are two to the 100 power possible images. Right. Mathematically, yeah. Mathematically, like that space. So two to the 100 is more than the number of atoms in the known universe.

Whoa. It's this weird setup. And now you're talking about video. Even just take two of those, right? Two of those layered. Two of the 200. That's crazy. It's two frames of 10 by 10 black and white video. It's crazy too when you think about all the different spectrums of vision that you can add. And then we're talking about adding layers of language into this. We had a conversation with Dr. Michael Greaves who came up with the digital twin concept. And we were talking a lot about how...

Sometimes people focus too much on fidelity, right? Like they want the highest fidelity digital twin. And we're talking about a realm where the fidelity could just be astronomical in terms of the layers of information that could be represented. So like a lot of the real work too is finding like the lower resolution slice that you can feed to the system so you can do what you want it to do. And it seems like through this conversation, yeah, like the way to do that is to find a good use case to build around so that you can find relevant information. Well, this was awesome. This was, yeah, I love vision. So I, and I agree with you. This is the year and, you know, we'll see more.

Absolutely. So much more to come. When I was a grad student, there was a slogan in my lab, too many pixels, so little time. And I think that's a good way to wrap up. That's good. Well, guys, this is great. Thanks, Jason.

Enjoyed it. Fantastic. Great. All right. Well, thanks for hanging out with us again here on Invisible Machines. Don't forget to like and subscribe wherever you get your podcasts. Always remember that you can watch this podcast, too, on our Invisible Machines YouTube channel. Head over to uxmag.com. We have a podcast page there where you can explore our back catalog and you can look at episodes centered around popular topics. Many, many thanks to our incredible production team here in the U.S. and in Ukraine. Elias Parker, Michael Lifinoff, Allison Harshberger, Ola Slessarenko, Vera Pakulkov, and Dorena Moskovchuk. Till next week. Owaki.

Frequently asked questions

How long is this episode of Invisible Machines podcast by UX Magazine?

This episode is 52 minutes long.

When was this Invisible Machines podcast by UX Magazine episode published?

This episode was published on December 6, 2024.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Invisible Machines podcast by UX Magazine episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.