Episode 014 – Spatially-aware Environments

What this episode covers

In this episode we’re going to look at IoT sensors and how that relates to spatial AI. We know we have computer vision, NeRFs and GaSPs and photogrammetry and (insert any acronym here) – but there’s so many more ways that we can make our local worlds much more intelligent through spatial sensing. Fortunately, we have some professors in the room. We have Mirek and Violet and William here with us. Violet, this is absolutely your domain. We’d love you to give us a bit of a better summary than mine of today’s topic and how it relates to Spatial AI – cheers! Violet 01:21Yeah, so I want to start out with a question to you all and then I’ll give you the question and I’m going to talk a little bit more before I let you answer, and then I’ll ask you the question again. I’m telling you that now so that I can get good answers from you all. So the question is, how does an environment become intelligent with spatial sensing? So what I mean by that is – how the environment itself becomes intelligent, not the agent, and not the individual robot. So we have a bunch of robots now in our home environment. Just think you have security camera or a smart nanny cam. Maybe you’ve got things like motion sensors and object recognition in that. Maybe have smart door locks. Maybe, you you smart lights. Maybe smart heaters and coolers if you a nest. Maybe your have Roomba. Maybe in the future we’re gonna have humanoid robots. Okay, so they all kind of live in their own world and they have their own reasoning and sensing. So something, you know, William has thought a lot about this as well, is: How do we make our environment intelligent? So not these individually acting agents, but how to we get them rather than being individually programmed? Violet 02:47Like right now I have a sensor or a light and I say, hey, go on every time it’s this time of day. But you never have any sort of coordinated actions across different devices. So in my classes, we play a lot with If this, then that (IFTTT). We use a little bit of, you know, some off the shelf open AI stuff. See what we can do to get different things talking to each other to do more coordinated action. But that’s all very still recipe driven – it’s orchestrated. It is not reasoning in any way. It’s not like you have some situation where, I don’t even know if this is a, this isn’t a good design prompt, but let’s say you had an AI that can control all those things in the home. Now an intruder comes into the house and maybe it’s just the camera that starts, but the lights start flickering to like divert them from the fact that you’re actually in the other room or, or that, yeah – Home Alone! Violet 03:52You know, it’s just like, there’s more, and I’m sure that’s like a terrible use case, but there is more capability of orchestrating programs that are multi-dimensional like that but require, that have multiple sensors and actuators from reasoning. Okay, so one more like little story I am going to share and then I’ll ask the question one more time: So I was recently talking to someone who worked as a data analytics something at Ford Motor Company and what was really interesting when talking them was it what really was impressed upon me is in every single one of these cars there are just so many sensors and actuators just in an individual car and same thing for our phones… …and it wasn’t until that moment that I really realized how much our world is just a fleet of so many sensors and agents and actuators out in the world — but they are all not aware of one another — they’re not orchestrated or coordinated as like a swarm. And the same in our homes even when I you know if you ever give presentation in a big room. If you just think about how many phones, individu