Hello, this is your podcast co-host, Daniel Whiteneck. I wanted to give a quick disclaimer before we jump into the content for this week. And that is that Chris and I recorded this episode a few weeks ago prior to when we had both seen the George Floyd video and all of the protests and calls for justice have happened in our country and really all around the world. The tone of this episode didn't really match what we thought would be appropriate for that particular week.
And so we held off on releasing the content. We also shifted focus a little bit and recorded a new episode on explainability and bias in AI. And we released that last week. We're gonna continue to try to bring a focus on explainability and bias and fairness in AI algorithms to the podcast.
We both think that that's incredibly important, especially as more governments and police forces start using things like facial recognition in some cases. But we also want to make sure that we keep getting AI content out there. And so we're gonna go ahead and release this episode. If you have questions about anything related to AI and policing and fairness and bias and all of those things, we'd love to have a discussion with you about that.
You can reach out to us anytime on Twitter or on our Slack channel or on our LinkedIn page. And we'd love to have those discussion. I'd love to hear what you were thinking about our content and about AI and fairness and bias in general. So please reach out and I hope that this episode is useful and beneficial for you.
Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at chang.com.com and follow us on Twitter at practical.ai. Welcome to another fully connected episode of Practical AI where Chris and I keep you fully connected with everything that's happening in the AI community.
We'll take some time to discuss some of the things in the latest AI news and we'll be digging into some learning resources to help you level up your machine learning game. I'm Daniel Whiteneck. I'm a data scientist with SIL International. And I'm joined as always by my co-host Chris Benson, who is a principal AI strategist at Lockheed Martin.
How you doing, Chris? How was the long weekend? I'm good. We're just coming out of Memorial Day weekend.
You get to get outside a little bit. I did. Of course, socially distanced, but outside. I did, and I was staying socially distanced, and I did that despite the fact that officially Georgia was the first state to open up.
Yeah, that's true. And it was something weird. It was like pool halls and tattoo parlors or something. Like everything about it was weird.
And so it's, yeah, I mean, it really, really was. I mean, if you look at the things and go, why did they choose that? Yeah. Once upon a time pool halls, but no more.
Not many a year. Yeah. I don't do tattoos, at least not yet. So maybe I should.
Yeah. It's been a while on the pool hall. I think my advisor, when I was at NCART, after my undergrad, I did a high performance computing internship at NCART and Boulder. And he was all into pool and had, like, he carried his case with his pool, a queue, I guess, is what it's called, around at the lab and everything.
So anyway, thanks for coming back to memories. We have a pool table in our basement. So usually that's, oh, wow. Sadly, you'd think that would make us quite good.
We're not. But recreational. Yeah. It's strictly recreational.
And usually it is accompanied with plenty of lubrication in the alcoholic sense. Oh, catch it. Yeah. It's kind of like we've had a couple of drinks.
We're like, yeah, let's go get a game of pool. And so yeah, that's what we do. Yeah. Anything else exciting over the weekend?
I had kind of a weird thing happen to me before we get started. So you know, I do all this animal rescue stuff. You know, we've talked about it. And so I get a call.
I'm probably the only person you know that has a snake hotline, so for our local area. So I go and help with, that's one of the things I do, is help out because people are afraid of venomous snakes. And other things, I get a weird call about a garter snake, which is non-venomous. But it turned out to be quite a big one, about three feet, which is very large for that, that was stuck in netting, like yard netting that would keep straw down.
And it was going to die because it was thoroughly entangled in it. And so I went to rescue without any gear. It was non-venomous. I wasn't too worried about it.
But the snake, I forgot the fact that the snake would be thinking I'm a hundred foot giant, relatively speaking. So I spent 15 minutes cutting it out of the thing it was trapped in. It constantly bit me over and over again. And so they have tiny little teeth, but just enough to get blood.
It doesn't really hurt. But it shoot on me for 15 minutes. So by the time I got him free, I looked like I was like an axe murder with blood running all down my hand. So it was really, you know, it didn't hurt, but I was just kind of going, this is just kind of a bizarre moment.
So that was my memorial day. But here we are talking about AI now that I've dragged listeners. Yeah, speaking of snakes, this is an AI podcast. And this is a fully connected episode.
So we can talk about whatever is on our mind in the AI world. I know there's been a variety of things going on, you know, some COVID things, some not COVID things. There's been conferences I clear was recently that around that was interesting that we just talked about the NVIDIA stuff, I think in our last episode, which was interesting. My NVIDIA Xavier NX did come to my house, which I ordered.
I'm excited to play with that, which I haven't yet. So maybe I'll give an update and a following episode about my experiences with that. But what else is on your mind? What's crossed your path in the AI world?
Well, before we even go there on your NX thing, we should ask listeners about things that they are doing, interesting projects at some point. And maybe you have an episode where we kind of talk about really interesting things. People are doing not really work related necessarily, but just things that are cool that they're doing, they're very creative on how I would love to hear what people out there are doing. Yeah, if you have an interesting side thing or interesting, unique project, let us know, reach out to us.
We'd love to talk about it and give me some inspiration. Yeah, we're on Slack and LinkedIn and Twitter and everywhere. You can reach us just about anywhere you might be. Yeah, for sure, for sure.
So what else is on your mind, Chris? Well, I'll tell you what I was thinking about when we knew that we were going to start planning this. I've had something on my mind for a while. And that is that we've started to allude to it in recent episodes this year.
And that is the fact that we're kind of getting to a turning point in the entire field of artificial intelligence while we have been so very focused on deep learning for the last few years. And pretty much the entire time we've been doing this podcast, we're starting to get to a point where a lot of the big advancements seem to have come out and we're seeing a lot of incremental stuff. And that's not to say we won't see some big advancements continue going forward, but it's kind of becoming a little bit mature in terms of there are models of various types, CNNs and NLP space. We've talked about, you know, Burt and all the others, GP2 and everything.
But we're kind of finding ourselves focusing heavily on these kind of the next version of the same model to some degree, you know, in the variants that are there. You know, more data, more performance, sort of. Yeah, exactly. And I've been thinking, I've been hearing more and more about like truly advancing the state of the art on the research side.
And that was really brought to the forefront a few months ago with the NURIPS Conference, when I always butcher his name, but Yoshua Ben-Gio, if I'm saying it correctly, and I apologize, if I'm not. But he did a keynote at the open of the conference kind of talking about advancing the state of the art from kind of where we are now to where we could go. And he had one way of looking at that. And then separately in the work that I do, at my employer, you know, we often have interactions and stuff with DARPA and I've been really becoming very aware of the way DARPA is looking at the future.
And I probably should, for those listeners who are not aware of it, I probably should real quick say what DARPA is. And that is DARPA is the Defense Advanced Research Projects Agency. And everyone calls it DARPA for short. Just rolls off the tongue.
That's right. It's quite a name. But it's been around for over 60 years. It is the original organization that invented the internet, which was the very first incarnation of the internet as we know it today, was four notes that were connected by the earliest version of the internet protocol.
And the idea was distributed computing across a wide area. These were all happened in the test case. They were all in the same facility. But the idea was that it could survive nuclear holocaust and all that by having nodes go down and still have the overall system, the network itself operating.
And so DARPA has always been incredibly cutting edge. And they're the ones that kind of lead at least the US government's military interests into the future. And typically on a horizon, it's like 15 to 20 years. So you'll see something happen.
It'll get developed over time and eventually it gets out there. And the internet's one really, that's the most famous example of it. So they are really looking forward from what we're and we can describe all this later. But the idea of what they call wave two that we're in now to wave three, which is kind of the next incarnation.
And between the NURIPS talk and the DARPA perspective and a bunch of others that I've read from other sources, I just thought it was time for us to start thinking about what the future looks like. So I know we're practical AI. In this context, it may not be the practical of pulling out your PyTorch and starting to work on a model on this episode. But I thought us talking about what we think it might mean to move into the future and where we might go.
And what we think of the current conversation might be a fun conversation to have. You up for it? Yeah, definitely. And I guess I can bring the practical side in here.
I will admit that with any sort of conversation about like AGI or general intelligence or like next things, I often come in with a good amount of cynicism and probably don't give it the respect that it's deserved in terms of the discussion. But I will say that some of the stuff that you're forming in and also the NURIPS sum, keynote, which we'll link to in our show notes. But it's definitely interesting. And there were elements of what he's talking about as far as system to AI and that sort of thing that are rooted in things that are being explored and experimented with and that sort of thing.
So it's not so much like Terminator scenario or like Singularity talk, but more kind of like you're saying pushing beyond the sort of very limited task oriented models to maybe something slightly differently or at least things that operate in a slightly different way. So yeah, I'll try to get off of my limited imagination a little bit to discuss things. That's funny. People throw around this sort of term AGI.
So your AI to AGI or artificial intelligence to artificial general intelligence. I was looking up some, while you were talking, I was looking up some definitions that people kind of have out there for AGI. Probably a lot of them are fairly ridiculous. Not surprising.
Open AI always talks about pursuing artificial general intelligence. Of course, open AI has been criticized in various ways, but I think that they're generally having good intentions in terms of how they're going about trying to push the state of the art and that sort of things. I agree. And we need that.
We need that. Yeah, definitely. That aspiration. Exactly.
So no organization is perfect. But I think they do a really good job at trying to get people thinking about new things and creative ways of going about things. And anyway, on their page, they're sort of about page. They talk about artificial general intelligence.
And what they mean by it is highly autonomous systems that outperform humans at most economically valuable work. I've seen other, so I saw other definitions about how like, or that sort of horizon of where we're shooting towards is intelligence that is able to learn across, you know, human tasks at human level, which right now we're at a very sort of specific task oriented model stage where like I have a really good machine translation model that translates, you know, English to French or something like that, right? Not in all cases, but generally it's very specific. Like that model is very specifically limited to that specific task.
And it's not going to do anything else, right? It's not going to sort of easily generalize to any sort of other human task other than translating English to French. And, you know, of course, there's multiple models and all that. I totally realize that now.
But I think in general, like the things that we focused on as a community have been those sort of task oriented things. I don't know if you have a thought on AGI and what that means. I do and another just to throw a couple of other buzzwords I hear associated with these as I answer this is the kind of narrow modern deep learning is often called weak AI, whereas the idea of AGI is strong. And so I think both you and I have had a habit of kind of scoffing over time a little bit at the AGI.
We'll admit it. Yeah, we do. And part of it comes from working in this field. And that is, you know, as working people in artificial intelligence, the sex appeal quickly, you know, drops away that it appears to be from the outside.
And you are grinding through working with data, and you are trying to create models and testing them and trying to fix it. And it's no different from any other type of engineering, you know, essentially, you're getting work done. And I know both of us are also software developers. And there's a lot in common, you know, it's a very bread and butter kind of thing, which most people don't think of AI as about.
But when you're working in the field, it really is. Yeah, I tell people like it's not like you stick your laptop in a corner and sprinkle some fairy dust over it and it becomes sentient and like starts learning, right? Essentially, what we're doing is a very dumb thing. We're doing trial and error, right?
Absolutely. A bunch of data, a bunch of examples for a specific task. And even though the models are very sophisticated in the definition, in the way that people have defined the model from sort of input data to output data, at the end of the day, it's a bunch of parameters that we're training by trial and error, in most cases, right? Not all cases.
Totally agree with that. And I think, and it's easy to lose sight of that because it is a very practical, hands-on, you know, scientific job that we're doing here in various forms. And so, you know, historically, we look at AGI, artificial generalized intelligence, and we kind of go, oh my gosh, you know, robots that are talking to us the way they do in Hollywood movies that so far away, we all know that because we know, we understand exactly what needs to happen and stuff to get there in the sense of, I shouldn't even say what needs to happen. We know that there's a lot that we don't know.
I should put it that way. There's so many things that we're just not there yet on. And so, it looks like a very distant horizon. But what we sometimes lose sight of as engineers working with the here and now of our tool set is that it is coming.
And we are actually very rapidly moving in that direction. There may be a long road from here to there, but we are moving along that road quite rapidly. And it's an evolutionary process. And there's a whole bunch of baby steps that get us from here to there.
And if you lose sight of the baby steps, you're like, oh my God, that science fiction, we're never gonna get there. But we are slowly working our way. And in the short time, you and I have been doing this podcast, you know, we're almost 200 episodes as we're having this one. We're now into the 90s.
And the field has changed dramatically in the time that you and I have been doing this, the show. And so, as we look at that and try to figure out where we're going, we're getting a call from luminaries now about turning that page into the next things. So that was really what I wanted to start talking a little bit about on this episode is just give us a little bit of context for the future and where we would go next. Change Log News is the best way to keep up with the ever-changing world of software.
We track, vlog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make changelog.com your daily destination or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers, it'll cost you exactly zero dollars and you can subscribe right now at changelog.com slash weekly. So one of the things that I enjoyed about some of the material that I was looking at when you pointed me to the NURPS talk and then I followed some additional links after that and was exploring things is this idea of system one versus system two thinking.
I think this is an idea that was developed in a book by leading economists thinking fast and slow. This idea of system one thinking, which are those things that we as humans think about or the tasks that we do that don't really require any sort of slow thinking or we don't have to slow down to figure it out. I have my coffee cup by my desk and I want to take a drink so I just pick up the cup and I take a drink. I don't have to think about like spend, take a moment, get out the chalkboard, write out how I'm going to pick up the coffee cup, write some papers and academic journals about my unique method for doing it and just kind of something that I just pick up.
I don't even have to expend, like I have to have the thought to pick it up and I have to do the motion but I don't have to slow down my life to think about how to do it right. Well in most cases maybe in system two thinking some of the words that were used as related to that in Yeshua's talk and in other articles were slow, logical, sequential, conscious, linguistic, algorithmic planning, reasoning. So this idea that there are those times in our lives as humans where hopefully most of us do think about more complicated things than picking up a coffee cup. We have to reason through certain problems to come up with a solution that is a sort of unique solution, maybe something we haven't experienced before.
It's expressed in terms of maybe linguistic elements or logical steps or sequence of things. And of course some of these words like the sequential side of things, the logical side of things, of course these are associated with some of these ideas that I mentioned open AI and others are exploring. If you think of a sequential series of steps that you have to put some logic into and execute with not that much feedback, we're starting to think about reinforcement learning which is a sequential decision making process where you get rewards from your environment and you actually can modify your environment in some cases. And so some of these words definitely get to those things that people are trying to push the boundaries of.
What's that idea of the different ways of thinking that resonates with you as well? It did and it's funny that book in particular, I've had it on my Kindle for a while and I need to dive into it knowing I keep hearing how good it is. You need to think slow about it. I need to think slow about the book.
There you go and I'll read the slow through as well. But we definitely are at a moment where we we've done amazing things in deep learning over the last few years. But I think we're all pretty aware that it is clearly nothing like we act as humans in terms of the way we process information. And you named a lot of that is the idea of something that requires your attention and your consciousness, both of which are core ingredients for system two, as we define what those mean, like with attention, it's the ability to focus on one or just a few elements at a time.
And we've seen that now crop up in a variety of deep learning algorithms at a kind of a basic level. It's definitely in some NLP areas and you're seeing it pop up in others as well based on those successes. And so, and the ability to kind of have what they're referring to as soft attention, which essentially allows you to focus on the things that you need, but they also evolve over time. And it's very much kind of encompassed by the idea of short term memory.
And Yoshua notes that attention is an internal action and it needs a learned attention policy. And so, all this kind of starts also feeding into consciousness in terms of if you think of consciousness not as a wishy washy thing, and I'm just horrified by the way as an aside at how poorly I'm doing explaining this after seeing this amazing keynote. You're doing beautiful. Oh my gosh, I'm taking a master's work and just killing it.
But he talks about the need for consciousness to be defined computationally. And so, not to be this kind of wishy washy, a theory, idea that we think of it as typically in our daily life, but something that you can identify algorithmically. The nature of consciousness is very short term in memory. It requires attention.
You look at something and the fact that we are going from moment to moment there, if you put that into a neuroscientific context, that that is what allows us to do what we're doing right now, to have this conversation and to do all the things that make us human in our daily lives. But we are approaching a time when we can with a very strict definition, potentially define what consciousness is and understand how attention and consciousness relate in a neuroscientific and human standpoint. And that is what is being codified as the idea of system to in a machine learning context where we're starting to track that. So, as I listened to that keynote a few months ago when he gave it in December, I was really struck by the fact that the science that we've put our careers into is really shooting along.
And we're not that far from certain areas that we may be able to computationally understand in a strict manner. And so, that's like I said, one of those first evolutionary steps that might take us toward that longer path toward AGI. Yeah, I was kind of struck when he started talking about consciousness and then brought in this idea of attention. And I think that it's certainly useful to think about sort of how to build a machine or intelligence that is conscious of the things that needs to be conscious of to do more complicated tasks like the system to sorts of learnings and that sort of thing.
So, personally, and this is partly my own opinions and partly my faith and my own values is that that sort of smaller scale consciousness, which can be quantified, is still vastly different than what makes a human a human. And that's another discussion that we can have. So, I think that there is human value that's separate from that sort of small scale consciousness that allows you to do these more complicated tasks. So, that's my own sort of thought on that.
But I do think it is really useful to define some measure, whenever we want to call that whether consciousness is a loaded term for some people or not, to define some sort of entity, which is that sort of entity that allows us to push machine intelligence to these sort of larger scale problems. I think that, like you say, practitioners who work in this every day, there's such a long road to like intelligence that would even generalize to many of these sort of system to tasks that I don't think we're in any danger of these sort of apocalyptic scenarios that people like to think about. But I think it is useful to think about what is that entity that helps us build up these more complicated tasks? I agree.
As a matter of fact, just as a note, the time horizon to get AGI has so many steps between here and there, and it will likely be long enough out, whatever that time length is. I can totally understand why people would doubt that we're ever going to get there. And a while back, I do this Atlanta Deep Learning Meetup. And a while back, we had dueling sessions where I was arguing toward why we would get to the concept of AGI.
Another friend of mine there took a different session and argued why that was just fantasy. And I still think I'm right on that note, surprising. But I can totally get why he would not think that that is a realistic thing. And that is because as we do what we do as practitioners, it is so many steps out.
It feels infinite. It feels like that thing that you're never going to reach. And so I'm sympathetic to people. I think that the work we're doing right now and all the work we've done in Deep Learning as different as that is from the neuroscience of a human brain that it's still incremental steps to get us there.
We are learning from what is working and what is not working. And we are learning from the unexpected side effects and things like that. And every little bit of that pushes us farther down that road. So I do think we're eventually going to get there.
And I think there's giant implications which we can talk about a little bit later in this episode about what that means and legal aspects and all that other stuff around it. Yeah. I think that one other example, which I think to talk about examples here to make things concrete is I work with our chief research officer's name is Gary Simons. And he's just a really fascinating guy.
He was actually the first linguist to take a computer into the field to do linguistics, which was basically a suitcase computer that he built himself. And then there was no word processor, right? So he programmed his own word processor to do linguistics in the field and pop a new giddy on this computer. He's been in the game a long time and has continually just innovated over time.
But I was talking to him and showing him some of these things when GPT-2 came out from OpenAI and the text generation capability of those large-scale language models and paragraphs of generated text that were logical and made sense and connected at certain points. And again, this is using this attention mechanism, which is popular in these models and was referenced in the keynote at NURIPS and all that. And he made the comments to me. He thought this sort of thing was impossible.
This sort of generation of text by a machine in a logical way was something beyond what he would see in his lifetime, for sure. And so I definitely think you're right that these sort of mechanisms and that comes about by thinking of, okay, what sort of mechanism do we need to push things further? Well, attention was developed and this self-attention and transformers in that push those systems to this new sort of level. And so I think that there's going to be continue to be those things that are developed and it requires a different way of thinking.
So yeah, it's a good point. We deserve a better internet and the Brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extension, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits.
We don't need them. Mix in ad and tracker blocking by default. Quick access to the Tor network for true private browsing and an opt-in reward system. So you can get paid to view privacy respecting ads, then turn around and use those rewards to support your favorite web creators like us.
Download Brave today using the link in the show notes and give dipping a try on Change.com. So I wanted to talk about the other thing that we introduced at the beginning and that was the DARPA perspective and I introduced DARPA for the purpose of making sure everybody was on board with what that was. But it's really interesting to see them kind of looking out. And so it's a great lens to read up on what DARPA is doing and what kinds of solicitations and announcements they're making out there in the public space because you may be able to infer kind of where things are going over the next decade or two but in doing that because that's their mission.
And it's kind of funny. Kind of the Hollywood-ish version of DARPA is still another like a spy agency or something. And you know, I was reading- Men in Black. Yeah, really.
I won't name the book. I'm not sure if I can remember it. But a friend of mine asked me to read a book and I stopped at the first chapter because the whole premise of the book was like DARPA, which was like the spy agency was sending agents out into the field to do nefarious things. I was like, I just can't take that.
It's just so far from now. DARPA is essentially, and this is almost it probably will offend them to dumb it down this much, but it's essentially a giant project management office. And you know, their mission is to look at what are the next great technologies that are going to lead us into the future. And many of those things, even though they may be looking with a military view, end up out in the general population just as the internet did.
And so it really does affect our daily life eventually. And so they have a three and a half billion dollar annual budget, which is a nice hefty little sum of cash to go try to figure out the future with. And another really cool thing that they do is that it's impossible to be a career DARPA manager. They do rotations.
And I forget what the exact time limit is. I think it's four years, might be three, but you never can bet your career on your DARPA performance. And the reason they do that is they want people to take risks. They want people to be willing to make big bets without it being something that will destroy their career.
So it's the safe place to do really revolutionary work. And that happens. And they essentially the entire organization, a bunch of project management offices and stuff. So the reason I say all that is that they have kind of it's been it's actually not terribly recent.
It was back in 2018. They released a PDF that's open to the public. Anybody can look it up called the three waves of AI and you can Google it and find it instantly. And it basically segregates the history as they see it of what AI is and is going toward.
And the three waves in if you think of the past one is what they call handcrafted knowledge. And I'll talk about that in a moment. The current one that we're in is statistical learning. And then the future that they're talking about the third wave of AI is called contextual adaptation.
And I think that the short quick version of each of those is that if you have a set of attributes about how sophisticated you're getting with your AI and in terms of perceiving learning abstracting and reasoning and how far each technology can go that we're at. And if you think like an example of first wave which was previous before we got to the current deep learning period would be like an expert system. And that's where you eventually capture a whole bunch of rules into a system and you have a flow that goes through them but it's a rule based system. So it's not inference.
It's not prediction in the way that we think of it today. It's not statistical. It's following a set of rules. And then we eventually got to this point that we're at now where we've been in this amazing deep learning revolution of recent years which they refer to as the second wave of AI.
And in particular things like perception and learning have come a long way of that. We haven't made so much progress in abstracting and reasoning that the idea is that we are learning and we have been even in the short time we've been doing this over the last few years we've come a very, very long way in terms of what we can do with those statistical capabilities. But as we started this episode with we're starting to get a little bit mature about where we've gotten to and we're starting to see many versions of the same models and technology coming out. So what DARPA is really looking for are organizations, academic, industry, within the military itself that are interested in this idea of contextual adaptation where you're essentially pushing all of those characteristics as far as you can and you're able to perceive form a contextual model and learn, abstract, reason and really all of those concepts that really lead eventually to the AGI concept that we've been talking about are seeing at least the next stage of realization.
There may be many waves after this potentially. But it's kind of taking us from where we are now in 2020 as we record this into the next some odd years of what this is likely to be. And so that framework, I keep coming back to that in my own professional life. And I think it's really important that we start recognizing that maybe we're seeing a fairly mature statistical learning marketplace, if you will, and people are able to put into production all these great deep learning models.
But a lot of the really cool research as we saw with that neuro up-screen note are now focused on what in this context would be third wave and what in that context was system to. Yeah, since we're talking a lot about generalization and also perceiving an environment and making sequential series of decisions, so solving these sort of longer or thinking slow type problems. It seems like a lot of the time when they're referring to these things, my mind seems to go to reinforcement learning. And I know that that's something obviously that opening eyes working on, but it also strikes me that reinforcement learning is not a new thing.
It's been around since whatever the 1950s, I think, even as an idea. So I wonder if that's a big piece of this sort of third wave or system to thinking or whatever, however it's framed, if that's so valuable, why has it not pushed forward more rapidly into kind of standard practice and practical implementations? And it's still playing Atari scenario and maybe not as practical as a lot of data scientists using it and that sort of thing. You think that's because of the models that are being used within the reinforcement learning framework, or is it because attention wasn't placed on it?
Or, well, actually, before we go too far, and I'll answer that, but we probably should know, you want to real quick define what reinforcement learning is for anyone out there that doesn't know. Yeah, so reinforcement learning is where you have an agent and an environment, and your agent executes what's called a policy to make actions in an environment. So if you imagine like trying to play a game, or if you imagine like you're trying to route a car through traffic from destination A to B, this is a scenario where you have a kind of goal or there's something you're trying to do, but it's not clear. There might be multiple sort of routes that get you there with the same sort of reward, and also the actions that you take actually influence your environment around you, right?
Like if I change lanes to my right, then you know, other people respond in traffic, right? So you've got this agent which is acting in an environment which it can actually influence. And so at each time step of reinforcement learning, when you're performing reinforcement learning, the agent makes an action based on a policy that tries to determine its future reward, and then the environment, you know, responds to that action with a reward in a next state of the environment, and you kind of loop through this cycle. And so this is used in robotics and other places, but you know, it seems like most data scientists I talked to that are even applying some of these more advanced AI models are not yet really thinking about reinforcement learning.
I guess my question is kind of getting towards like, if that's to become a more pillar of this sort of new way of thinking, is that just because we haven't been trying to solve those types of problems or the models applied within reinforcement learning weren't good enough yet? Well, it's only my opinion, but my sense of it is that we're still early days on reinforcement learning. And this current incarnation, by the way, that we talked to take any confusion away from people is called deep reinforcement learning, we're applying, you know, the traditional deep learning ideas, such as having, you know, a bunch of nodes that are connected and you have a, you know, like back propagation that is doing air control and applying that within this learning model, which we call reinforcement learning, which wasn't always had it originally had nothing to do with deep learning. So we took something that existed.
Yeah, it's independent of what type of model you use it. And we've retrofitted it with deep learning capabilities in that which gives it this statistical, you know, graph of possibilities, which basically extends what's possible with the model, makes it a lot more granular. And I think we're very early days. I think that like you and I have been in a unique position to see as part of this podcast, even all of the different uses that you could use deep learning for, you know, we had guests from Google that were using it in ways that I had not thought about.
Obviously, there's robotics, we just heard about, I recall there was a Pac-Man example where Pac-Man had been observed for a while and it was a Pac-Man or it was essentially reinventing the game from just observing it for a period of hours. But we're, I think, really early days. And I think it's a transitional technology, a transitional type of model that takes us a little bit forward. I don't think it lines up perfectly with like the deep learning versions of NLP and, you know, CNNs, which are kind of the deep learning version of machine vision.
So this is one of those evolutionary moments where we're having a technology that helps us take, you know, half a step forward from where we already were, and we'll keep building on it. So I think we're going to see a lot more happening in reinforcement learning in the years ahead, at least that's my expectation. Yeah, yeah, I definitely am looking forward to it. And I know it was at a event in Chicago before all everything got locked down.
And in that event, a guy was presenting about application of reinforcement learning in a marketing context to manage like how they executed their marketing campaigns in terms of, do we use this campaign combination with this campaign and how is that going to influence next week sales and then this campaign with that campaign and they're managing like seven different campaigns and over weeks and weeks. And this is a sort of sequential thing they have to manage and figure out and simulate. And I found it interesting. So I haven't heard a lot of that sort of filtering into those.
And that was a more, I guess, practical industry application event. And I was kind of surprised to see it filtering in there, which was really interesting. So yeah, there definitely are a good number of those use cases out there that people are exploring. And I think they are starting to filter into the real world.
You know, there's one other topic I know we're getting short on time here that I thought we should talk about. And I know you mentioned earlier that I believe we're at the, is it the two year anniversary of GDPR? Two years. Happy birthday.
So happy birthday, GDPR. And we're in this wave right now of AI ethics is a really, really big topic in the world. And what does that mean? And if you look at what we've just been talking about these past few minutes and the vast difference between kind of today's AI and tomorrow's AI and then the day after tomorrow's AGI.
And what are the context of each of those means? So we are still very much lagging in terms of applying legal and regulatory constraint around that because we're still trying to understand it ourselves. And it's a very fast moving target. Yeah, even for wave two or system one AI.
Absolutely. We are way behind even in system one or wave two, as you said. And so as that starts to evolve toward way of system two and wave three, we are still trying to figure out what that means and that fast change makes it even harder. So it's a really interesting, you know, and that has a lot to do with how these technologies are going to interact with us.
You know, it's the human actors that are being used side by side with all this working together as a system. And so it's going to be interesting in the years ahead to see how the evolution of human culture, be it laws, be it regulations, be it ethics starts to constrain or shape the future of these technologies as they evolve. It's not strictly a technical conversation, which we have a habit as engineers to fall back into. Yeah, for sure.
And I think that there's still so much that we have to explore on the technical side and solving, of course, the complex problems that are associated with these things. But if they aren't able to be applied in a way in the real world where things are regulated and people's data is their data, and you know, there's business problems to solve and non-technical people that are involved and all of those things, then you know, it's strictly an academic pursuit. So I think you are seeing a lot of that friction these days with regulation. I think as we close up, I wanted to, you know, just share, we always try to share some learning resources here in these fully connected episodes.
A couple that I just wanted to mention, of course, we'll link to the talks and such that we talked about. But that book that we mentioned was the title is Thinking Fast and Slow. So if you want to learn more about that kind of different ways of thinking system one and two, it's definitely something that I want to look into and read a bit. The other thing I was going to mention, we talked a bit about reinforcement learning and I think one kind of fun thing that you can do on the side and also learn about reinforcement learning is using OpenAI's gym to do some sort of simple reinforcement learning problems like the card and pull problem or, you know, racing a car around a little track or something.
It's really easy to use the OpenAI gym that creates these environments for reinforcement learning and TensorFlow has a tutorial with their TF agents framework that's built in TensorFlow to use OpenAI gym. PyTorch also has an example using the gym. So I would recommend if you're wanting to learn more about that subject, you can get hands on it in that way. So I have a learning resource of a very different type from usual within a particular reason why I'm recommending it.
So one of the things when you work in the defense industry and you also give, you know, a lot of conference talks like I do prior to the COVID-19 situation, you are often asked about, you know, the concern of AI and autonomy and their intersection with weapons and war and things like that. And it's a topic that scares probably most people that think about it, I would imagine. And so I get asked about that so often that I wanted to point everyone to a public document that is, I would argue a pretty good news thing that if that's really something that you're worried about in about the future, as we talk about AGI and all these things going forward and what happens in our world, there is a document called the Department of Defense, directive 3000.09 that's 300.09 autonomy and weapons systems. And what it does is actually quite an old document from 2012, but some really smart thinkers before the day of deep learning, we're kind of thinking through what are the implications and constraints that need to be applied to autonomy.
And so you can go out and Google this, it's publicly available from the Department of Defense. And if you read through it, you realize they have handled so many of the use cases before we ever got to this modern day and age, you know, this is legally binding. This is what governs us in the United States, at least, about how we do this stuff. And the first time I read that document, which was coming into this career, I meant, I had a sense of relief that smart people had come along before and been very good about thinking.
So if this is a topic, it's a little bit of a dry document, I'll confess, but it has some really interesting things in it in terms of how we were keeping people safe. And it was one of the very first things I had to learn. And I didn't think it to recommend it a while back, but I keep getting asked about that over and over. So if you want to understand how we think about that, it's a really good document and I suspect, despite the name of the document, you'll go to bed, sleeping a little sounder, knowing that this is the reality.
So I just thought I'd offer that. I know it's a little bit unusual selection. Awesome. Good pick.
And thanks for putting up with all my opinions and cynicism and optimism and, you know, all the emotions that went into today's episode. No worries. Yeah. If we're going to really be practical, we have to keep our eye on the future.
Yeah. And so sometimes we got to set TensorFlow and PyTorch aside and figure out what direction we're going. This was a fun talk today. Thanks for doing it.
Yeah, for sure. Talk to you later. Take care. Thank you for listening to Practical AI.
We appreciate your time and your attention. Word of mouth is the number one way people find new podcasts. If Practical AI has helped you on your AI journey, please do tell a friend. Hey, they'll thank you later.
Special thanks to Breakmaster Cylinder for the beats and to our awesome partners for their support. Shout out to Fastly, Linode and Robar. If you and your organization would benefit by speaking directly to the AI community, you should sponsor Practical AI. Podcast advertising is highly effective and we would love to work with you.
Head to changelog.com slash sponsor to learn more. That's all for now. We'll talk to you again next week.