Hello, this is Ted. And this is Brandon. And we are very far away from each other tonight. Often when we record the show, we're sitting next to each other in the same room.
And today, I don't know how far apart we are, maybe 3,000 miles? Is that how far Mars is from Earth? Because I'm sitting on Mars today, and you're in Georgia. Where are you at, actually?
I'm in Atlanta. Yeah, I got it. Atlanta. And I'm in Portland, which is where we normally sit on a couch next to each other, and it's cozy.
Are you sitting on the same couch that we normally sit on? No, I'm not. I'm sitting on my office, Jared. It's just not as fun.
OK. I tell you already, I like it better when you're here, Ted. Well, I like it better when I'm there, too. I'm on the 19th floor of a high-rise hotel, and looking out at the beautiful city of Atlanta.
So live in the dream. That's right. That's right. Yeah, so should we talk about some AI?
This is the concerning AI show for anybody that might have gotten on the wrong show. Yeah, it makes it sound so sophisticated. It's a show, not just a podcast. I like it.
Yeah, for those of you who are more into the truth-telling and advertising, it's two guys on a podcast. It's two guys talking to each other, and that's all it is. Although we do use very high-tech phones to record it. It's true.
It's true. It's true that we use the internets. The internets. Yeah.
Yeah. Double plus good. I understand we have some follow-up from the show where we were trying to understand this robot investing advertising campaign. You want to share that with us, Brandon?
Oh, yeah. I was just talking to Ben, like moments ago. And he said that he thinks the reason why those billboards are up there that say invest with us, because it won't be a robot managing your money, and then there's a picture of a robot in a suit. And I must say, very handsome robot.
That makes me want to invest with him. But his take on it was. The reason why that billboard exists is because most people associate robots with stupid and terrible service and cutting corners and trying to save money at the expense of providing value. For example, those phone trees when you call, press forward for this, and at any time you can press star to spend another half hour of your time trying to make your way through that whole thing.
Well, even Siri, that happens big promise is four years old and it's still not very good. I feel like it's worse. Well, is that crazy? It may or may not be worse, but I think people's experience with it is that you promise that this thing was going to be good and it really doesn't do what I wanted to do.
It mishears things and yeah, so I can understand where it's coming from and I think that actually is a really good explanation. It's sort of playing on the AI hype and saying, look, our expectations have been dashed so many times. Why would you think that a robot would be better than a human and anything? They could just entitle it.
The other people use robots prepare to be disappointed. That's right. And you can save money on using a robot, but why? Because your service will be terrible.
You'd be much better off just sticking with the humans. Which is funny because I never thought of that because to me I'm excited about promise. Well, that's because you're living in the future, Brandon. I don't know how to feel about that, but I'm going to feel optimistic and good at school.
So thanks for the compliment, Ted. You're welcome. I think this is a good lead in though to Google's news this week. So Google had their IO conference, which is their developer conference.
And it seemed like everything they were announcing had to do with deep learning or machine learning. It was all about AI. I don't think they're using the words AI anymore, but they're using deep learning and machine learning. And of course, AlphaGo is a Google project now and they're announcing all kinds of things.
And what did you say they call it? Google Home? Google Home, yeah, to compete with what? Amazon's Echo?
Which both of these are devices that sit in your house and you talk to them. Which is called a smart speaker. Yeah. Smart speaker, you can say play Cat Stevens, right?
And it just plays Cat Stevens. Which... Echo, get me the groceries and don't break the eggs. Maybe not quite that.
Well, yeah, but the Google one you're supposed to be able to say, you know, make me a reservation at eight at my favorite tie restaurant and it'll know what you mean. That's amazing. If that works. But it probably doesn't work.
It probably doesn't in the same way that robots are bad so far. And Ben's right about that. But if you think about who's likely to get this right, who's likely to actually be able to order you an Uber if you just speak out into the world? I mean, I guess Siri could probably do that for you now.
But who's likely to be able to do it over the din of your kids streaming from the other room? It's probably Google. They've got all the data, right? They've got all of our search data and they've got all the Google voice data.
I love Google voice data. Have you used Google Photos? No, I don't think so. Google Photos is amazing.
You say, I happen to use an iPhone and I have Google Photos app on there. It syncs with all of my photos. And so then I do a search and I say, show me photos of hugs. And it does.
You know? I mean, I sort of expected maybe now machine learning could show me photos of cats. But show me photos of hugs or show me photos of the beach. I find it pretty amazing.
Show me photos of stadiums. And it shows me both inside and outside photos of stadiums. Well, we got a lot of training data. Yes.
Yes. And that's why it seems to me like if anybody's going to crack this nut, it's going to be Google because they're able to tap into so much data. I have my headphones on to sort of protect from noise and there's an absolute deluge happening outside right now. I am hearing some noise through the connection here.
Yeah, it's amazing. I'm sorry. That's not really part of the program. Yeah.
So AI really worked. It could just edit this podcast and take that noise right out first. It would probably take the whole aside out. Right.
In real time. It's funny as we're talking about this echo. My brother-in-law works for Amazon. And so we've been having a few conversations about their drone, drones projects and stuff.
The whole get me my groceries and don't break the eggs this time. Is actually not maybe that far down the road. Are you serious? You know, his take on it is that it makes a lot of business sense for Amazon to do this drone project because they are at Christmas time going to soon.
They will be saturating all of the shipping, the other third party shippers and can't sell and get as much of the product as they need to if they stick with conventional mechanisms. So it's not just easier for them to get more trucks? Yeah. You know, I be in the kind of the enthusiast that I am.
I didn't go there. I was more like, yeah, awesome. And maybe the landing pad will roll out on its own and roll back. That's the sort of conversation we had not.
Yeah. I mean, there's some boring way that they could do this. I mean, Amazon does have this service now where, you know, if you need Legos, like right now, they'll bring up to you within a couple of hours. They do with all kinds of things.
They'll bring us our groceries. And grocery delivery services have been tried for more than a decade. And none of them are really caught on. But why is it that Amazon seems to be working where the others haven't?
It's kind of a strange thing. Yeah. Well, yeah, it's an exciting time here. So speaking of exciting times, maybe that makes for a decent segue.
You're the king of the segue in the last couple of weeks here, Brandon. OK, cool. Anytime we're king and Brandon are used in the same sentence, I feel good about that. So thank you.
All right. So what's on the other side of the segue? Well, exciting times. This podcast has sort of rekindled my interest in machine learning and AI.
And so I've been, you know, I wanted to do a little dabbling myself, get kind of involved in it again from after spending, you know, not really being involved for a decade or so. So I figure, hey, you know, it seems like some of our listeners are in that same category of want to do a little dabbling or even some of them are making it their career, you know, graduate students or startup people who have startups and one others, a nice representation. So I thought I could share a paper that I spent a few hours with this week that I think is pretty good. And that paper is in fact by Pedro Domingos from YouTube.
See that for those of you who don't live in the Northwest, that means the University of Washington in Seattle. And the paper is called A Few, Who S will Think to Know About Machine Learning. That's a very user friendly title I have to say. Well, this is the thing I love about this paper is there's kind of like theory and then there's practice.
Ah, that makes me think of another little comment. In theory, theory and practice are the same, but in practice, they're not. Nice. So this paper is kind of trying to introduce you to some of the things that are hard one through experience, some of the AI folklore, if you will, the black art associated with machine learning and AI.
This is a machine learning paper, but in my mind, those two things I kind of are kind of synonymous machine learning being the more respectable term and AI being the more hype cycle term. And since you've spent some time with this paper and I haven't yet, well, what makes sense for me to read the abstract and then you tell some interesting things about it that you gleaned. It sounds like a plan to me. All right.
Sounds good. Machine learning algorithms can figure out how to perform important tasks by generalizing from examples. This is often feasible and cost effective where manual programming is not as more data become a data or data. Boy, I stumbled over the word data.
Isn't that crazy? All right. Sorry, back to the abstract. As more data becomes available, more ambitious problems can be tackled.
As a result, machine learning is widely used in computer science and other fields. However, developing successful machine learning applications requires a substantial amount of quote, black art, unquote, that is hard to find in textbooks. This article summarizes 12 key lessons that machine learning researchers and practitioners have learned. These include pitfalls to avoid, important issues to focus on, and answers to common questions.
Yeah. So I just love this paper. I think if you're a doubler or practitioner, it's useful to check it out. And I think it's a great paper and I hope people will dig into it themselves.
So instead of going over all of it, I'm just going to pick two things that I like. Perfect. So the first, I think, is just a simple way of talking about learning. And it's a section two of the paper.
And the title of the paper is learning equals representation plus evaluation plus optimization. And when you first look at that, if you're kind of new to machine learning or something, it makes no sense. Or what does representation evaluation and optimization? What do all those parts mean?
Yeah. But they're really pretty accessible, I think. I'm going to kind of probably butcher these as I talk about them, but hopefully it can kind of make it clear a little bit. Okay.
Good. All right. So I'm going to ask you three to ask questions too, because you haven't read the paper and stuff. So, you know, please don't let me go on and on.
That's what I'm saying. Please don't let me go on and on. All right. Representation.
Representation. Yeah. So representation is simply how you describe the preface putting something else on the stack before we talk about representation. The learning problem that they talk about is classification and the entire paper is in the context of that particular problem.
But a lot of it's a well, very mature part of machine learning and a lot of other problems can kind of map onto it. And the issues are the same across them. So, so this is learning a classifier. And a classifier is given some input.
Tell me what type of thing it is. Is it class A, is it class B, is it class Z, what is this thing that you're showing? So that's classification. So how you learn something like that from test data.
Well, one of the key choices that you have to make as a machine learning or AI practitioner is how are you going to represent your classifiers. And the representation that you choose really constrains the kind of classifier you can learn. And as an example, imagine that the kind of classifier that you need to learn is actually the one that's the true classifier is actually the sculpture of the finger. I know that's that's kind of out there crazy.
I don't know if you've got to ask some questions because I'm getting crazy. I know I'm getting caught in the language of classifier. Yeah, that's why I'm going to the that's why I'm going to the finger and the sculpture instead of that. So let's say that in truth, we don't know what the true classifiers, we don't know what the what generates the label in the connection between the labels and the things that we're seeing.
We don't know what it is. But let's say that in real life, it was actually the sculpture of the thinker. Yeah, you mean you could give some data to the thinker and you would get some output? And you would get the right output every single time.
If you gave it a picture of an apple, it would say that as a picture of an apple? That's that right. As long as you have as your classifier, the sculpture of the thinker, but what if you choose, you're going to learn what the classifier is, but what if you choose as your representation paints? Yeah, paints.
Could you possibly get to the sculpture of the the thinker from paints, like a set of paints like Picasso or or somebody with you, right? Yeah, no. No, it doesn't work at all. So, let alone the sculpture of the thinker with the representation that is possible with with paint or words.
There's another thing, right? You can come up with Hamlet. You can come up with some sort of a play or a poem or a novel. Yeah, not unless the words are made out of stone that you somehow stack up and it looks like the thinker.
Yeah, it's not easy, right? The representation you choose constrains what can be represented. So in machine learning, representations include things like parameters to a function. A lot of it's a lot of it's familiar with like what a line is in two-dimensional space, right?
There's different formulations of that, but probably the most common is the slope intercept form y equals mx plus b, right? There's two parameters there. Yeah. So you want to learn a particular line that separates classes and says everything on this side of the line is class A, everything on the other side of the line is class B, something comes in, I just ask, is it on which side of the line is it on?
I know which it's going to be then what you're trying to learn is you're trying to learn where should you put that line in order to get as many things right as you can. So to separate the frogs from the bees. Yeah. If your representation is lines, then that's the only thing that you're going to be able to learn.
If the actual classes can't be separated by a line, then you're only going to do as well as a line can approximate that. Yeah. And if you're trying to classify all beaches versus stadiums versus hugs, you're going to have a hard time. Well, maybe, maybe not.
I mean, so the lines generalize to more than one, you know, more than two and three dimensions. It's hyperplane. It's basically drawing hyperplanes. You know, it's that's.
Yeah. So you're running into trouble. Sure. So maybe that's not the best choice of a representation to try to learn.
Okay. So maybe you want something that is closer to K nearest neighbors or so both of these things, functions like this, parametric functions and K nearest neighbors are kind of these discrete representations. Well, actually, I'm not sure about that. I'm going to stop there because I think I'm getting off into an area where I'm not talking right.
Okay. But the point is you have to choose your representation. You have to choose your classifier. What are you choosing?
You're choosing the how you're going to represent a classifier. Okay. You can use paint, you can use words, you can use marble. Yeah.
So you can use marble. What are you going to use to represent your classifier? Okay. So that's part one.
Yeah. Part two is evaluation. So let's say you come up with a classifier. How good is it?
Well, that's maybe a little easier to think about. It's a little straighter, a little more straightforward. You say, all right, give me all of my training data. And you know, I'm going to save a little part of that training data off as a test set.
And I'm going to train my representations and come up with, I don't know, maybe 50 representations that I think are pretty decently trained against that, that training data, that 90% of the data that I've trained against. Now let me take all 50 of those and score them against the other 10%. And that's going to tell me which of those is the best. And in fact, maybe there's five or six that are pretty close.
And the other is good. Okay. So if I'm, if I'm trying to do this Google Photos thing, I train with, you know, a million photos or a, you know, a hundred million photos and then I test on some that haven't been trained. I can't trust, you cannot test on the ones that you've trained against.
That is, that is one of the things that's also about this paper that is a classic, just a classic blunder, the most famous of which is never get involved in a land war in Asia. That's my favorite one. Yeah. So that's well known than that is never test using your training data.
All right. So that, so then I give it a picture that it hasn't seen. I give it a picture of, you know, Atlanta from the 19th floor of my hotel room and I say, well, I guess I can still label this still has the right class on it is still labeled, but it's not data it's ever seen before. That's the key thing.
Uh huh. So you can ask, you know, what do you think this is and you can know without having a person have to get involved. You can know whether it's right or wrong, right? I see.
Okay. Because it's already labeled training data. It's labeled data. Okay.
All right. So I'm going to go ahead and decide, you know, I know according to them, according to your fitness function or your objective function or your scoring function, those are kind of all names for the same thing. Um, which of those is doing well? Uh huh.
So classifier A says, Oh, that looks like Chicago and you say, Oh, that fails. That was bad. Yeah. But it did really well on these other 10 examples.
Right. So then you figure out what the, you know, which ones are doing the best in general across your your test set and, uh, yeah. Okay. So that's part two.
Okay. And part three, part three is optimization. All right. So how do you, uh, in general, the hypothesis space and by hypothesis space, I mean the space of all possible classifiers is just massive, like massive, massive, massive.
So how do you, given a few, uh, hypotheses, that is a few classifiers that you've scored, how do you get even better ones? You know, how do you, how do you find, well, let's say, uh, how do you use those to point you towards an even better classifier? Mm hmm. Cause that's learning is.
And is this the local maximum problem where you, well, some methods have, uh, get stuck on, on, on local optimum, for example, greedy algorithms that just, uh, you know, or like, like a gradient descent or gradient descent, either one, uh, where you're just, you look around and you say, uh, around where I am locally in the hypothesis space, that's the direction that things get better. Let's go that way until you get to the top of a hill. And then you're like, up, it's down everywhere I look, must be there. And there's no way to get to another place from there using that same optimization, you gotta use something else.
All right. So those are the three pieces right there. Representation. You put them together and you get learning.
Yeah. Does that make sense? Is that kind of clear issue? All right.
I'm sure it makes sense to people that know what they're talking about. It doesn't make sense to me. Yeah. Well, I'm disappointed by that, Ted.
I'll admit it. There's going to be lots of people who are saying like, I get it. Totally makes sense. I'm sure that lots of people listen to this or we smarter than me and know a lot more and they're all saying, oh, yeah, I got it.
I got it. Well, as long as you're sure of that, I'm positive. And some of them are probably saying, well, that was obvious. Why they even spend all that time talking about it.
So maybe if you're confused about something or you have a better way of talking about this formulation of learning equals representation plus evaluation plus optimization, let us know. We want to do better, especially me because, you know, it's my fault. All right. So the other section that I just adored out of this paper, um, yeah, is, uh, number eight.
And the title of that one is feature engineering is the key feature engineering is the key. All right. So let me give us some confusion. I mean, sorry, why don't you give us some, uh, tell us about that one, Brandon.
You must be really safe because you're clear in Atlanta and I can't reach over there to strangle. Okay, Brandon, go ahead. Go ahead. Tell us about feature engineering is the key.
Well, I'm going to keep the super high level because I've learned my lesson because in the way I've learned my lesson is through high, uh, through representation evaluation and optimization. Uh, no. So feature engineering is the key. This is really, really matches up with my experience that how you actually design your features, uh, just makes the difference between success and failure.
And this, yeah, you know, there's not a lot more to say than that other than there's a lot of knowledge that gets encoded in how you choose what the features are that go into your learning. And let me, uh, let me try to come up with an example of that. Yeah. I'd love to know what a feature is.
You know, I listened to another, Brandon, you know, last week or last show you, uh, you gave me some grief about being a promiscuous podcast, podcast, podcast listener. Well, it's been, you've gotten your share of grief too. But yeah, carry on. Yeah.
And so one of the shows I listened to is, uh, is called talking machines. And that's a, I love the name cause it's kind of like our name. It's can be read two different ways and they work both ways, but they are often talking about features and I never quite figured out what a feature is exactly. Well, let's talk about recognizing, uh, words in spoken language.
So speech recognition, not natural language, understanding, but speech recognition, just what are the words in what was just said. Okay. One type of feature could be the waveform itself split up into 20, you know, like let's say a five second snippet of the speech waveform digitized at, uh, you know, a certain frequency. Okay.
So let's call that 10,000, a 10,000 element vector that is just where the diaphragm on the microphone was, how displaced it was over that five second interval and shift forward a call it a 10 milliseconds and take the next one. There's a lot of overlap, obviously, right? Or you could say, that's it. That's, that's five seconds and then the next five seconds, I'm going to do the same thing to you in the next five seconds and there's no overlap.
Well, already just in that terrible choice for features that we're talking about right there, you've faced with a choice. Do you overlap the windows so that there's some redundancy in, in the features that come from, from moment to moment through time or do you, uh, do you have the non overlap, in which case, why did you choose where you drew the boundary or let's forget these absolute values altogether and instead let's encode the energy at various frequencies. So how much energy is at one kilohertz at this moment? How much energy is at 500 hertz, how much is at five kilohertz, how much is at each of these different spots?
Well, do we make it logarithmic? Do we make it linear so that we're, you know, how do we bend those up? How do you, so are you start to see what, how do you decide, you know, like, how do you decide what the features are that you're thing is going to learn from? Well, yeah, and it seems like if you choose features that are, uh, discard too much information, then you're going to have a lot, a real hard time learning.
And if you choose features that include way too much information, you're going to be overwhelmed with data and you may also have a hard time learning. Yeah, you'll actually learn irrelevant things and you won't be able to find the important stuff among it. Yes. It all just appears like noise.
Yeah. How do you pull out the actual data? This is one of the non-intuitive things about machine learning as well is that more input is not better. Mm.
Yeah. So you would think, hey, if I add another feature, it's at best is telling me nothing. I mean, at worst is telling me nothing and there's probably a little information in there. So you're thinking, Oh, if we add another feature, hey, great, where, you can't hurt us, not true, not true at all.
Oh, and you're saying it hurts us even more than just the extra processing power. Absolutely. To try to pay attention. It's way worse than that.
Yeah. It's actually in the way. OK. Yeah.
So the sentence here from this section is, this is typically where most of the effort in a machine learning project goes. It is often also one of the most interesting parts we're into, intuition, creativity and quote, black art are as important as the technical stuff. Yeah. That's been, that totally matches up with my experience as well.
So this is a wise paper. If you're interested in dabbling or if you're a practitioner yourself, you might pick up a few nuggets as well. I highly recommend this paper. I'll read the title one more time is or I will split in the notes.
Yeah. We'll put it in the show notes, which you can find it concerning.ai. Yeah. Cool.
Should we call it? Is that a show? Yeah. Just if you want to get in touch with us, go to Facebook and search for concerning AI.
We have a group there and you just ask to join and we'll add you and we love your comments. Clearly you're a lot smarter than I am. I don't know if we're smarter than Brandon, but we'll figure that out as we go. And it doesn't actually matter.
We're all learning together. You may be smarter than me, but you're not the king of segues. Just two. Ted is holding me behind the king of segues.
So it's true. So if you're looking for that title, don't bother. Don't bother. Just go find someone on the podcast.
Yep. Done. All right. All right.
So we'll see you next time. All right, Brandon. Thanks, Ted. Have fun in Atlanta.
OK. Have fun in Portland. This is it. Bye.
Bye.