Bandwidth for ChangeLog is provided by Fastly. Learn more at Fastly.com. We move fast and fix things here at ChangeLog because of Rollbar. Check them out at Rollbar.com.
And we're hosted on Lino's servers. Head to lino.com slash ChangeLog. This episode is brought to you by DigitalOcean. They now have CPU-optimized droplets with dedicated hyperthreads from best in-class Intel CPUs for all your machine learning and batch processing needs.
You can easily spin up their one-click machine learning and AI application image. This gives you immediate access to Python 3, R, Jupyter Notebook, TensorFlow, Scikit, and PyTorch. Use our special link to get $100 credit for DigitalOcean and try it today for free. Head to do.co slash ChangeLog.
Once again, do.co slash ChangeLog. Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and snag with us around various topics of the show at ChangeLog.com slash community.
Follow us on Twitter, we're at PracticalAI.fm. And now onto the show. Kornick, thanks for joining me here at O'Reilly AI. It's great to have the chance to talk to you.
I know you just got out of your talk a little bit earlier. You talked about portability and performance in embedded deep learning. Can we have both? So I want to dig into that a little bit more later.
But first, I'd love to hear. I know you work and help lead the Movedius group at Intel. And I'd love for you to just kind of let the audience know what Movedius is, if they haven't heard about it, what you're doing, and what you're working on now. Yeah, hey, thanks, Daniel.
Yeah, good to talk. My name is Kornick. I lead kind of VPU architecture at Movedius as part of Intel. VPU for us is a kind of visual processing unit, and that's the kind of key engine we have in our kind of product line.
So yeah, I kind of lead that architecture. And at Movedius, we're very passionate about machine learning and computer vision at the edge. This is something we've been on for a long time, going back kind of five, six years, even before we were part of Intel, and we have kind of multiple products now in the field. And yeah, we've learned a lot, of course, as a result of all of that interaction with customers over the years.
And yeah, the goal of the talk this morning was to really kind of reflect back some of that knowledge, just in what have we learned about tuning neural networks for embedded silicon, and then also tuning embedded silicon for neural networks, right, to kind of just reflect back what some of the radiologies are when you go to take a network to the edge, what's kind of really required to make that run really, really well. Awesome, yeah. So just to kind of dig into that a little bit deeper, when you're talking about customers that are tuning neural networks for the edge on things like VPUs, which you mentioned, what are some of the kind of customer use cases around this, and people that have found a lot of value in going down that road? Yeah, sure.
So we have customers who are engaged heavily in things like digital security and kind of smart city type use cases. We're really making more intelligent cameras. That's one big use case. We've also shipped a lot of products on drones.
That's another use case. As well as a lot of things around, you know, robotics and smart devices and camera devices as well. So, you know, there's things like the Google Kits products that's on the market now that uses our kind of Myriad 2 silicon. And a lot of the DJI drones have used Myriad 2 silicon as well.
And they have things like you can wave at the drone using your hands to control it and infect your palm, and the drone can land on the palm of your hand. So really, really compelling use cases that have been enabled through our silicon and through the use of, I guess, both vision and AI kind of working hand-in-hand. Awesome, yeah. And just to kind of confirm that, I was actually at Go4Con last week and one of the keynotes, I think on the second day or something, they used a drone with a Myriad chip in it to do some facial recognition and all that is some cool stuff.
So let's kind of dive into a little bit more about what you talked about. Is there, in these types of use cases, where you're wanting to run your neural network in a drone or in a camera or whatever it is, explain a little bit the tension between kind of portability and performance that we've seen in the past and the state of it now. Yeah, sure. So I guess what we've seen is a lot of, you know, an archive where if you go to NIPs or these sort of conferences or CBP or like leading academic vision conferences, we'll kind of find there that people are, there's a lot of work being done to kind of optimize neural networks for things like kind of ImageNet or MSCOCO or kind of academic data sets.
And that's awesome in terms of pushing the envelope over the field and, you know, advancing the science and it's moving super fast, right? So then, you know, typically when embedded engineers will start off a problem, they've access to that sort of research and these sort of models and then they kind of want to do something that's going to work for them on their device, right? And one of the things they would find is a lot of the models that are available out there were tuned on ImageNet, which is great at recognizing, you know, a thousand classes of images, you know, we can differentiate one sort of whale from a different type of workplace and this sort of stuff, right? Very, very fine-grained classification.
Important problems, yeah. Yeah, not so much in the real world, right? We have different problems to solve. So then in the real world, we may care about, hey, like my robot wants to be able to recognize, you know, a hundred common objects found in the home or with this sort of, in the security camera, we want to be able to recognize these different types of objects that are happening.
Yeah, so different problems. And often those problems are simpler than the thousand class problems of ImageNet. So one of the things we were talking about this morning is using techniques like model printing and sparsification to, you know, if you're doing what we call domain transfers, you go from your thousand class problem, you know, say if you were taking ResNet-50 and you're now retraining that for your home robot which wants to recognize a hundred images, you'll find that you can get away with a much simpler network with a less representational capacity to solve that hundred image problem than the one you started off in the thousand image problem. So we were sharing some results in some techniques specifically around channel pruning, which is a very, very powerful technique when you are doing domain transfers with simpler problem domain and also looking at techniques like sparsification, which is introducing more zeros into a neural network because that's great in terms of on platforms that support, you know, memory compression of neural network models, it'll enable those models to run much more faster in bandwidth limited devices, such as those typically found on the edge.
Awesome. So in terms of, like, let's say that I'm working on, you know, I'm working on one of these robotics problems or whatever it is, and I'm using a neural network and I want to pursue some of these methods to kind of prune it down or optimize it for that setting or for that architecture. What's kind of the process and the barriers that I would face as of now going into that and what's kind of the state of the usability of these tools and that sort of thing? That's a great question.
Because for sure, we were presenting a lot of work this morning saying, hey, you know, we're able to take a network and do this sort of pruning and quantization and sparsification and then go from eight big weights to four big weights and this sort of stuff. But, you know, straight up today, pretty non-trivial to repeat the results that we were kind of showing this morning, right? To bridge that gap, you know, working, we now work in Intel as part of the AI products group. As part of the Intel AI products group, there's an open source project called Distiller.
It's one of the resources listed in my slide, I think on the final slide, and I believe they'll get posted to O'Reilly at some point. Yeah, we'll put them in the show links here as well. So yeah, there's a link to something called Distiller. And there, one of the things we're doing is, you know, if you went back maybe 12 months ago, you'd have found, oh, like, this is an awesome quantization technique that somebody published, you know, some grad student kind of put this, you know, publish a PyTorch fork or something with this, right?
And then here's something else that was available in TensorFlow for quantization, and here's something else that was available in a different framework. What we're doing is really kind of taking all of those techniques that are available in a fairly frequented way across the internet and trying to put them under one roof in a way that's kind of a little bit easier to access. And that was kind of the goal of the Distiller project, is really to show that. And it's an ongoing project at Intel within AIPG to have this kind of set of tools.
So they're available in PyTorch, and that's great because PyTorch can export to Onyx, which is then why we're available. But in addition to the work we're doing, it's entirely appropriate, though, to give a shout-out to the work of TensorFlow coaching we're doing. So there's under TensorFlow country, yeah, there's a bunch of useful tools there on both quantization and on pruning as well, right? And there's a pretty strong ecosystem there, also showing a variety of techniques.
Okay, yeah, so it is at least to a point where I could, you know, get a model off of some repository maybe in PyTorch or whatever and have some tooling that's publicly available to prune that down for certain architecture. Yeah, yeah. What about prepping the model for certain maybe specialized hardware? You mentioned like BPUs, and I know there's a lot of other people pursuing things around, of course, GPUs, but also FPGAs and other things.
What is kind of the state of the art? Are these kind of pruning methods and all that tied into that world, or is that something totally separate? Yeah, and that's also a good question, and it was one of the kind of the goals of the talk today was to show that, hey, you know, here's kind of four key techniques that you can use that will work well on any hardware, and on some hardware will work extra well, but if you employ these techniques, you're not going to hurt your model's ability to run across a whole range of silicon, right? So then those techniques specifically are kind of model creating sparsification, using a quantizing network to 8-bits, and then doing further quantization on weights to use kind of a lower big depth, right?
So if you employ those kind of four techniques, you will still have a model, if you take a model and you're represented in Onyx or in orientation, you'll still have a model that can work well on a wide variety of devices, but on some devices, it's going to work extra well, right? Because different silicon will have different abilities to run quantized models at varying degrees of acceleration, and also different silicon will have varying degrees of, let's say, weight compression and technology. And even in extreme cases, for sparsity, there's some silicon out there that can process sparse networks directly and in an accelerated fashion, right? So again, for a variety of silicon, you can employ these four techniques and get really, really good results across a range of silicon and even better results in some silicon.
So that was a core point, right? But to answer the second part of your question, in the final slide, we're making the point as well that, hey, if you set out to have a single network and you know the piece of silicon you're running on, absolutely, there's other techniques you can employ to really fit that particular piece of silicon as best as you can to really make this one network shine on this combination of this network and this silicon. And there's been some very interesting work published on that in the last couple of months, and it's a pretty hot research topic now is showing how to like using, maybe familiar with kind of auto-anal, right? So being able to use that type of techniques to kind of refine a model or to learn a model that works really, really well on a particular version of silicon with these types of performance and trade-offs.
So that's a pretty active area of research. It's pretty interesting. Awesome, awesome. And I know that one of the things that I've appreciated as kind of like as I'm hacking on things at home is that a lot of the stuff that you come out with through Movidius makes it really easy to experiment with neural networks on a lot of different types of devices through like the neural compute stick and other things.
I was wondering if you had any interesting stories or customer experiences that you've heard about of people enabling these sorts of things with these devices. Yeah, we really enjoyed the experience of launching the first version of the neural compute stick based on Maria 2. It was great to get out there and meet lots of developers. And also, you know, when we launched that, I guess it was kind of, we announced it some time before and we really launched NCPPR last year.
Yeah, it was great to see what everybody was doing but also to kind of show them hey, you know, AI at the edge is possible, right? If you go back 15 months and people, you know, where it was two years ago people really started AI with the cloud, right? So our first goal was to kind of break down those perceived barriers and for more people to be able to use AI and to see AI at the edge is possible, right? So that was our initial goal and it was a great experience.
Very enjoyable talking to all the developers. A couple of things we've seen. We've seen people use this. One of the software ambassadors for Intel used this to do a prototype kind of water filter.
So kind of taking a glass of a microscope putting that off to a camera into a Raspberry Pi with the video's neural compute stick connected and being able to show that you can actually use this to detect water impurities. So to have an entirely offline water impurity detection device that could be used, you know, effectively like on-premises at the edge with no cloud connection or anything like this. Super cool idea, right? And being able to show that's possible.
Equally with people putting them on a drone to detect sharks in the water also doing kind of prototype medical imaging to detect melanoma skin also kind of driven by image classification. So they're just a few things but there's been a lot of other fun projects posted on GitHub and I don't have a link to our model zoo site and example site but I can provide you with them for the blog page also. Awesome, yeah, we'll make sure that gets in our show notes for sure. Yeah, well, I appreciate you taking time again.
Kind of to wrap things up here I was wondering from your perspective since you've been working in this space for a while what can we look forward to over the next couple of years with performing AI at the edge? What are you excited about and what do you think we'll see over the next couple of years? Yeah, I think we're definitely going to see a lot more silicon to come available both from the various intel also from a bunch of competitors and I think it's going to be really interesting as inference silicon you know, there's a lot of metrics that business people will track like the number of like ops per watt we can deliver or the number of ops per dollar we can deliver and we'll expect both of those metrics to progress at a really, really fast pace over the next number of years and if I look at what people are able to do with the first version of the neural compute stick with the capabilities that has and while I can't just close product roadmaps with some visibility of the type of things we're going to see in terms of the volume of compute that various people can bring to market at much lower price points and much lower power points Jumping off into the abyss is kind of my skill and so I'm not saying that it's not scary I'm saying that perhaps my skill is just not being able to estimate how scary it will be New episodes premiere every other Wednesday Find the show at changelog.com slash AFK or wherever you listen to podcasts Well thanks for joining us Mike It's great to chat with you and meet you here at O'Reilly AI I've heard about Michelangelo this email platform that you guys have developed at Uber and I'd love to hear a little bit more about it but first give us a little background of who you are how you ended up where you are Yeah thanks happy to be here Yeah so I currently am the product lead for ML infrastructure at Uber and that encompasses a lot of things most notably the Michelangelo platform a little bit of background on me is I'm an electrical engineer by training and I have a school I work at Google and one of the kind of places I got my ML chops so to speak which is weird to say is I work on the ads team at Google specifically the ads auction group and I was the product manager for all the ML signals that go into the ads auction there so these really like real-time high-scale super-productionized ML systems that predict if you're going to click an ad and if this ad's going to be relevant and stuff like that so that's kind of like where I've learned how to do ML right and probably best in industry in terms of productionized machine learning and then about three years ago I joined Uber where we started the Michelangelo which is not named after me in any way That's a shame Yeah people get that question all the time and we started the Michelangelo platform which helps people which helps data scientists and engineers across the company build ML systems kind of prototype explore ML systems build them and then deploy them into production and serve predictions at scale Yeah so why if you're in a company that's trying to build up their AI presence within the company why would they need an ML platform why isn't like Jupyter Notebooks everywhere just fine for people One of the things so kind of like the state of Uber's ML stuff about three years ago was that a lot of people were trying to do that so there was a lot of people grad students learn how to build their ML models in their grad school classes and whatever and they have their own ways to do it everybody has their own I use R I use Python and what we saw was that people were either trying to productionize an R model an R runtime production at low latency which is just very challenging and people will cringe when they hear that today Secondly you would see teams that did have data scientists that did have engineer support they would build up these bespoke towers of infrastructure per use case basis that would tend to be less well built just because they had lower resources but a duplicate of a different piece of infrastructure that people would build to serve these models in production across all the different ML use cases the company has and then kind of the scariest is people just wouldn't get started at all because some people wouldn't have a way to get their models into production so we saw the opportunity to build a common platform to help people have a unified way to build models and to and this is the trickiest part put those same models that they prototype on into production to make those predictions and along the way bring a lot of data science best practices build into the system reproducibility common analyses and all that kind of versioning and all that kind of good stuff that is kind of like these data science best practices that aren't yet really well established and we have a lot of really well established software engineering best practices that everybody knows CITD and version control and stuff like that that stuff's not as well appreciated in the data science community and it's just because a lot of this work is new know and it's not like these guys don't understand the importance of it but it's just like the best processes and the best patterns for building this stuff and not yet we've not really converged on those yet so kind of spent a lot of effort to focus on what where we think this stuff is going to go and to help build the tools to enable like empower data scientists to kind of do the right thing from the beginning awesome so how many people are using Michelangelo at uber that's really hard to say i would say we probably have more than so this platform supports machine learning use cases across the company so everything from like fraud related things to pretty think how long it's going to take a car to get to you to even like uber eats like ranking dishes in the uber eats app all of the main ml stuff runs through this platform now but this is just like an interesting kind of platform development challenge is you know we have a lot of people who like kind of use it they're like hey i kind of want to build an ml thing and they dabble and explore a couple little models they want to make but maybe they never end up fully deploying that model to production and so it's kind of tricky to say like how many actual use cases do you like do you have on the system we know it's well over 100 but you know it's hard for us in the platform to say is this something that's this team is just using this as an experiment or is it like fully productionized and deployed across the whole company and that's just like an area that we just underinvested in a little bit but we think there's a lot more to do there is there um like as you've seen people start to use the system are there features of it that you thought that kind of surprised you in the sense of how people relied on them or things that people needed that you didn't expect that they would need or other things yeah that's a really good question and i've been reflecting on this a lot recently and the you know i'm the product manager's kind of job but but uh the thing that i would say that kind of has gotten disproportionate adoption given our maybe even like under investment into this where we could have we still could do a lot more in this space but our users just adopted this overwhelmingly and they love it as our feature store which is part of the platform and what that allows so you know common problems for for managing features related for ml workflows are that you have to clean your data and transform your data and combine it all and also historically for into a training data set so you can train your model but then once your model is created how do you do all those same transforms in the same way the same pre-processing to that data in real time when you deploy your model so there's kind of this like dual type of etl it happens in different compute environments it's really tricky and possibly on a variety of resources yeah and i mean we see a lot of like vendor solutions here but i feel like we don't see anybody really tackling that kind of stuff i think it's partially because it's not sexy all the work on that stuff and also because it's super hard to do properly and we've provided some nice ways for people to define their feature transforms to the platform and then be confident that that those transforms will happen consistently across both computer environments you know real time and offline but i think the other interesting thing that is let's take the uber eats world for example they probably have more than 10 different models that they use to predict to rank issues and whatever they do a lot of those models use the same kind of features and before this feature store data scientists didn't have any insight into hey other people that were working on similar problems what kind of feature pipelines that they built and then when this feature store came along now when a data scientist wants to start a new model they can just look look and see what features exist that are relevant for me let me just like start including or start off more and start my model exploration process with the x features that are most relevant to this problem from the beginning so there's a whole new element of collaboration visibility feature sharing that was previously not there i really don't see much solutions in that space in industry today either so i think that's a really promising area cool yeah i look forward to hearing more about that and definitely if you publish anything about that we'll be happy to post that on the show links here yeah the other thing i was curious about just from the fact that you know you mentioned before that the incentives for data scientists are kind of different and not always aligned with producing you know production ready models and all those things how do you how do you build up a team to build an ml platform where really you kind of need a software engineering experience to be able to build something that's production ready but you need the knowledge and the expertise around machine learning to be able to understand you know what to build so you're it's going to be relevant to the people you're building it for yeah so i think one of the nice things is that we've had a little bit of the leadership in our organization has been a relatively forward thinking to be willing to fund an ml platform the development of an ml platform much earlier than i think it's common in industry and that's allowed us to get it wrong a couple times before we got it right but we feel like we really got it really right now and there's like a tension between data scientists want this nimbleness and flexibility through other exploration prototyping stages and you know if you think of any production system it's super stable and and so how do you kind of accomplish both of those constraints it's a challenge and so what we some of the design philosophy that we're taking and we're you know this is always developing is we're trying to allow data scientists to work within our system using the tools that are most relevant for them so we'd love them to work in jupiter notebooks and write all their models the way they normally would and we can provide some helpful apis for them for example the feature store stuff to pull in their data so they don't have to reimplement a whole bunch of work that already exists in terms of like enterprise intelligence you know that's already done but after a certain point when the kind of prototyping stage is complete through if you think of like this machine learning lifecycle where it's like now i want to actually use this in production and maybe it doesn't mean you're going to launch into the whole company and it's going to be done with the project because you're like i want to experiment with this on live traffic we focus on making it relatively low activation energy to take your prototype and transform it into something that can go into these productionized well engineered hardened systems that we can be confident will be stable from a systems perspective and we still want to give data scientists the ability to monitor these models that are in production or not just you know systems issues like whatever applies to typical micro services but also like the data science monitoring how accurate is this model over time and any model drift stuff like that and so there's a story for data scientists throughout the life cycle and a story for engineers throughout the life cycle and the balance is and the challenge is like how do you balance between those at the different stages taking into account all the priorities for both stakeholders throughout awesome yeah that gives some great perspective well to kind of end things out here are there places online where people can find out more about what you guys have done and maybe also some things that you put out there that you might want to share yeah that's a good question we published a blog post about michelangelo and i think october 2017 and uh it's pretty easy if you just search michelangelo ml platform on uber on google rather you can find that and uh we publish a lot of other pieces about related ml work we've done and i think we're likely to in the near future open up to come on a little bit more michelangelo so stay tuned cool awesome we'll look forward to that well thanks for uh thanks for joining and uh enjoy the rest of the conference appreciate it all right thank you for tuning into this episode of practically i if you enjoy the show do us a favor go itunes give us a rating go in your podcast app and favorite it if you are on twitter or social network share it with a friend whatever you gotta do share the show with a friend if you enjoyed it and bandwidth for changelog is provided by fastly learn more at fastly.com and we get your airs before users do hear changelog because of rollbar check them out at rollbar.com slash changelog and we're hosted on lino cloud servers and at lino.com slash changelog check them out support the show this episode is hosted by daniel whitemack and chris benson editing is done by tim smith the music is by breakmaster cylinder and you can find more shows just like this at changelog.com when you go there pop up your email address get our weekly email keeping you up to date with the news and podcast developers in your inbox every single week thanks for tuning in we'll see you next week we'll see you next week