Skip to content
MLOps is alive and well episode artwork

Feb 7, 2023 56 min

MLOps is alive and well

from Practical AI Hosted by Daniel Whitenack and Chris Benson

Worlds are colliding! This week we join forces with the hosts of the MLOps.Community podcast to discuss all things machine learning operations. We talk about how the recent explosion of foundation models and generative models is influencing the world of MLOps, and we discuss related tooling, workflows, perceptions, etc. Sponsors: Fastly – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com Fly.io – The home of Changelog.com — Deploy your apps and databases close to your users. In minutes you can run your Ruby, Go, Node, Deno, Python, or Elixir app (and databases!) all over the world. No ops required. Learn more at fly.io/changelog and check out the speedrun in their docs. Changelog++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with extended episodes, make the ads disappear, and increment your audio quality with higher bitrate mp3s. Let’s do this! Featuring: Demetrios Brinkmann – X Mihail Eric – Website, GitHub, X Chris Benson – Website, GitHub, LinkedIn, X Daniel Whitenack – Website, GitHub, X Show Notes: MLOps.Community Upcoming Events: Register for upcoming webinars here!

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

MLOps is alive and well

0:00 56:52

Chapters

  1. 0:07 Welcome to Practical AI
  2. 0:42 Demetrios and Mihail
  3. 1:53 What is the MLOps Community?
  4. 5:13 Chris throws a hand grenade into the convo
  5. 9:44 MLOps vs DevOps when deploying
  6. 15:30 Ops vs experiment tracking
  7. 16:43 Where is MLOps going?
  8. 25:57 Is this even legal??
  9. 32:22 Sponsor: Changelog++
  10. 33:20 Is generative model bloat an issue?
  11. 36:34 MLOps' diverse uses
  12. 42:52 You are not Google
  13. 54:06 Wrap up
  14. 56:06 Outro
of matches

Transcript Auto-generated, may contain errors

Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

Well, welcome to a very special episode. I'll say this is an episode of a podcast, actually of two podcasts, Chris. This is a different kind of episode than we normally do because we're privileged to be joined by Dimitrios and Mikhail from the MLOps Community Podcast. Welcome, guys. Yeah, we're hiding over here in the shadows. In the MLOps Community. The shadows of the MLOps Community.

We don't really have a guest this week. We have four hosts is what we have. This will be interesting. There's a lot of chefs in the kitchen right now. Yeah, that's right. Which is obviously what listeners want more of. We're just going to answer each question with more questions. I don't know if we have the host with the most, but we have the most hosts. There we go. The real question is, who's going to ask the first question? Oh, I was going for it, but now you can. But technically, that was a question. I'm not sure. So maybe I'll answer your first question with my first question.

ML Ops and community. I love everything you all are doing with the community over there. Of course, the podcast, which we'll link to and the meetups and the content that you're putting out there. My sort of basic question, though, is like, we have heard ML Ops however many times on this podcast, Practical AI, and we've heard even conflicting. sort of statements about what that even is. So you're branded the MLOps community. So from your perspective, what does that mean? Who's in the community? How is it defined? All that stuff. So at a really high level, MLOps, it's an abbreviation of machine learning and operations. And one really quick, easier way to think about what MLOps does, it's fundamentally about the question of how do we take machine learning that you typically see in research and deliver that in the real world. So until relatively recently, I want to say the last few years, machine learning was starting to get really hot.

Maybe it really kicked off in 2012, 2013, is when we had this resurgence of deep learning that really picked up. And people got excited about image recognition systems, speech, ASR systems. And for a long period of time, a lot of machine learning, even cutting edge of machine learning, still happened in the lab, in the research lab, either at big institutions or even just at actual academic institutions. And so it was around, I want to say around the time that TensorFlow came out, when we started really getting this effort to standardize the processes of going from something that you see in research, new models, new AI systems, and try to actually put it in front of people. How do we get these systems to the point where they can start delivering real value to real people?

is pretty much a set of frameworks and tools and systems that people developed that are based a lot in software engineering as well as DevOps to make that translation process from research into production much easier. And so it's really just a set of principles and techniques and tools that a lot of the communities developed to make that translation possible. Yeah, I think around the time we started this, Chris, I think one of our feelings was, and now we're a good ways in, but I still feel this way, that it's not an easy leap to sort of just start learning, like, oh, I'm going to do the fast AI course or something. And then, like, okay, let's say you get a position. as a data scientist or like whatever the title is and like, okay, whatever's happening, like in that position, like there are overlaps between what you learned in like a fast AI or whatever, like certainly that's like wonderful stuff, but like, it is only like a very small component of a larger, like system of like practicalities that are not that easy to access and learn about. Totally.

They're still trying to standardize like the other pieces, depending on your use case, MLOps might be one thing or another thing to you. It's really not super clear right now. Like if you're using deep learning, that's one type of MLOps that you're doing. And if you're doing a recommender system and you like to play in the trees and the decision trees, those random forests, then it's a different type of MLOps that you're doing. So like, I think we should dive as we go here, we should dive into the weeds on what those are. But I actually want to throw a hand grenade into the conversation and say like DevOps or DevSecOps, depending on where you're coming from, you know, and like the differences and MLOps.

How does it integrate? Is it some of the same? This is the show where we can clarify all of these things between us here. We should just dive in. By DevOps, do you mean a Docker, Chris? Oh, yeah. What do you mean by DevOps? Something you put in the container? Well, once again, I mean, I don't want to put my definition. These are the things that have ops at the end that everyone is talking about. If you are out there in the world and you're doing models of various types, it might be deep learning, might not be deep learning, you know, Demetrius points, but, you know, it also integrates in with software. And so...

Where does one stop and the other start? You know, if we can start sorting through what we think these things are, I think that would be useful because you put a bunch of data scientists and software people in a room and they still today don't necessarily know what each other means when they're having these conversations. I aired a bit of my dirty laundry and my pet peeve of like people saying a Docker or Dockers. I deployed a Docker. But yeah, I think that's like the minds that you hit when you put a software engineer and a data scientist together. I think it touches on a really astute point, which is that the definitions are a little fuzzy.

To be perfectly honest, they're very fuzzy in terms of where does the domain of one begin and domain of another? How do we cross these territories? Where are the fiefdoms? Where are the boundaries set up? Fundamentally, MLOps has been and is a very interdisciplinary field. Maybe one of the easiest visuals that we like to discuss when we try and explain what MLOps is is imagine a three-circle Venn diagram where one is data science, the other is software engineering, and the third is DevOps. MLOps sits squarely right in that little...

intersection area of those three circles. Because a lot of the stuff that makes MLOps and the way people use it does touch on pretty much every single one of those three circles. There's modeling, right? There's fundamentally building models, your training systems. You have these data concerns that really falls out of the data science category. You have software engineering, right? Because you're trying to actually build out good systems that have to be engineered. You have to write code. It has to be well-tested, unit-tested code. And then the DevOps is really about how do you streamline that process of doing this once, putting it in a Docker, and then doing it multiple times. And then just repeating that over and over and over and doing it systematically so that, hey, I have 100% ideally uptime on the machine learning systems I put out into production.

And that's really the DevOps component. So really it borrows a little bit from every single one of those circles. I see what you did there. And just shout out to Docker Compose because you can do so much with that. Let's just not forget the unsung hero of this whole thing and give a huge shout out to Docker Compose. Or the Bash script. The true OG hero. Yeah, exactly. The unsung hero in reality. But there's, you know, like Mikhail was saying too, I think that it's not only one person that should be tasked with this MLOps.

title and a lot of times you can get yourself into a lot of trouble if you're trying to build out something that is going to productionize your machine learning and you expect your data scientists to be able to do everything or you have a very like naive sense of what it takes to actually productionize machine learning and not just like productionize it in a way where you get a model out but really make a process out of it and i really enjoy an idea which is MLOps is not just putting a model out there once it's the act of n plus one so MLOps is a machine learning model in production but not just one n plus one and that's when you get into how are the processes around this and what do we really need to be thinking of when we're productionizing our machine learning I have a question for you just straight off of that. Can you talk about, like when you're talking about getting that out in that N plus one model, how is MLOps in that capacity, like kind of late in the process there as you're kind of getting it out into wherever your environment, the cloud or wherever it is that you're putting the model, where does that differ from the DevOps world? Because it's easier to see to the front end because you're kind of starting with two different things that you're trying to put out there in the world. You know, in one case, you're starting by training a model of some type. In other case, you're working on software and all that. And you're, you know, you say, okay, we're ready to go. But somewhere down those pipelines, they start looking very similar in a lot of ways.

How do you look at that? How do you differentiate what those are? Do they merge? Do they not merge? How do you see it? So it's the fun one that we talk about a lot in the MLOps community, right? Like, isn't MLOps just DevOps? And the dirty secret, I think, is that, yes, it is just DevOps, but then you sprinkle a little bit of data on it. And so magically, it's not DevOps. And I think Mikhail can really like add a lot of great insight to this. But we were just talking about this probably like a few hours ago when we met. And it was very much like MLOps is more about software engineering than it is about the...

modeling piece i think like it's much easier for someone who knows about software engineering to get into mlops than it is for like a machine learning researcher to get into mlops and that's what i think we've seen over the years and we really have recognized in the past couple years that there's been a very clear engineering discipline that comes with mlops as opposed to in the beginning it was it wasn't so clear it was like oh data scientists should be doing this and now it's a little bit like hmm maybe it's not the easiest thing in the world to teach data scientists how to code You know, it's funny that you say that and I'll leave the company a name, but a few years ago I worked at a different place and I had a team for a first AI team and I had a team of data scientists and I was like the only one to come from kind of a software development background and everybody else was kind of straight data science. And that was what we found. It was a struggle a little bit to try to figure out how to create that ML ops line. And I was like, there's an entire engineering skill set that I think people come into it without realizing it up front. I think there's a, it's like an insight that they have, you know, that you just stated that it takes them a little while to realize that. And the interesting thing is,

when we look at how MLOps, the evolution of MLOps over the last few years, because it seems to have been born a little bit in the wrong way in that data scientists started learning these principles and these software engineering practices, right? So we had these data scientists like, oh yeah, like unit testing for data, that's super novel. And then the moment you had any person that came from DevOps or traditional software engineering start entering these conversations, they're like, what are you guys talking about? Like, this has been a thing for like decades. Like, what do you mean testing your model code? What do you mean testing like different parts of your software engineering stack? Like, what do you mean using a Docker? Like, what do you mean by like making it seem like this is some profound insight you guys have really found? And yeah, in a way there was, I don't want to say there was tension, but there was definitely like a...

WTF, what are you guys talking about? This is not as special as you guys make it seem. Yeah, you have the different sides of the spectrum, right? Where coming at it from this engineering point and they're like, this is just DevOps. And then you have the data scientists coming at it from the opposite end of the spectrum where they're saying like, wow, this is so much different than anything that's ever been out there before. And so we need to build a whole new discipline around it.

Yeah. Also, I think that the part of the confusion is like, just like everything else, like labeling and naming, like some of, I think there's been data scientists and researchers over time. They're like, yeah, this sounds great. Like machine learning plus ops. And what that ends up being is experiment tracking, right? Which actually is something that also is not new because like people have been doing this in high performance computing for like decades and decades. Right. But this sort of solution around, like, let's say weights and biases, ClearML, like I love these tools. We use ClearML. Like the value I get out of that is.

experimenting at scale, tracking experiments, making sure I know like what assets are what, like maybe logging data sets and like input output, all of that stuff. But that's all like experiment tracking and production of the model. I sort of tend to think about ML Ops as like everything after that, right? Like you have a model, that's all great. Like whatever happened before, even if you want like lineage about what data was input to your training, which output, what model, like that's all track. That's great. There's still some huge hurdles to get over in terms of like this model being called from within a software application that has like real users on the other end and like all of the potential implications around that. So in my mind, that sort of distinction is really confusing for people. And I don't know, maybe you can comment on this because it's maybe something that's come up on your podcast.

I'm not sure. Have you also seen that confusion of like ops versus experiment tracking? And the fact that a lot of data scientists might think they're, it's like, I think that word that you're saying doesn't mean what you think it means. Little princess bride in there? Yeah, exactly. I think that word doesn't. That's so classic. It reminds me of the meme too, where you have the little girl who's standing in front of the burning.

house and it's like worked fine in my jupyter notebook it's an ops problem now i love jupyter and all that but when i like when someone told me there was like an export to python script function and jupyter notebooks like it absolutely frightened me to like no end like i'm like why does this feature exist this is like i guess i understand like where it could have come from but it just like frightens me that maybe it's that evidence of like you know The disconnect between, oh, I wrote some code and did this thing between that and like software development and integration. I think it's evidence of that gap for sure. I've kind of been perusing your goods on your podcast, which is awesome. You know, various inflammatory titles such as Airflow Sucks for ML Ops. We learned from you guys, though. Let me just tell you that I learned from you on the title or the naming conventions when Luis was on and it was like.

MLOps doesn't exist or something like that, or MLOps is a lie. Oh, yeah. I was like, these guys went and did it. Oh, man, like my lifeblood. It's amazing that we went from that title to having the MLOps community on the podcast. That's why, yeah, I knew we needed to have a conversation because it was like, all right, I like it. And I think most people that if anyone knows me, they know that.

the most critical of MLOps is myself. And I feel like I kind of have to be. And that's why I love when Luis says that kind of stuff. And we went and we, uh, we grilled him hard when he came on our podcast too, because it was very much like, okay, what do you mean by that? And why is that? And I'm very much trying to see at every juncture, what is going on with MLOps? And is there something new? Like my new favorite thing to ask people is, is MLOps going to get Hadooped?

And that is like with chat GPT or with foundational models is MLOps going to become obsolete or it's going to become something that old legacy companies use. And so that's my big question right now. And what I love thinking about and those thought exercises that are going through my head. And so I almost feel like the pressure of needing to do that. By the way, you realize that there's a very, very current version of Hadoop, you know, that a certain large company has a big fear of right now. I think the big fear is instead of Hadoop, would it be Googled?

Yeah. Because of chat GVT. You know, you've seen all the stuff in the last few days about, you know, is the search algorithm going to be gone? Is it done for? I've got a bit of a contrarian stance on that. OK, go for it. When I see that coming up a lot, I'm just kind of like.

there's no way it's going to be replaced because first of all, I said it once and I'll say it again on this podcast, like my new year's resolution was to be as confident as chat GPT. I saw that. I saw that. Because let's be honest, like it's the amount of bullshit that it spews out is incredible. I want that kind of confidence when I'm spewing bullshit at people, you know, like give me that, please. And so that's the first thing, like, you don't know if you can trust it. But the other thing is there's a lot of stuff on there. Like I just see them as two completely separate uses and it's going to definitely take parts of what you would Google. And

that is for a good cause because a lot of the stuff that you do google right now it seems like the user experience isn't that good and so i would like to have a better experience when i'm doing that like for example if i just want to know the recipe of whatever, my favorite vegan bean burger. I can ask chat GPT and I don't have to get like this verbose, fully SEO optimized with a ton of ads and pop-ups and everything. That user experience is horrible. And so please like don't make me have to go through that. But I'm feeling some bro love here, by the way, as a vegan. I'm just saying thank you for using that example. All right, there we go. There we go. Nice. But yeah, then there's going to be a lot of other use cases where it just isn't the right medium, in my opinion. I don't know about you all, but that's my take on it. Just bringing it back to the ops perspective here. It's this weird like.

Because even with like fine-tuned or like transfer learned models where you have this like foundation model type of workflow, generally up until recently, the thought process was, okay, well, I'm still performing a task. And that task, like I can generate like a table of tests for that task, right? And like minimum functionality, I can put that as part of like my automation for when I release this model. Like however you do that, whether that's like, you know, whatever MLOps tools you want to use for that. And now you've sort of got this, like everybody's thinking about these generative models and like.

okay, well, if I'm completely open domain, how do I mitigate risk in those situations with these sort of open domain models? And I think there's, so I think there are ways and I think that some of the same workflows will apply, but I do think it does kind of tweak your mindset a little bit to think about some of these things. I don't know if that's something that you all have been thinking about with the kind of most recent wave of like generative models. It's interesting because they do change the paradigm quite a bit in a lot of ways. There was a hackathon. So I'm based in the Bay Area and there was a hackathon when it was like last week.

hosted by scale, where people were just hacking on different things related to AI, but a lot of them naturally, because generative AI is a hot thing to do, there were a lot of applications built around generative AI use cases. And one of them was actually someone tried to basically build an entire back end using nothing but LLM calls. They were basically just making calls and updating literally data structures in the back end fully using generative models. Naturally, this wasn't amazing, right? I mean, in a sense, it was a little bit hacky to try and update dictionaries or literally objects via these API calls using things like codecs and whatnot. But it does force you to ask this question of, well, right now, it's maybe not great. Maybe now it's a little bit brittle, but we see this out.

let's let you know imagine for a second what 10 years from now looks like i mean the 60s computers took an entire room right and there was only ibm that happened but here we are with them like literally on my wrist you know or in my pocket and so it's not unreasonable to expect that like some of the ways we've been doing these workflows are going to be touched by this right it's going to change how we operate with our software engineering systems and by extension by our machine learning systems i mean what can be done away with once we have the only interaction being like language to language style uh you know kind of translations i also wonder if there's like a meta layer here where like part of the question is how do we test and like set up the ops around large language models and generative models. I think that's one question that can be asked. And are those the same or are they different? I was going to ask that anyway. How do they relate? How are they not the same?

Sure. And how can one feed into the other? Right. Like what you were talking about just now, Mikhail, is that there's this idea of like, how could generative models help me with my ML ops? Right. Like, let's say I'm trying to put a model into production and I'm trying to test that well, or like I'm like searching through logs and other things and trying to parse that out. Like if I can ask an agent to like help me do some of those things, which to be honest, are fairly predictable if I've seen them before. Yeah, it's very meta. I know. But there's like the one side, like, how do I put these things into production? And as you say, Chris, like, are there differences with that? And the second place, like.

Well, can I actually like bring those around the other side and help them do my, have those models help me do my MLOps tasks, right? So I am 100% in the same boat as you. I know the CEO of you.com. He posted on Twitter, like, what is the best use of a large language model in your mind? Of course, I instantly thought like YAML fluency.

That would be incredible if it could do that. Like, let's just be honest, how much time would it save if it could just go and set up my Kubernetes cluster for me? And I've talked to a lot of smart people about that. And I think a lot of people are like, no. that's not going to happen. Like that's too far. But I also am a little bit like, yeah, well, there's a lot of other stuff that we thought wasn't going to happen and it's happening right now. So I don't think that's too far. Like that is my pain point. Like we are this far into Kubernetes and that still hurts. Like if you're not just using someone else's implementation, you're going to set up your own cluster. It's still painful. And like that, this is known stuff. It's just a pain in the butt. That's a great thing, right? I don't think that's too far at all. And I want it now.

yeah exactly maybe it's just like a fine tune away let's be honest maybe that's all it is and somebody needs to think about that i'm sure i'm not the only one that has thought about that and so but i'll let mr eric chime in too i know he has some strong thoughts and i'll like preempt his thoughts on this with His idea of an incredible app on top of generative models was stealing somebody else's IP. You want to tell us about what you've created, Mr. Eric? Oh, that sounds perfect. And by the way, just disclaimer, I'm not implicating myself in any of the following conversation. This is not hacking advice. Oh, goodness. This show just took a turn. Oh, boy. I didn't realize I was going to be basically implicating myself criminally by coming on this podcast. But we're going to edit this out, right? This is edited out. Sure, sure. We'll say that now. But we didn't sign a contract of any type, so.

And there are no intelligence agencies listening, I promise. Absolutely not, right. Absolutely not. Yeah, I mean, I guess I'll maybe spend a little bit of time talking about the use case that I think Demetrius is referring to, which is a little bit of a toy use case. We didn't do it because we were trying to get sued. That wasn't the goal, but hopefully it should spur a little bit of creativity. What a start. Well, okay, maybe like 10% we were trying to get sued. No, the use case to give an idea of what has become possible outside of, you know, yes, the really practical use cases, generating YAML.

generating unit tests. I think these are all, on a very serious note, these are things that are going to be possible. I actually don't see this being as too far off. And I don't, you know, not even like a decade. I see this being like the next few years. Being able to spin off things like that is just going to be totally feasible. You know, this is like language-based kind of generation, generative modeling.

The other one that we haven't really touched on as much, which is also really capturing people's interest, has been more of the image-based, vision-based, whether that's a single static image, or that's an entire video, or in some cases even audio, just different modalities besides pure text. And so one that has certainly become really interesting because of the rise of things like DALI, promoting AI, and then stability, stable diffusion, has these incredible photorealistic images in different styles that you can just prompt with literally human text, which has never been possible before, or at least not at this level of quality. And so one of the things that my co-founder actually, yeah, that we were working on was this little toy application that we called Rick and Mortify. And the basic use case was we were big fans of the Rick and Morty TV show. And, you know, we love it. I think it's super, like, it's great, great, great, great content. That did not stop a cease and desist letter from coming. There's a backstory here that I found out after we released the application, which if you guys are curious, I'll go into. But we were really trying to...

test this hypothesis of like, okay, you have these vision-based models that are incredible. You have these language-based models that are incredible. How can we like merge them to actually try and do something between, like kind of at the intersection of them two? And so what we came up with was like, well, can we personalize episodes of Rick and Morty? Like if I, as a super fan of the show, want to imagine a new episode, if I provided a premise, if I provided, you know, a set of characters that I wanted, maybe myself eventually, but let's just start with the basic characters that are in the show already. Rick, Morty, Summer, you know, Hoopybutthole, Mr. Meeseeks, like all these folks. That is an actual character name. I just want to point out for the audience that did not just make up a name to try and be profane. That is a character, Mr. Poopybutthole.

I'm not sure we've ever had that word on the show before, actually. I didn't know it was a word until the show. So we're all new to this game. What we essentially built out was you could come into this application. You could provide a premise of what you would like the episode to look like. So Rick and Morty go to the Practical AI podcast and have a great conversation about generative AI. And you pick your characters. And using a combination of vision-based generative systems, Stable Fusion, Dali, and then the GPT-3s, these kind of latest generation GPT models, we're able to generate not only the visuals, we're able to generate a script of effectively a storyboard for a new episode. This is just like a first use case.

we're literally getting new episodes. You know, maybe there's like five, 10 frames of this episode, but you're seeing some flavor of a plot with dialogue with accompanying visuals. And, you know, sure, it's a little bit rough, but extend this out to the future, right? Like what happens then? Like this has become literally a fully fledged episode. And that's where this could become, right? And so no cease and desist yet, but people did play with it. People had a lot of fun with it. We're not making any money on it. So, you know, don't worry, Dan and Justin, like it's, it's just for, it's fan art Friday. We're just trying to show our appreciation for the show. You raised something. It's kind of funny because when it just in day-to-day conversations about AI topics and like,

new model comes out and you see all the media stuff where people are just kind of bashing it and telling you all the problems with it and everything. And to your point, you just said this. That's why I bring it up. It's like, but think of what we can do with this tomorrow. Like today we have this and it's imperfect. And yesterday we didn't have any such thing. Think about if today we have the imperfect thing, tomorrow's going to be pretty amazing. And to your point there, like what you're doing there is like way out there, but it's not far from being very mainstream. So my question about that is like.

you have the training data and you have the base show that a human made, right? And so without that, like I'm trying to extrapolate 10 years in the future. And if it's just going to be us remaking a bunch of shows that we made back when we used to do it all by hand, or is it like, that's kind of my question there, right? I think that's a generative function right there. I don't think you have to start with where you were in the sense of part of that function at the very front end of that workflow is going to be generating different possibilities that are not directly linked to the training data and then like carrying it from there. I think it's a future entertainment.

Hello, friends. This is Jared here to tell you about Changelog++. Over the years, many of our most diehard listeners have asked us for ways they can support our work here at Changelog. We didn't have an answer for them for a long time, but finally, we created Changelog++, a membership you can join to directly support our work. As a thank you, we save you some time with an ad-free feed, sprinkle in bonuses like extended episodes, and give you first access to the new stuff we dream up. Learn all about it at changelog.com slash plus plus. You'll also find the link in your chapter data and show notes.

once again that's changelog.com slash plus plus check it out we'd love to have you with us What I think is kind of disconcerting is the right term. I don't know what the time scale on this or whatever, but like imagine like all these data sets around language models, especially are scraped from the internet. Right. And like a lot of these image data sets are scraped from the internet. Right. So there's a proliferation of these models and the internet is being filled with computer generated content. Right. So the next scraped versions of the internet, like the next common crawl, the next, whatever, like what proportion of that data set is coming out of generative AI models. Now, I think there's interesting things going on, of course, with like.

Detecting what is AI generated and what isn't I think I've seen like what open AI came out with for their own system and of course like you say Chris everybody has like criticisms of that already but there's like many other people exploring this as well like GPT zero and like other things that are exploring like how do we determine what is AI generated and not And so that's where like, I don't know what the implications of that are for these large scale data sets down the road, but that's sort of where my mind is going more so than, oh, should we be populating the internet with a lot of this generative art and other things, even if it's terrible? Well, I'm having a lot of fun with it right now, but there's sort of second order effects, I guess. No worries then, man. Yeah. Well, I'll be dead by that time. I don't know. Maybe. No, no, no, no. This will be soon. It's a really interesting thought experiment. And again, one that is closer than I think we believe. This idea of what happens when...

the bulk of these data sets are actually generated by systems. And I can see some positive effects and some that you would be like, OK, this might be a questionable downside. On the positive side, these models, like the generative ones, especially the language variety, are very good. I mean, grammatically correct. They're very semantically good in terms of what they output. Does this mean that now the bulk of the data that's being trained on is just going to be way higher quality writing than you would typically find in a typical Reddit post or something just in the deepest corners of the web? And then after training on data that's significantly better, you have this compounding effect of, well, now the data I'm training on is better, so the model is only even better at writing of some kind. And then we mix it in with some other more diverse writing, and then it's going to continue to compound itself in terms of quality.

The downside, I mean, one you could hypothesize is that like these models are only able to generate certain kinds of distributions of data, right? Only certain kinds of things they can write about or talk about. So now when you're just injecting all these training sets with like a very skewed distribution of topics, of ideas on these topics, et cetera, how do you ensure that actually you're still giving your model enough of a versatility in what it sees these next generations that are then trained on these data sets to ensure that it's still a general purpose model, right? Like what if 90% of the content that's put out there is just really bad marketing copy, you know, Facebook ads or something. You overfit. Yeah, it'll overfit exactly to that. And then what happens to these next generation systems? They actually might be hurt in the long run. Yeah, it's like we're in the golden era.

Yeah, I think that you can have an MO ops community, in my opinion, for a long time because there's such a wide variety of problems that people are still dealing with. So on this like very far end, we're talking about generative AI being part of like my day job as part of an international NGO. Like we're dealing with problems still like where, hey, there's no Internet in this place. And like we're running our model on like an Android tablet or like whatever. Like there's this range of like.

okay, what's the problem that happens when I scrape the whole internet again? All the way down to like, how do I run this small model on an Android tablet? And I don't actually see that changing for some time. Yes, the world's like changing a lot, but there's still like such a wide variety of issues. And I think fun challenges to like...

wrestle with around this concept of operations plus AI and machine learning. And so, yeah, it is sort of hard to define in that sort of way, but it's also really exciting because of the wide variety of things that you could be involved with. As a DevOps person or software engineer or data scientist, there's plenty of problems that have to do with running even a much older language model on your phone and other problems that have to do with various different scales or different modalities of data, all these things. Yeah. And that goes back to kind of in the pre-show that our listeners were not there for. We kind of talked a little bit about that level of diversity. Like, how do you guys see that? So, I mean, Daniel, you've kind of talked about almost an extreme case of kind of edge concept, you know, being your target, you know, and having this slow Android thing. And yet we're used to being just like gluttonously resourced in the cloud. You know, everything you could possibly want there, ramp it up, ramp it up. You got all the stuff you want there and you have this incredible diversity. So for you guys thinking about ML Ops, you know, so much and how do you deal with that? Like when you go from big company to one person struggling to work at all.

How does MLOps look when you're talking about diversity of use cases plus diversity of users that you're serving? How do you make it a thing? How do you keep it all together there? A few answers to this question, actually. So the first part of the diversity of use cases, and in a sense, we've gone a little bit backwards from something we said before, where on the one hand, we're like, oh, generative systems are just going to do it with all of this.

And then now we're back at like, well, but it's, there's like enough stuff that we need to solve that will probably be around. Both and. We're not quite there yet. Exactly, exactly. And to that point, I actually, I was actually asked this in a podcast once a few weeks ago where I was asked like, what do I believe will be the position, like the role that will exist 10 years from now? And I was asked like, will it be like a data scientist, a prompt engineer, an MLOps engineer, like where would I put my money? And I still answered MLOps engineer, like machine learning engineer, you know, like kind of in a similar category. Because I do believe that these same problems will persist whether or not they're for old school decision tree based models, discriminative models that we use, you know, maybe five, 10 years ago, even pre deep learning.

or these new GPT 500, whatever, will come later models, stable diffusion 10,000. You know what I mean? Like the same problems will persist. How do we operationalize it? How do we make it scalable? How do we keep uptime on models so people interact with this? These are all questions that MLops is fundamentally trying to solve and address. And whether or not you're using Kubernetes today or some prompt engineering-based Kubernetes tomorrow, or make it even more concrete, like Airflow today versus like an Airflow for prompt engineering, which is what people are actually developing today, the same sort of principles and the same concerns are going to apply.

And so I don't see that going away anytime soon. As long as there's a machine learning system, as long as, you know, we, I would assume all of us here in this room, in this virtual room, anyone who's listening here believes that AI is going to be the future. It's going to be here for decades. Then the same questions will still have to apply. And so that's like the first part I want to just like throw out there is there's, I'm here to say.

Whether or not it's Kubernetes or whatever comes after Kubernetes, we hope it comes soon. We hope it comes soon. Someone needs to get it soon. I have a tip there that I'll provide later. Yeah. I don't know if you all have seen what Eric Bernardson is doing with Modal and Modal Labs. I've been playing around with that recently. I've been pretty floored by it. But anyway, that's a whole other side topic and episode, which hopefully we can have soon. Totally. But I think to me, the second part of the question, which was how do different organizations think about maybe the MLops question and something that we do want to address here, which is it does depend. It really does depend in terms of the maturity of the organization and budget.

time, et cetera. These are all different axes that fundamentally define how a team or a business should think about its approach to MLOps. And there's different axes that we can go into exactly what they are, but open source versus not open source. You're going to stitch together a bunch of open source tools, or you're going to use SageMaker out of the box. How much money do you have? Can you get by with just spinning everything up on your own? All these different axes different organizations have to think through. And it becomes not a one-size-fits-all. It really is a function of all these different parameters to really tailor the right solution to the organization. And there you go. You didn't realize that Mr. Mikhail

Has a little consultant in his blood. Do I get to put another plug here? Another shameless plug? Is that what this is? As many as you want. Yeah, as many as you want. That is the most consulting answer you can possibly give to any question. It's like, well, for this rate, I can tell you more details on what comes after. Yeah, exactly. You want to go at it. But I think I just will add a quick piece to that, which is there was a time in the ML Ops community that like a week wouldn't go by in our Slack where someone would not share the you are not Google blog post.

It's like the amount of people that try and go at it and try and get that, especially because Google puts out so much great thought leadership on MLOps and they have the zero, level zero, level one, level two, or they have the like ML test score, all that stuff. And people think that straight out from zero to one, you need to be creating everything automated. It needs to be like the most. high-performant, bulletproof system that you can think of. And it was just setting up a lot of people for failure. And I think we've moved past that because I haven't seen the You Are Not Google blog post being shared as much in the community, which makes me think people recognize and they're a little bit more self-aware when they're trying to create their systems at their jobs. You don't think they're just off in the corner crying? Or they just don't think being Google is a good thing anymore, just objectively, given...

Yeah. I think, uh, I was listening to, I don't know if you all listened to, um, uh, Indie hackers podcast, but, uh, recent episode, like, I think they made this good point. They're talking about like, cause they talk about a lot about like bootstrap startups and stuff like that. And, uh, and one of the points they made was like, you know, whenever, um, like base camp was around or like starting up in that sort of thing, like they made such inflammatory statements about like, like taking venture capital is stupid. Like, why would you ever do like, there was a need for that voice to be in there. Right. That voice is still there with the founder, you know, but now like you hear less of that, I think, and it's sort of.

of like normalized and that like there's still different perspectives but there's not like as much like it's become more normalized to have these like more nuanced discussions. I think with MLOps there's sort of like you know we've all made our sort of inflammatory like podcast titles like whatever it was MLOps is dead or you know I forget what it was and so there is probably like still this we're kind of feeling out where like the normal is and where things settle down to and providing like a balanced perspective where yeah like you probably shouldn't be doing MLOps the same way Google is. but you also should probably be doing MLOps. It's just like, where on that spectrum do you land and what type of tools make sense for you? You know, in real life, I don't think it's you probably should not be. I think it's you can't do MLOps the way Google does. I mean, just from a resource standpoint, you know, most companies don't have that team available and that set of tools. And, you know, I like the fact that we're talking it in a more realistic context here, you know, for the vast majority of us out there that are not, you know, accessing the best of the best in all the categories.

It's not possible for many of us. We have to settle for something that's doable, kind of going to Demetrius' point. You know, you have to find that level that you can do it, you can sustain it. And yet it's still incredibly productive, even if it's not the Google version. Yeah. And one thing I'm fascinated by, just because there is almost this open source, you've got one person trying to hack something together and looking out there and seeing what's on the market that they can get for a price point between free and cheap. And then you've got the Googles that have built everything and have so much time and ability to do that.

In between those two points, you have a lot of companies that popped up and they popped up in the last like three to five years. And as Mikhail was saying, We came at MLOps from this data science perspective. And so I think in the beginning a lot, we're trying to cater to them. And then some were like, whoa, wait a minute, there's like platform engineers. And so they try to cater to them. And then you're seeing now, I almost wanted to like change my title for what I do on LinkedIn as I'm just going to be like the, I ride hype waves because MLOps was a complete hype wave. And we felt it. And especially like, I just got lucky because the pandemic hit. And right when it hit, I was working for a company that was trying to sell MLOps tools. And that company went out of business, but I was.

in the MLOps world. So I figured I would start this Slack community and then it took off, right? And so I was able to ride that hype wave and we really felt it over the last two, three years. And now it's like, all right, now there's this generative AI hype wave. And so if you talk to VCs and the ones that poured a boatload of cash into the MLOps market, they're now like, yeah, that's kind of not really that big of a deal anymore. We're not going to follow up on all those investments. What we're going to do is invest in the next AI tool. And so I love thinking about that and how now what is the new hype?

cycle of the AI and generative AI, large language models, like where's that going to play out? You know, we've been talking about it. There's a lot of potential out there, but where are we going to go with that? And is it like just a bunch of money pouring into it? And who knows what's actually going to happen? It's kind of funny when you say that. And it's like, there's an oversimplification, you know, when the market observes these things, you know, and you know, now we're past that, you know, that's been solved. You know, we're going to.

Web3, you know, whichever hype cycle it is. But I think when you look at what's happening, it takes all these things to have gotten, you know, right now we're saying generative because that's kind of the sexiest part of the puzzle, but it's not just generative. It's the fact that you have MLOps now that's matured a little bit and is supporting all that. You have the large language models that people like Daniel have been working on. If you have all the transformers that are well established, if you didn't have.

all of those components, the current hype that's now being attributed to generative would not be happening. And so it's once again an oversimplification by the market on the sexy piece, but it's that whole ecosystem that's evolved over the last few years that enabled all that to happen. And so we're seeing a really cool moment, definitely, but it's really the fusion of it all as opposed to just being generative. You couldn't have generative today without that. I think that's such a fantastic point. I just want to sit on that a little bit longer because when the GPT-3 paper came out, this was, I guess, toward the end of 2020, maybe middle to end of 2020.

This is like a long thing that I remember reading. And the most interesting, in my mind, achievement of that whole system was not even anything about modeling, right? Like the fundamentally, the building blocks of the AI architecture, if you want to call it that, was just we've been using these systems for years in some sense. It was really the fact that they had built this incredibly good software infrastructure to build out, train large systems at this scale over this many GPUs, at this latency, to make sure that, you know, the updates for the gradients could happen fast enough that this wasn't going to take 50 years to train. And they did that all, which fundamentally was really like an ML Ops challenge. I mean, at its core. You know, being able to architect that kind of system is the complexity. It's not the fact that there was this new scientific achievement that we really came up with. It was really like an engineering achievement. And so in that way, it totally was a layering on of things that we had seen before. But, you know, now they're like old hat, right? Like now we have open source repos that can literally approximate the same effect. Like deep speed from Microsoft has become really fast and very used. You can train large systems at that scale.

But the people that pioneered those systems really had to solve some ML Ops challenges in their own right. It's all right there. They haven't gone away. You know, it's still right there. It's a team effort for all of those different parts, constituent parts to put the whole together. But at this moment, the word generative is the front man, apparently, you know, as we record this today. It's pretty amazing that it's that ML Ops take that has put that all together and kept it together, kept them in lockstep so that you can create the new things of today. Yeah. I mean, another example of that is like all that we're hearing about, about language model chaining and that sort of thing.

this is like if you look at each step of that process right link chain or whatever it is those are things we've been talking about for a long time like there is an operational burden though to like chain these things together and make them work well like in concert right that's really like one of the fundamental things that we learned like when we talked about chat gpt right like the Like the language model existed, reinforcement learning existed, human feedback existed, like all of these things existed. Like the chaining of them together in a certain workflow is the real interesting piece. So I do think that's really exciting where a lot of those sort of chaining operations and like bringing things together in unique ways, that is a lot more possible because the tooling has gotten better. It doesn't like move us past MLOps. Actually, it just like we use MLOps slightly differently. And if anything, I think it becomes more crucial because there's so many moving pieces, right?

Shoulders of giants all the way through. You combine that with the Einstein turtles all the way down kind of thing, it's shoulders of giants all the way down. The one thing to add me to that is very often when I have described sort of the progression of MLOPS over time, like people often like to use the Gartner hype cycle, right, to describe trends in technology, right? You have kind of the initial hype, then the waiting, the kind of the trough of just disillusionment, and then this gradual climb upwards, right? I've often said that right now we're at that gradual climb upwards. Like the MLOps systems and the technology developed are like standardized and they're becoming more commonplace and people are using them. But the thing is that linear climb upward, while it is where a lot of value is extracted, is not really what a VC would look for. They would want this. They want something that's like exponential in growth. And so they're never going to ride a hype train that's like a linear climb upwards, even if that's where people are actually deriving value. They're like, well, what about, how do we get to the next thing in the next Gartner hype cycle where there's a next big inflection point up that we need to ride? Because that's where you get real 100X gains if, you know, that's the kind of thing they're looking for. You know, and ironically, the place where you find all that value there after that initial hype.

is from the trough of disillusionment. That's where you kind of go, now I understand. Now I know what we really need to do. And you get that really good growth after that. That's a great point. That's a great point. Yeah, this has been amazing. Yeah, this has been awesome. I have not.

needed to carry the conversation at all. So I appreciate you all doing the heavy lifting for me. Usually I am constantly thinking about that and you made it very easy today. So I appreciate that. This was a fun conversation. Yeah, this is awesome. We should do this more often. Yeah, we should. We should circle back and see if any of the things that we said actually are true next year, which maybe some of them I'm going for like 25%. If I get there, I'm good. Well, I didn't put my money on any of these predictions, so I don't really care. Right. You've already explained that you're not culpable. I'm not culpable. On the hook for anything. Rick and Mortify.

It's just a toy project. Don't worry about it. I come in peace. I keep waiting, you know, because we can see each other on video. I know it's an audio recording, but I keep waiting for the feds to bust in behind you on your screen there. You know, you know, the door goes flying back. It's a different sort of BBC kid moment. And then my mic just goes out and it's like, well, he'll be back probably. Maybe he'll be back. That's right. But then Demetrius would have to carry the conversation.

Yeah, that would not be good. I'd be kind of looking for you. Well, next year we'll see if this or any of the other things come true on Practical AI plus MLOps Community. Definitely check out the show notes for those Practical AI listeners out there. We're going to include links to all the great things going on in the MLOps Community, Slack channel, podcast, events, newsletter, all sorts of amazing stuff. And yeah, thanks guys. It's been awesome to have you on the show. Thank you. Pleasure.

Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Change Talk podcasts. Check out what they're up to at Fastly.com and Fly.io. And to our Beat Freaking residents, Breakmaster Cylinder, for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.

Frequently asked questions

How long is this episode of Practical AI?

This episode is 56 minutes long.

When was this Practical AI episode published?

This episode was published on February 7, 2023.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this Practical AI episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.