And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io. Well, welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist at SIL International, and I'm not joined by Chris today, but we have a very exciting guest. Today we have with us Jay Alomar, who you might know from a bunch of great content out there, including the Illustrated Transformer and other articles also with Cohere. Welcome, Jay.
Thank you. Great to be here. Thank you for having me. Yeah, yeah. It was great to see you at EMNLP back in December. We tried to sit down and have a conversation then, but it didn't quite work out. But still really great to see you there. What were your impressions just generally about like the experience at EMNLP and like the NLP crowd gathering there and what they were talking about? It was like just after ChatGPT came out. So yeah, what was your impression and some takeaways? Yeah, these are incredible events. Like the amount of condensed download that you get in a conference like this of so much research. A lot of it you've caught up to a lot of it.
is like new and for you to explore. And also like a lot of people to meet that, you know, in my case, most of them I've never met before. A lot of them like I've connected with online, but then having the chance to sort of, yeah, go deep into some of these topics and think through with people what they're thinking about. And also like observing what themes are sort of coming in, let's say multiple poster presentations or workshops. So yeah, definitely awesome running into you. I'm glad we're catching up after that. I'm glad we got to shoot also one of your works in a video we've rolled out in. It's part of my excitement when I was there, I was like,
People who don't go to conferences, they never really get the sense of what happens inside. And so I'm glad that you were doing a bunch of these interviews during the conference to cover it for people who are not there. And I think there's room for a lot more sort of content like that. So yeah, it was definitely interesting and eye-opening and quite intense in terms of just how much information and social interaction is done in five days. Yeah, I love it. But at the end of each day, I'm so tired. I love being around people. I actually do enjoy that. But being an introvert, I'm absolutely exhausted afterwards. 100%. And it's like on a weekend. So it's like also you're...
know you're working through and then you know you don't have a weekend and so you're working through it so yeah definitely need a day or two after to sort of cool down Yeah. And you mentioned some of the content that you've been creating. I think many probably know your name or would have like recognize your blog and some of the blog posts, like I mentioned from like Illustrated Transformer. But then like you have so much after that as well. You're just mentioning some recent things you put out about stable diffusion. Where did you start developing this passion for like more of the educational side of this kind of state of the art AI stuff? And what's your perspective on that? And what from your perspective are you trying to kind of achieve with some of the things that you're putting out? Writing publicly and like learning publicly is maybe one of the greatest.
life hacks I've ever stumbled into. It started about eight years ago when I was just getting into machine learning. There was an event that happened, which was TensorFlow becoming open source. And I thought, OK, now it's a good time to jump into machine learning. I've been eyeing the field with interest. I've been seeing a lot of the deep learning developments, but I had no exposure to it previously. So I was not necessarily working in the industry, but I was extremely interested in it, and I wanted to learn. It's very easy for you to spend some time learning about a thing, but then three months in, six months in,
You need some sort of an artifact to point to that really gives you a sense of how much you're progressing in that field. And for me, some of these initial artifacts was writing a tutorial that sort of captures what I've learned in those two or three months. It also serves as me just writing down my notes for me to learn a concept a little bit better. So I did that a few times, and it developed in so many different ways. So it opened so many doors that pulled me closer to machine learning and helped me learn more and more and more. If I'm to understand a new paper that came out, I can read the paper and maybe understand or grasp, let's say, 20% of it. But if I'm to explain it, I really have to go in much deeper into it, or if you're going to implement it as well. Because if you're writing about it or explaining the work, you don't want to write something out there that is incorrect. So in a way, it forces the social circuits of my brain to exert that pressure for me to learn. And so that's been extremely useful.
so many doors. So through the blog, Udacity reached out to me and I worked with them for about two years, creating lessons on their deep learning and NLP nanodegrees, which include creating a bunch of videos and code examples, which I continue to do after. So that included, let's say, the YouTube channel that I've been creating some of these videos on that explain a bunch of these models. But yeah, it's a very fast moving field. And there's so many exciting things happening every couple of months. And when I get some time and I sort of have my eyes on something that is especially interesting, yeah, I would sit down and try to sort of do a write-up. And so...
Some of these milestones of, let's say, models have been, so yeah, the transformer was a major one, GPT-2 was a big one, GPT-3, BERT, retrieval augmented transformers, I've written about that, really interested in multimodality, so when a language model has the ability to read or look at images or generate images, so I have a write-up about one of these models. And then, yes, the latest has been image generation models, so stable diffusion, how these models work, and how text language models factor into their composition is also another fascinating component for me in that. Yeah, definitely one of the most useful things that I do. And just, I'm really excited to see people, you know, finding that content helpful in their journey. People love that visual aspect of, you know, don't assume that the reader knows everything about the topic. So a lot of people jump into the field, either, you know, they jump from computer vision to NLP, or they jump from being a Python backend developer to becoming. And so just thinking about some of the content pieces as being gentle on ramps for them is something on my mind as I write these. And I encourage people to write whatever they learn. And we need so much educational content out there.
most useful um life hack i can think of yeah i totally agree with you i think generally the podcast as we've developed it you know chris and i have developed it over time our focus is mostly on sort of like community and like bringing guests in and all that but the one sort of like selfish amazing thing that i found also is like having conversations about all these topics and occasionally chris and i do like just episodes him and i where we dive into a topic like we did one on chat gpt we did one on stable diffusion and others where like having to talk about these things like in a time window and try to also not say that many inaccuracies. It is a challenge, but it's been really useful for my own learning. That's for sure. Do you have any tips or sort of cheat codes for those that are like, Oh, I would really like to kind of get into this arena of creating educational content around technical topics around AI or NLP or whatever it is. Cause like you say, there's such a need for good content around these things out there more so than.
Like the research paper is great, but you know, followups to that and educational content around things is really useful. So any tips or, or things you'd like to share for those out there that are thinking about either blogs or videos or whatever it might be. Yeah, so the first one is start. Just pick a medium and start with it, whether it's audio, whether it's video, whether it's like, say, short tweets or tweet storms or a blog or, you know, TikTok video. So just pick something or experiment with a bunch of them until sort of you find the channel that you're comfortable with. But definitely what I see a lot of people do is waiting and waiting and waiting for them to have, let's say, their magnum opus and have that sort of masterwork be the first thing that they release, which is...
the wrong way to go about it. So you'll definitely improve a lot quicker. People will not judge you harshly for sharing something that you've learned out there. And a lot of people are held back by some sense of imposter syndrome because they're learning something. And there are so many experts out there about this thing. So they're like, what can I add to this conversation? And in fact, you can add a lot. You're seeing it with fresh eyes. You're seeing it at a different time where there are different resources. Even if the only thing that you're doing is putting together a curated list of resources that you found helpful.
is a useful thing in its own sort of regard. So definitely starting and with time finding your own voice and finding your own comfort and improving your craft is something that will come with practice. You'll definitely not be happy with your first output, but as long as you're putting it out, you're learning and you're sort of nudging your way closer into that place. To me, it's always useful to sort of, yeah, emulate your heroes. You know, what kind of content do you consume? What kind of content did you find helpful? What exactly about it was helpful to you? So in my case...
Coming into machine learning, I can identify a few writers and bloggers whose code or articles really were helpful to me. So Andrew Karpathy, for example, had this article about RNNs, the unreasonable effectiveness of RNNs. That was one of the first times where text generation really clicked for me and that it's finally really possible for software to generate text and that is somewhat coherent. I learned a lot from just the styles of Chris Ola, from Andrew Trask and sort of writing. a neural network tutorial with just 13 lines of Python. And that was really one of the first times where machine learning sort of really clicked for me. So yeah, that would be, I think, the second one. Like, see the sense of what connects with you. Try maybe to emulate it. Don't be shy of stealing. You know, the Beatles spent years and years just doing covers until they were comfortable with their own sort of sound. And so that would be the second one. But, you know, really comes down to just create, put out there, get some feedback, continue creating, and just move the cycle.
I remember at EMNLP when we were chatting a bit, one of the things you mentioned was how you were really enjoying, I forget the exact words you used, but it was something like making NLP boring or something about applied NLP or something like that. What did you mean by that? And what are your kind of passions around like your day-to-day work at this point? Yeah, so I mean, we're blessed to be working in a field that is very hot, very rapidly moving. And every now and then you see something that is super impressive beyond the capabilities of what you think software should be able to do.
Now, you can see a demo out there, or you can come across social media posts that show that model X does Y, which is mind-blowing. But then how reliably, really, is the model able to do this specific task? So a massive GPT model is able, let's say, to answer a question about programming. And you can see a demo or a screenshot online of it answering a question correctly. But then is this really reliable for you to build a product around for this specific use case? And the example here is that for some use cases it is, for others it's not. But if you're just seeing these demos, you have to really ask yourself, is this a cherry-picked example or is this reliably something that the model is able to do? And for this example, which is answering questions about programming, that is a large class of complex problems that models are currently not able to really be reliably good at.
as evidenced by Stack Overflow banning the posting of answers from GPT models. So that's one example of use cases where, yes, you will see flashy demos, but the reliability isn't there. But there are other use cases where the models can reliably generate sort of use cases. And so I work with and collaborate with a lot of developers and companies who are trying to roll out these models. And they would come in with a specific understanding, but then when they try to use it for a use case where the model is expected to reliably be correct, they find that the demos in the real world are a little different.
And so, yeah, there's a little bit of education and, let's say, a learning curve of how to best roll out these models and how to think about the various use case and how they differ, for example. And so, yeah, I'm really excited in these, let's say, playbooks of how to roll these models out reliably, which ones are ready to be used now for various use cases. Like one example here is, let's say... neural search and semantic search. And it's using embeddings to create search systems that go beyond just keyword search systems. So that's a very, I'd say, reliable and mature use case of AI. It's not part of, let's say, the generative AI hype out there, but it really should be. And so that's a little bit of the example that I feel we as an industry owe the people who are just catching up to these developments to have a discerning eye of, yes, there are a lot of exciting things here. Let's not get sort of...
tricked by the hype across everything and have some non-realistic expectations for some use cases that are still futuristic. The models are still able to do incredible things right now, but some of them will continue to be developed. Yeah, that's a little bit of how I think about it in the, I mean, that is definitely prefaced by saying that there is something really special happening here. And software being able to understand, quote unquote, understand and generate text coherently, you know, has a potential that is beyond what we can really fathom. So it's right to be excited about it. Then yeah, go deeper, be a little cautious, be discerning of cherry-picked examples versus reliable use cases.
when i started first consuming like your blog content and other things and thinking about transformers and then like going over to a notebook trying to do a few things at a certain point it was quite difficult to like overcome that barrier and start to integrate some of these things into your applications and we moved to a time when it's so easy to pull these models in now the problem is not so much like good tooling around this because it is fairly easy to do this at this point in terms of like an integration but more so around like the workflows and best practices and how to judge, like, is this model a good fit for this use case? Or how could I use this model? That sort of thing. Would you say that's also how you're seeing it? And I'm also wondering, you know, you mentioned talking to clients and customers and people doing, you know, building up new applications around these technologies. Have you seen a shift in terms of like most of those people being now like software engineers instead of, let's say, like data scientists or that sort of thing? And if so, how does that influence kind of how we think about building AI tooling and who we're building it for, maybe?
For the first point, definitely there's a lot of playbooks being created. So first it started out with prompt engineering. So how do you get the model to do some behavior that is useful for your use case? And they're definitely capable of doing it. But then that's really not enough of a competitive advantage for you to build a product around.
And so your playbook will need to include a bunch of other components. So can you have access to some proprietary data that others cannot have? And then one of the, let's say, also differentiating factors here is can you fine-tune your own models so that you just improve upon just the baseline model that is publicly accessible or even open source in some cases? How can you continue to improve the quality of your own fine-tunes and continue to collect the data that improves that model? And let's say observing the generations of your own model. If you're a company or a product that is building around a specific model, there are dynamics there that really affect the economics.
In image generation, for example, it's extremely useful to have a public gallery of your model's generations. So Midjourney has one of these where if you want to create a specific kind of image, you can give the model a few prompts and explore that generative space. But you're really helped by the model having a massive gallery of hundreds of thousands of examples that really nudge you towards the best way, direction, visual space, kind of prompt type or description of style. And so that's, let's say, another example of one element of a generative AI playbook. Keeping on top of the research is also another source of really good ideas for how to roll out a lot of these models. So in, you know, two, three years ago, there was a lot of these demos when, let's say, GPT-3 first came out of the model answering something factual and bringing, let's say, a specific fact, so asking a question in the model. And it's surprising when that happens, but then you know with time that that's not a necessarily reliable use case for these models up until now. But there is a way towards one where you're not just asking the generative model for the information stored.
inside of its parameters, but you aid it with a search component. There's a part of your system that goes and searches a database or the web and then retrieves relevant articles and presents those in the prompt to a generative model and in this process sort of improve what kind of a model. So that's another sort of element of, if you want to tackle this use case, don't just rely on the pre-trained model's parameters and information stored there, but let's say augment it with a retrieval component. And a bunch of companies are figuring out, yeah, a bunch of these. And I think as a community, we're also sort of working on that together. I'm working on sort of some write-ups too.
try to codify and make public some of this sort of gray knowledge that's coming together across generative AI use cases on both the text and image space. Definitely the last few months, to address the second part of your question, there's been an absolute explosion in the public's excitement of text generation models. So yes, ChatGPT is one of these models, and then the generative chat battles happening between Google and Bing. these product rollouts and sort of the waves that they're making throughout the industry are definitely putting text generation and language models at the tops of the minds of a lot of people. And a lot of developers are sort of trying to figure out, you know, how they can start using these models, how they can think about them. Yeah, it's been an absolutely tremendous couple of months to see that growth, which is not just developers, but it's like people in the street, just, you know, your parents coming up and saying, okay, we finally sort of get what you do, which has been absolutely surprising. Yeah, I really like how you highlighted this sort of like...
gray area or gray matter whatever you described around this like knowledge of how to put these various pieces together into a solution i guess it's sort of like solutioning with a certain set of like potential pathways forward with a state-of-the-art model so like used to like when there was sort of data science hype before all the ai hype it was like okay you need training data you're going to train a model that was sort of like the playbook now you're in the scenario where you're like okay well i have a pre-trained model am i going to do some type of fine tune so that's like something there am i going to focus on prompt engineering? Am I going to chain multiple things together? Am I going to do some retrieval and pull in external knowledge into that? So there's so many of these things where the chaining and the assembly of the solution is actually where the value comes out, which I think is a really good thing for people to think about. I'm wondering, because you're in this all the time. You're seeing new problems. And I know you've even shown me cool solutions you put together around topic modeling and labeling topic names with generative models and that sort of thing. When you come to something like that, how do you parse through?
like this intuition around, well, is this a situation where I'm like chaining multiple models together? Is this a situation where I really need to focus on prompt engineering? Is this a scenario where maybe I should be focused on fine tuning any sort of guidance or thoughts around like how to develop the intuition around that sort of solutioning? Maybe part of it is just experience, but any suggestions around that? Yeah. I'm in complete agreement to what you said. There is this.
frontier forming between where you train models and where you're just a user of models because of just the very high quality of pre-trained models and their ability to solve general problems. So they're general problem solving. The vast majority of my work is above this frontier. So not on the modeling layer. I would want to explore what is possible with pre-trained models. Generally, we have to fine tune. And that is aided by just the surprising thing of these large generative language models. They do few-shot generation really well.
So if you give them three or five examples of a type of generation or a style of generation that you want, they tend to catch on to that and give you something that is good enough for a lot of use cases. And then I know that fine-tuning would be the next step to that. So if that is not enough and if there's a use case where, okay, I have 500 labeled examples or 1,000, that's when I would try to reach to fine-tuning. But in context of just providing a few examples to the model really helps solve a lot of use cases. And so, yes, the solutioning aspect is where the headspace...
try to think about just using the APIs of these models and how to chain them together, how to think about fine-tuning and not fine-tuning, but let's say embeddings, and then using those embeddings for specific tasks for retrieval and then chaining that with generation. So this is a vastly underexplored and, let's say, new frontier because you can completely spend 40 years just learning the training layer and the various model architectures and the various ways to improve the data and fix the data. So we need people in all. heights of the stack. But then the engineers right now have this widely available sort of an underexplored area of what can you do with pre-trained models. A lot of them serve via APIs, and you can definitely do a lot. Could you just give people maybe who aren't familiar with Cohere some general intro to what Cohere is trying to do and what they offer? Absolutely, yes. Glad to hear that. Cohere offers an API for large language models. And the goal there is to make using language models easier for every developer or company out there without thinking about hiring an army of people to train large transformers.
Our founders came out of Google Brain, and one of them was the co-author of the Transformers paper. And we have teams that are focused on training, let's say, two kinds of models. So the Generation Large GPT models, we train, let's say, both of these families in-house and continue to develop and improve them. And the other family is text embedding models. And these are the ones that can power use cases like neural search, semantic search, text classification. So if you want to classify messages by topic or by sentiment, they're very sort of capable in that. And the latest release has been the multilingual embedding model that supports over 100 languages. So if you want to do semantic search or neural search, you don't have to build 100 different pipelines for each language that does stemming and very language-specific pipelines. You can just throw it all at the embedding model and just retrieve.
the best results. And so that's the core tech, and the company offers all of that via API. And yeah, we invest a lot in the content and educational side. It's still an area that is quite new. Large Language Models as a Service is a new brand of company. It's only been around for two years. And so, yeah, we focus a lot on the educational side of the various concepts that are needed there to help both developers, but also a general audience capture and build those intuitions. And that's something that companies had to do throughout the development of technology. So the majority of executives now...
would know what an API is, for example. But 15 years ago, maybe 20 years ago, API or big data or cloud hosting were all sort of deeply technical words or words that didn't yet develop. And that is the same now with things like embedding and fine-tuning and base model, yeah, language model. And so, yeah, definitely the education is a part of that. And a lot of it is just us learning with our developer community and sharing the common lessons. So developer number 10,000 doesn't have to repeat the mistakes of the previous developers. We're really passionate about that as well. Yeah, part of the advantage of having some of this, and I'd be curious,
to hear your thoughts on this as well. So, you know, going back to like something I said earlier when I was first learning about transformers and trying to get hands on with these things, like there was no like easy cloud API for me to just like. access and use. I think now, even for open models, that's shifting a lot where you can host things on Hugging Faces Inference API, or you could use any number of these services. So you could use Cohere's large language models. You could use Replicate and what they have in their cloud APIs and that sort of thing. So a lot of these are available now where it's a small number of lines of code and you're able to access the API. What do you think are the implications of that for how people are thinking about using models maybe differently than they thought in the past because of these access patterns?
Maybe it's less people are thinking about the training side and just like chaining. I don't think it's an accident. We've seen a lot of like chaining things recently. But yeah, I don't know. I'm curious to hear about your thoughts on the implications of how this landscape is changing to where like we've kind of gone from, well, let me download model weights to like I could chain this API together with this API and that sort of thing. I mean, definitely it makes it a lot easier for.
a much more larger group of people to start experimenting with these models because it just lowers the barrier of entry so much and enables people to not think about moving tensors across GPUs and watching out for GPUs running out of memory and updating model weights. And it's just another abstraction. And you can think about it just like every other cloud service out there. So if you want to build a new website, you no longer need to buy a physical machine, ship it to a data center, maybe go physically to that data center and sort of put the code on. It's been abstracted away as a service you can reliably access somebody else's.
has the world's foremost experts making that service reliable for you. And you can focus on your core business problem, the core sort of product that you want to do. And knowing that these other pieces are there and are being handled by people whose sole job is to maintain the quality and increase the quality of these models and the uptime. And this is especially... A factor when these models are massive, they need to be on so many different machines and GPUs. And it's such a hassle to deploy your own model. You need a PhD maybe to really wrap your head around everything that is involved in something like. And so, yeah, it just frees up people to think about, OK, this is an API. Think about the frontier of the next level of services that are now finally possible that weren't possible before. And let's say we saw new industries come out of these developments in AI.
Writing assistance, for example, is a type of industry where there are so many companies now that didn't exist before. And these companies just rely, in general, rely on APIs, and they can focus on really creating the best domain knowledge for them to help out their customers. So it really helps in the specialization and sort of that abstraction of not having to worry about this lower layer in the stack where others are sort of handling it for you. And fine-tuning becomes as easy as uploading a text file rather than a process of babysitting a model for... a week to see you know what happens and so that definitely increases the cycle of experimentation but also the ease of deployment um in accelerates let's say the coming of the next generation of products that are just now possible Yeah. And one of the things that we talked about before we started recording was some of your excitement around like multimodal models and where those are going. I know that's also increasingly easy to kind of like tie different modalities together. I think the light bulb is going off for a lot of people. I even just had a conversation yesterday where someone was talking to me about like a large language model. Could it do this or that like for their use case? And I said, yeah, but like what if you just add the image component on as well? Like you can, you know, generate the copy for your ad and the image for your ad, you know, with generated text and that sort of thing, which I know a lot of people are trying and a bunch of.
other things too. So what's on your mind in terms of kind of multimodality? I know you've written a lot about stable diffusion and other things recently. Where's your mind with respect to that? And what are some of the use cases that you're thinking of or the sort of applied things that are interesting to you in that space? Yeah. So on the research front, I've written about Gato, DeepMind's Gato model that does images and text and a lot of other modalities, which is an interesting research development. On the more applied side, we've released a notebook that does a little bit of prompt chaining.
A few researchers from DeepMind had this paper called Dramatron, where they shared a system and a bunch of prompts that uses language models to write a screenplay. And it doesn't write it from one prompt. It's seven different prompts that do different things of the story, and then you end up with the screenplay. So there's a prompt to generate the characters and a prompt to generate the setting and then the beats of the story. And so you build this knowledge hierarchy. And so we have a notebook that...
showcases how to do that with Coheres models, but also plug in some calls to stable diffusion models to generate, okay, so these are the descriptions of the characters. What might these characters look like? So that is an AI image generation sort of flow. This is a description of a setting where the part of the story takes place. How can we sort of visualize it enough? And these are sort of the flows. And we see, you know, libraries kind of like a line chain that are empowering a lot of this chaining of the various text models, but also...
potentially in mid-generation models. So yeah, that's the most, the use case that I went down. And it was only really possible because, you know, like for me to have a quick time to experiment with them because there are APIs that do them. So on the stable diffusion front, I can share this code because it's an API call to, let's say, Stability AI's stable diffusion models. But I've been wanting to do that with mid-journey, but mid-journey does not have an API that you can call. I have to do, you have to do it through like Discord, for example. And so that's also, let's say, a differentiating factor for these various products, you know.
which one supports API access, which don't. So it will factor into, let's say, developer adoption for them on the infrastructure layer. But yeah, that would be specifically the one that I've experimented with the most. Awesome. As we kind of wrap up here, usually we ask our guests sort of like, what's on their mind moving into the next six months or so? What are you most excited about? What have you not explored yet? But what's like on the top of your list to dive into? Any thoughts?
Yeah, I'm really excited about use cases that use both generation and embedding in the same sort of flow. That's one area where I think about a lot, and let's see the topic. Modeling and, let's say, cluster naming use case is one of them. And we've open sourced a library called Topically that does exactly that. So that's one area where I'm working closely. And I think these models can help us really understand large collections of data using that and create interesting visualizations of them as well. So, yeah, for me, it's mostly, yeah, the interaction of these sort of two systems, hopefully, let's say, supporting multiple languages as well. So multimodality is interesting. Also multilinguality is interesting for systems that can support even more and more data that's out there. I think these are most likely the ones that are just top.
of mind for me at this time. Awesome. Well, thanks so much for taking time to chat, Jay. I'm glad we got to do this and I hope to have you back on the show in a year so we can talk about all the fun things you've explored in the interim. So thanks a lot. Amazing. Thank you so much. So good to catch up with you and chat about all of this. I look forward to speaking again.
Thank you for listening to Practical AI. Your next step is to subscribe now, if you haven't already. And if you're a longtime listener of the show, help us reach more people by sharing Practical AI with your friends and colleagues. Thanks once again to Fastly and Fly for partnering with us to bring you all Change Talk podcasts. Check out what they're up to at fastly.com and fly.io. And to our Beat Freakin' Residence Breakmaster Cylinder for continuously cranking out the best beats in the biz. That's all for now. We'll talk to you again next time.