Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30-plus regions on six continents, so you can launch your app near your users. Learn more at Fly.io.
Hello, Jared Santo here, Practical AI's producer and managing editor of all the shows here at Changelog. Daniel and Chris took this week off, but we didn't want to leave you hanging without anything to listen to. So today's episode is going to be a little different than our usual fare. AI is permeating the entire software industry, so we found ourselves talking about its impact, sometimes in practical ways, other times in less practical ways, on many of our pods. So today, we are serving you a sampler platter. You'll hear a segment from this week's JS Party Podcast, where me and my two co-hosts, K-Ball and Nick Neesey, discuss the recently announced...
Devon Project, which is making waves in developer land. You'll hear a segment from a recent GoTime episode called How Long Until I Lose My Job to AI, where Johnny Borsico and his experienced panel of friends discuss using CodeGen AI to augment your dev skills instead of replacing you. And finally, you'll hear a segment from The Change Log, where my co-host, Adam Stachowiak, and I talk to Jose Valim. Jose is the creator of the Elixir programming language, and he's been a guest here on Practical AI in the Past, talking about Elixir AI tooling. Today, you'll hear us question him regarding Elixir's place in a world increasingly influenced by large language models and how he thinks about it as a language author and promoter.
Hopefully there's a little something for everyone on this episode. And if it's not approaching AI from a perspective that's compelling to you, don't worry. Your regularly scheduled programming with Chris and Daniel will be back next week. Okay, first up, it's JS Party and Devin. uh there's some news that's good there's also some news that's maybe bad maybe good i don't know the thing that everybody's talking about this week at least as we record and last week as well is devin devin d-e-v-i-n the first ai software engineer according to the makers of devin which is cognition labs a new company which raised a series a led by founders fund headed by scott woo who seems to be a very intelligent person even from a young age if you watch that video of him doing math very quickly at ages when it seems like you shouldn't know math very quickly and um
They got a demo out there of this new AI software engineer. So I could say more. I'll stop right there. You all have probably seen the demo, Gable and Nick, or at least heard about what's going on. This is a new tool which can start from scratch and do some cool stuff. I'll just leave it there for now. We can talk about the details. If you're excited, you too can pay for the right to have a software engineer that can only fix one in seven of your tickets and spin up lots of new ways for AWS to charge you money without your oversight.
Sounds like an intern. No, just kidding. Sounds nice. What are you referring to? Is this some specific things that Devin's been up to? So high level, there's a couple of things that I'm referring to here. So one is like, they're pumping up the marketing on it. This is a standalone software. Why get a coding assistant? Get something that can go and do your software. And they publish some data on it. And like, it does do better than the state of the art in terms of tackling a, going from a GitHub issue to, okay, I'm going to actually solve this implement a change and get it to happen. But like the number they published, I think was 13.86% of issues unresolved. So that's about one in seven.
So you pointed at a list of issues and it can independently go and solve one in seven. And first off to me, I'm like, that is not an independent software developer. Like that's, and furthermore, I find myself asking if it's success rate is one in seven, how do you know which one, right? Like are the other six of those, it just got stuck or is it submitted something broken, right? Because if it sets up something broken that doesn't actually solve the issue, not only do you have it only actually solving one in seven, but you've added load because you have to go and debug and figure out which things are broken. And like,
Like you have a whole bunch of additional loads. So like, I think the marketing stance there is a little over the top relative to what's being delivered. The other thing, and this is around, I think a part of what they do is, oh, it can spin up resources for you, right? And they show this cool demo of like, you pointed at this thing and it allocates a bunch of different production resources for you. And the person who's handled DevOps in me before, and now they, you know.
engineering leader who has to sign off off on our digital ocean or aws or google cloud or whatever expenditures you might have looks at that and is terrified by i'm going to give an llm which is known for hallucination which is you know these things are not you have to design application i'm building applications but you have to design around their unpredictability and their willingness to lie and i'm going to give that raw access to spinning up resources in my cloud like that sounds well it sounds like something i would not sign up for i'll say that okay okay well let he whose success rate at issues That is greater than one in seven. Cast the first stone. I was wondering how what Nick's ratio is over there. You know, like one in seven sounds about the way I would do. I pull off the easiest one first. Does Devin know what the easy tickets look like? You know, because that's the skill right there. I'm only here counting on my fingers trying to see if I'm within that ratio.
But do you know when you fail or do you just throw out a broken code and you're like, here you go? It's more of a question of do I know when I succeed, I guess. Right. Which is, yeah, same thing. You think you succeeded until you find out later that actually you failed. You know, that's been my experience. Or you succeeded under the constraints that you put yourself under, right? Or that was actually specified in the ticket itself. But you actually failed at some other unnamed, unlisted constraints that were unknown at the time, but are obviously clearly there in production. And so in that context, you failed. It's not easy.
It's not easy to succeed in this world. Well, what about cable? What if you can't you point Devin at like a $5 a month digital ocean and say, you know, deploy to this and like, can't you cap your risk? I guess on the, on the DevOps side. You probably can. And like, I do want, so I'm taking a hard skeptic stance on particularly the claim that this is an AI software engineer. Like don't hire a person, use this thing. And this is their claim. So I think it's fair for you to be that harsh on them because they say meet Devin, the world's first fully autonomous AI software engineer. That's a very bold claim. So I think it's fair that you're being that harsh. Go ahead.
They're showing some cool stuff. It looks like a pretty interesting tool to put in the hands of someone who knows what they're doing and is able to validate it and is able to say, okay, go and solve this relatively well-constrained problem where I can easily validate the correctness of your output. Go at the sandbox where I know that you're not spending up massive amounts of resources in a way that I'm going to regret. Or even go at this non-sandbox situation, but I have the knowledge to check what you did, look at the logs, and be like, yeah, that's okay. Those are really cool things. That could be really valuable. That could dramatically increase somebody's productivity.
Those are so far from being something that I would trust independently to replace a software developer that they're not even in the same country, like maybe not even in the same world. Like these are just completely different claims. Yeah. I think that the sensationalism of this comes from not what it can do now, but what it represents and the progress that it's made when comparing to other things like whatever it was comparing that, you know, 13% to, to other AI chat things that can do things. It's way better than all of those. It still sucks compared to a human, but it's made like monumental progress in terms of AI. And I guess the question is, does that continue?
Can it get further than that? Or will it reach some kind of limit? And then the other piece of it, I think just from a marketing thing, and I'll be honest, the only thing I've seen on it really is a fireship video is that it's already doing some work on Upwork. So in a way, like that's a marketing claim that they competes against real humans for jobs. Truth. According to them, I haven't confirmed it. What you said is true that they say that. Yes. So this is a struggle with it with all of the LLM world right now and all the world, because on the one hand, you get it. We have been in a place where we're in the rapid part of an S curve. There have been some very rapid advancements in the core capabilities of these things.
And they are super freaking cool, like really cool. And also they have a lot of limitations. A lot of those limitations are baked into the architecture that's being used. And so you get kind of a situation where like there's a bunch of people doing really cool stuff with this and like figuring, trying to figure out what it's good for, but it demos way better than it does anything reliably in production because you can get a really cool outcome, you know, 40% of the time, some situations, 70% of the time. And like you show that and people are like, oh my gosh, this is going to take over the world. And I...
would not trust a, for example, AI software engineer that even that they can handle 70% of my tickets, but 30% of the time spins up millions of dollars of costs for me, right? Or like other things. And once again, like I'm not trying to take away from the technology, but I don't think these hyperbolic claims actually serve anyone except for getting attention. They get attention. Okay, great. And you're going to get a whole bunch of people who buy this thing are disappointed. If it costs them a bunch of money, they'll sue your ass off. And like, why would you do that to yourself? It's somewhat similar to generative AI in the image. Let's just stick with like static image world.
where everywhere you see is impressive results. And they'll be like this new, like mid journey seven is off the charts. Amazing. Here's nine examples that'll blow your mind, right? And if you click through on that, they're all going to be very impressive. Like those are amazing things. But then you have to stop and think, well, mid journey didn't create nine examples. That blew my mind. Mid journey probably created 40, 50, maybe 500 examples. And then you human decided which ones were amazing. And you cherry pick those out as the examples. And like, that's a great team work guys, right? Computers plus humans equals better results. And so.
There's a cherry pick, and that's what code review on these things will be. That's what happens when you tell Copilot, no, I do not want that function, right? It's all, as Hipster Brown calls it in the chat room, human in the loop. And that's exactly what is necessary. And I think the reason why you call them hyperbolic claims, K-Ball, is because they're saying it's a fully autonomous AI software engineer. Human out of the loop. Let it rip. And maybe fans of the bear will like to say let it rip, but those of us who aren't fans of Devin are thinking, let's not let it rip too much because it might just tear the whole thing down. Now I'm being hyperbolic. Nick, you're not along. Do you agree with me? Somewhat, yeah. I think, like, yeah, it's humans who are...
deciding what is good out of that and kind of helping to train that going forward. But in a way, like I was trying to think and trying to relate this to another article I saw that wasn't about Devon specifically, but was about like prompt engineering as a quote unquote profession being taken over already by AI because an AI can iterate and more quickly come up with a way to answer the questions that you want by appending exactly what it wants to hear at the end of a string. And I think the example that I heard from that was like, we want you to answer this question. And the AI is quote unquote incentivized to answer it a little bit better if you put it into a scenario that it likes. The AI is Captain Kirk on the Enterprise and it has to answer this question to save a planet from whatever. And the question could be, what's 2 plus 2? Or something really simple. And by putting in all of these extra prompt words that the AI is coming up with on its own, it's making better results overall. And I'm just wondering how that marries to the idea of humans being the ones who curate the good ideas that come out of it. Well, prompt engineering, I've been convinced by SWIX that it's a code smell. At first I was convinced this is the new thing that everybody needs to learn. And I think it's just a leaky abstraction that we're currently dealing with as humans because...
the tooling is not good enough so that we have to engineer the prompts. I mean, Google's search box is prompt engineering, right? Like knowing how to Google, it's the exact same thing. It's just way harder. And it's like way more magical now to like tell it the magical incantations to get the best results back out. And so the fact that it knows what results are better to me is not intelligence or anything. It's just like, we just need that to go away. And I think that's, I think Devin's actually an example of where they've productized and hidden a lot of the innards that we've currently been exposed to in order to make the tool work better than it would for an experienced user to use it. Like they've actually turned it into a product. And I think that's great. I think it's one step.
on a long line of iterative improvements that will make it so the prompt engineering, I mean, you're just going to basically talk to it in layman's terms and it will know how to feed itself the correct prompt, so to speak, in order to get the, to get the goodness out. But I don't know. Hey, well, back to you. Yeah. I mean, I think so high level on all of this AI stuff is there's really cool stuff there. We're figuring out how to use it. And the current state is clearly intermediate. However, the thing I want to keep coming back to with this is like.
There are things that it's like, okay, this technology is immature and we're going to evolve around it and figuring out how we handle prompts and managing prompts and what's generating them and whatever, like that fits well in that bucket. And there are things that are fundamental pieces of the way the technology is designed, right? LLMs, machine learning models in general, are statistical, probabilistic. They're very different than most things you think about in software where you're trying to make something that is logical, consistent, like you could put A in, you could B out. And that is not there with these things. And so you can design applications around that. And there are things that you can do to...
to sort of pin that down to add validation that is outside of the LLM and do other things. And maybe Devin is doing that. But I think the more we start looking at these sort of places that require judgment, places that require precision, places that like if you just make some random up, it can cause a lot of problems. Like those are not actually like there's a fundamental thing about what the technology does. That means it's not necessarily going to be a good building block for that. And so making hyperbolic promises about where it's going to develop that depend on it being a fundamentally different technology than what it is. I feel like they are setting yourself up for a lot of heartbreak. What about the job market? Do you think it's fundamentally affected?
by tools like Devon as they progress over the next three to five years, because we're not talking about humans out of the loop. I think we're all in agreement here that that's not feasible or smart, at least in today's technology plateau of LLMs, but less humans in the loop. You know, that seems like it's very feasible if these tools continue to iterate and even just not have revolution, but evolutionary advancements from here. Yeah, if it makes me three to five times faster, do we need three to five times fewer engineers? Yeah, I mean, I think there is. So this is a technology that has the potential to dramatically impact the productivity of software engineers. And I think.
There's a couple of different things around that as we think. So short term, that can create some disruption, right? Short term, that means that a company that had been running on, say, five engineers and might have needed to hire and expand to 15, now they don't have to expand nearly as soon and things like that. So I think there is the potential for relatively short-term disruption. I will say both the history of economics broadly and software in particular is that every time we make it easier to code, we discover there are whole worlds now that we can address and build software around that we couldn't before. So if, for example, and this actually, there's a particular example of this that I think is interesting to dive into. So one of the big economic challenges in the tech industry in the last four or five years is that we had these massive tech companies with incredibly high revenue per employee, Google, Meta, Netflix, like the Fangs mostly, right? And so they were able to set the salary bar.
that was super high. They were paying ridiculous amounts of money. That's a technical term, ridiculous, for software engineers. And then when we had very low interest rates and a ton of VC money flowing in to the industry, there were lots of companies whose fundamental business economics do not support that level of salary per software engineer, who were nevertheless paying that amount of salary per software engineer based on VC capital. And sort of this thesis that, okay, we'll be able to scale out of this and we'll get whatever. And I think that caused a lot of distortions and problems in the field. Now, if suddenly software engineers are three to five times more productive,
the range of businesses that could use software, but previously could not afford to compete with the FAANGs, et cetera, of the world. There's a whole set of business models in there that become viable because it's that much cheaper to develop software. And so I can imagine this actually dramatically expanding the number of viable either software businesses or businesses that are non-tech, but would like to include software or could have custom software and dramatically expanding the number of those that happen. So I think long-term, I don't think it's a negative impact on a software engineering career path. I think that there's, what it means to be a software engineer looks a little bit different when you have.
different types of tooling. That has been true as long as I've been around. Just JavaScript land, right? I remember when jQuery was a revelation. Oh my gosh. This is going to make me so much more productive. It did make me so much more productive, all these other different things. And now the level of tooling that we have there that supports our productivity, building things on the front end is astronomical. And has that taken away from the number of people writing JavaScript? Speaking of astronomical, Astro has a new database.
What's up, friends? Is your code getting dragged down by joins and long query times? The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. Graphs work well for use cases with lots of data connections like supply chain, fraud detection, real-time analytics, and generative AI. With Neo4j, you can code in your favorite programming language and against any driver. Plus, it's easy to integrate into your tech stack.
People are solving some of the world's biggest problems with graphs. And now it's your turn. Visit Neo4j.com slash developer to get started. Again, Neo4j.com slash developer. That's Neo4j.com slash developer. Up next, we have a GoTime podcast fireside chat between longtime programmers Kent Quirk, Sharon Diorio, Stephen Pyle, and host Johnny Borsico. It's one thing to have Jan and I, you know, pump out snippets of code that is part of a larger whole, right? Whereby I'm the engineer. I'm engineering a solution.
not just, I'm not just a code monkey, just clacking out, you know, syntax. I'm trying to fix a problem. I'm engineering a solution to a business problem. Now I could go, you know, as high level as I can. I can just open up, you know, Copilot in chat mode and say, hey, this is what I'm trying to accomplish. Start spitting out, you know, files, right? Now, maybe today, right? It can build somewhat trivial apps. I've seen YouTube videos and clips and things of it, spitting out entire working React apps and all these things. And that's, that's great. And I think over time, it's going to come even better at doing those things. But I have a hard time trying to sort of correlate that or trying to replace solution building. Because to me,
solutions aren't static right when a business comes to me says hey i need you to build a solution to this problem I build it. They take it into production. They do stuff with it. And they come back and say, hey, you know what? This is great. Now I need to change it in this way. Or I need to account for this exception. Or I need to account for this particular use case. Or this specific customer. Where 90% of the time it works this way for every customer of this type. But for this customer of that type. On Ultimate Thursdays. It's a completely different thing. Exactly. So now how are we supposed to treat those entirely made up solutions? Am I just feeding that back into the system? And saying, hey, so now I account for these alternative approaches.
Is it going to be like it was when the first generated code frameworks started hitting the scene? And you'd go in and there'd be all this code. It was like, yes, super fast. If you had to do an ORM, it'd run all the code for you, et cetera. And then you needed to change something. And all of a sudden it was like, you know, change management in some of those days was. Yeah, or regenerate the whole thing from scratch. And oh, sorry about all your customization. Yeah. So that'll be another big test that AI has not yet proven.
It can do. Well, so let's talk about art for a second, though, because this is, again, a similar thing. Like, everybody's really excited. Look at the images I can generate with, you know, mid-journey or whatever. With stone art. Well, right. But the point is, again, it's going out and giving you the average solution. It's going out and going, here are the things that look most like what you described that somebody else has created already. And kind of kind of cobble pieces of that together. Or here's an opinion formed by the loudest voices out there that I sucked up as source data. Hell yeah. But, like, I sat in a meeting today where an artist went over her design, basically her design process for a big design project. Like, here's...
the resources I looked at. Here is the feeling I was going for. Here are the things I considered. I looked at these typefaces. This typeface reminded me of this building architecture, which is relevant to the site. And then that artist proceeded to churn out, over the course of a couple of months, 200 pieces of support art for an event. That was a brilliant design exercise by somebody deeply steeped in art and creation who then studied
the event and what the event needed and integrated all that. And yes, some random person could have sat down with mid-journey and said, make me this stuff. And it would have been much less good. But people who don't know the difference would have been, sure, it looks fine. I mean, we've all seen that, right? My document has 37 fonts and 12 colors, but it looks fine to me. But there's a big difference between something crafted and something just slapped together. And yeah, I guess I think that AI is going to make it easier to slap together. For most people, though, would you argue? So here's what I'm not saying. I'm not saying that
these things generated by AI. Like if you're a connoisseur of a particular art, you're an architect of a particular kind of application or solution or thing, whatever it is, you can critique the output of Gen AI as it stands today. Again, arguably, it's going to get better at what it does, right? But you can critique the output today and be like, this is subpar, right? This is not as good as what I could have come up with. But for most people, it's good enough. It depends on what they're using it for. And so again, if you're just doing something for yourself, who the hell cares? I mean, yeah, I'll slap something together out of two by fours if I'm building it for my garage. I don't care. But if I'm going to sell it?
If I'm going to make a business around it, that's the part where I'm saying, I don't think the AI stuff is there. If you're just doing a hacky project for personal use, yeah, I mean, maybe you would have had to pay somebody to come in and slap that shelf together in your garage if you didn't have the skills to do it yourself. And so now there's this kind of, you know, yes, there's a few things that I couldn't do before that now I can do today for myself. Design that invitation for my kid's birthday party. Hell yeah. I can't draw, but I can use an AI. There's nothing wrong with that. And so yeah, there's probably some, you know, the kid next door, you're not paying 20 bucks to do that for you. But that's now. What happens in the future as AI evolves and improves?
Did we get to this uncanny valley level of like, oh, no, it's not just good enough. It's like, it's like the standard. Now it's building a whole kitchen. Right. Do we need to worry about that? And how much coding out there is most of what's out, you know, how many cruds have we ever created in our lives? How many cruds are still being created every day? Yeah. Okay. So that's a problem that's largely been solved greater or lesser degree, but yeah, I mean, it's white box, right? You make white box easy to do. Yeah. And tech has always been making things that had gates around them or real or created limited availability or, and making them more available, like artisan things that used to be.
Only certain artisans could do that. Only musical artists had a studio and an audio engineer, and now they can go and create their own tracks with one app at home. For me, that's the part that's like, oh, I can't complain. I can't be the curmudgeon. Complaining about this latest thing that might make something that I do more accessible to other people. I've benefited from these other things that came along. It's time to share the wealth. I don't want to. I'm really rooting against it. I don't think it's about gatekeeping. I feel like a big chunk of my career has been spent trying to help.
people learn to program. And so I'm not thinking that the reason I'm skeptical is because I don't want other people to do what I do. I think it's more because I feel like the hype and the reality are distinct, that what the reality is producing is mostly devoid of creativity. People are confusing knowing what to look up with being creative. And I think knowing what to look up is a skill and a lot of us have it and the better engineers I think are better at it. And so yes, AI helps to ease that problem, but knowing what to even ask about.
You know, like, or looking at a new solution to a problem. That's something that I think is well beyond what AI is capable of now or in the reasonably, like the LLM model, I think is fundamentally non-creative. That's my take on it. So I see. We're not at the, uh, so one thing you mentioned, like the whole teaching, like y'all remember when, um, maybe it was during a, maybe first or second Obama term or something, but there was this giant push to teach everybody how to code.
Right. Like it was, it was everywhere. It was in the media. It was in newspapers. It was like, you know, we need to teach our young how to program. Now I'm looking at a clip from, you know, NVIDIA CEO, like three or four days ago, something saying, Hey, people shouldn't learn how to program. You should now let, you know, the new programming language is a human language. And I'm thinking, man, you are sitting here. You stand to gain billions of bajillion dollars, right? If your wish comes to, cause you're producing, you know, chips and stuff for these things. Of course you're going to say that. Right. So this, this, so I mean, what I'm definitely not on the, don't teach people how to program camp. No, no, I'm going to, I'm going to take a slightly spicy take here.
I don't think he's completely off. Now, in the time we've all been engineers, we've seen waves of different things that are going to come and take our jobs, you know, offshoring and, you know, cogeneration, now AI, and they haven't. And my theory is that the key thing that an engineer has is the ability to communicate. And even when you're supposed to be communicating to people on the other side, product, the business, whatever affectionate term you use for them, aren't always as good at that, although it should be part of their job. But having somebody who can think back and forth, there will, I think, always be a need for those people.
Because every CEO thinks they have the answer to every question. That's what they need to do. Right? But they really shouldn't. If they have a business that's big enough to grow, their biggest skill is finding the people they put in the right place. So if your job right now is like doing creds for a company that can't even explain what they want, I wouldn't worry because they're not going to be able to explain what they want to AI. Right. Yeah, no, it's the thinking logic. Yeah, there's the think about, break something down into steps and think logically. Like I once did have a client very early in my career who was a pretty good business person.
who really wanted to automate his business. And he was able to sit down and explain it to me. If he had had the tools to program, he could have written his own code because he thought about it really logically. And it was just my job to basically take dictation and turn it into Pascal for him back in the day. But that's few and far between. Quite honestly, most people who specialize in business aren't specializing in thinking logically. They specialize in thinking about people and, like you said, about communications. So does AI then make you more what you already are? If you're a logical thinker, you'll benefit. And if you're not, you still struggle?
And who gets to train the agent? I want to be in the training site. I want to be the one doing the building of the things that you use. That's a good question. I think at the point where it becomes like a personal AI, where it's just like, it's tuned to you. Your data doesn't get shared. It's just, you know, then it becomes like a superpower, right? You can just, it's like your co-pilot. But how would that work if it's only got your data? It's basically replicating to a point you. There's a generic, it's trained on the universe and then specialized for you is the way that we're seeing all this.
It's like creating your own GPT, but it's based on a larger model, right? Yeah, exactly. So, I mean, like my company, we built, we have a query engine that looks like SQL. And then we built an AI where we train that AI, like as part of the prompt, we basically can go out and get your data and all the names of your fields and the data types of your fields. And we can plug them into the AI query so that when you say, show me my slowest service, it can go, all right, what are the fields that are named according to?
you know, time, duration, and what are the things that look like a service name? And now I can write a query for you so it knows how to query Honeycomb. And then it can write that query for you from your inept prompt because it's been specialized for that particular type of application. And I think that's a really cool use of AI. That is a productivity boost for people who are already technical. Like, you know, full disclosure, my startup is a Honeycomb customer. So, you know, I will go in the dashboard and I can formulate those queries, but I'm already a highly...
technical user who knows how to use these kinds of tools to get and know exactly what kind of data I'm looking for and when I've found it. Now for the layperson, right, who doesn't know, like the layperson, like the more I think about it, the more I'm thinking, okay, if I'm a, on one end of the spectrum, you have the complete layperson who is using perhaps, you know, chat GPT or something like it to maybe generate copy and not hiring a copyright person like you might traditionally do back in the day. I'm sure the copywriters of the world are suffering right now because content creators are suffering because this stuff is now being generated. So if that was your job, absolutely you're impacted. And the layperson can now bypass you and get to something that, again, they're good enough. They can get something good enough to achieve some means. On the complete opposite end of the spectrum, you have people who engineer software.
Again, given the context of the conversation, we're talking about, like, how safe is our jobs, right? And so when I'm asking this question, I'm not asking, is the layperson going to find ways of reducing, right, their reliance on sort of, I don't want to say lower skills, just a different kind of skill, right? I'm thinking, like, for people like us, as software engineers, who presumably will be impacted by this to some degree, right? And we already are, right? For us, there's also a micro spectrum, whereby if you're on the sort of the lower end of that spectrum, and if the only thing we're doing is generating crud, well, I'm sorry, your job is indeed in jeopardy, if that's the only thing you've been doing.
with your career. On the opposite side of it is the highly specialized person who understands a business problem has to debug and troubleshoot and then talk to people and integrate different things. Institutional knowledge. Right. All that stuff. I mean, I don't see that skill, right? I don't see that being replaced by AI anytime soon. Am I wrong here?
I don't think so, personally. How many of those people do we need? And that's the thing, right? Does he have musical chairs? We should be looking for a chair now? No, any business is thinking, do I need 1,000 engineers when 500 will do? I mean, I think, as we've all said, it makes us more productive today. So I'm writing more lines of code per day than I was five years ago. Right, right, right. So that's good. But we all kind of expect productivity to continue to rise. So this is a productivity tool. Not a replacement tool. We're also using languages that are more expressive than they were.
The code I write in Go is probably one-third the length of the same code I write in C++ or used to write in C++ back in the day. So that's also a productivity boost at some level, at least if you believe the old metrics that it's basically you can write the same number of lines of code per day no matter what language you write it in. But I think actually knowing how to use it is a skill to put on your resume before too long, or if not now. I mean, even if I didn't want to use it, I would because it's becoming of like, it's going to be a point where like, Oh, you know, I use Copilot all the time. Oh, good. You have the point for you to get the job. Is prompt engineering already in your link? It's going to be soon. No, but if you put it in your interests as AI, it shows up in the keyword searches. There you go, right?
I mean, that's a good point. But to me, it's like back to my woodworking thing. It's like, I know how to use a power saw. I know how to use a drill press. I know how to use a lathe. You know, those are kind of expected. Today, if I'm going to do woodworking and say I only use hand tools, people are going to look at me like, I don't have time for you. And the same is true of like, if you're not using Copilot, what am I paying you per hour? Why are you not as productive as it could be? Yeah. I think at this point, the only people who really aren't using it are people who are doing like very arcane languages or people who their businesses, they don't allow it. Their company doesn't allow it. I think everybody else has at least tried it.
I mean, if you're a company that doesn't allow such things, like I understand not bringing sort of open source code into your organization that might be the wrong licensing model for you or something like that, right? You don't want to be in some hot water. All you have to do is look at, you know, Oracle and Google over the whole Java thing. I think those were the companies involved. But if you allow your engineers to use a model.
where you can control the kinds of things that were used in the model, right? For the training. And you can have, maybe you can run your own internal, right? Gen AI for code generation, whatever it is, right? I think if you're an organization that is afraid of these things, you should at least follow that route as opposed to saying, hey, nobody can use any Gen AI coding tools whatsoever because I think you're going to lose people if you do that. Because I'm going to look at my peers that get to use these things and they're learning those skills, right? And then now I'm falling behind because everybody's using some sort of code generation tool and I'm not, right? I mean. This is where I hope somebody reaches out to somebody who hears this and reaches out and can answer that question of like, can you?
have a copy of Copilot that you train on a specified set of repos and only those repos. Private repos. Well, I mean, I would, you know, as we joked, I mean, if I'm a Go developer, I'm training it on Johnny's code. Because if I don't like it, I'm not with Johnny. But I mean, there's people out there. One neck to choke. I get it. I get it. It doesn't matter what language it is. There's people out there that you respect their code and you'd be like, yes, I would like my code to be more like this. I wish it would learn that that's what I was thinking or that's the way this problem should be thought about. I mean, yes, there's precious few of those people and those people will probably never lose their jobs, but.
For the rest of them, your models are going to have to work with the tools that are out there. And I would love if this is a possibility now that you could train Copilot on what you'd say to train it on and not all of GitHub. I think there are companies that are working on that product. I feel like I've even seen a product announcement like it. But yeah, I mean, you know, the thing about it is you can take one of these LLMs and you can essentially subset it and you can make a tiny compact LLM that will run in a box that you can actually stand on your desktop. And then you can further train that with new information. So that's exactly what you want to do here. You want to take a coding centric.
LLM, like a copilot, and create the mini version of it, and then train it on your repositories, and now it knows how to write your code, and it's also not talking out to the cloud while you're doing it. So there's got to be businesses like that. And this is where we redact all of this and put it into our business plan. What's up, friends? There's a new book out there called The Hacker Mindset. This is a productivity cheat code to unlock new levels of success in your career, in your career pursuits, and in your personal growth. This book is about leveraging the principles of white hat hacking and applying those skills to the broader world. It's available for pre-order right now, and it's not your typical productivity guide. This is written by Garrett G., a seasoned white hat hacker with over 20 years of experience. This book reveals the secrets of hacking and how you can apply those skills to overcome obstacles and achieve your goals. So don't miss your chance to get ahead and get this book, The Hacker Mindset.
You can pre-order your copy today at thehackermindset.com. Be among the first of many to tap into this power of hacking for your success. Join the movement and embrace a new way of thinking. Again, that's thehackermindset.com. Last up, we have Jose Valim, creator of the Elixir programming language on Changelog and Friends. That is our talk show flavor of the Changelog podcast. To Jared's point earlier, AI, as it's known today, GPT, ChatGPT, and others are not that good with assisting with Elixir programming.
And so I guess the question is, what does it take to make it good? You mentioned embeddings earlier. You mentioned documentation being more readily available. What does it take from a, I guess, a leader in the Elixir world to enable LLMs to be better? Like, what role do you play in that journey for them to better consume the documentation and better know how to programming an Elixir to help folks like Jared and myself or our team or others to really become better and more proficient at Elixir versus just like anytime Jared asks ChatGPT for assistance, it's just like, no, it's not good. So just quit.
So I think, if I got the question right, I think we did our work correctly in the sense that, at least from the language point of view, in the sense that documentation was always first class. So documentation is very easy to access. So if what you want to do is to configure an LLM, it's actually very easy to access that programmatically, send that, extract information. And we talked about one of the things that you also have to do is try to get understanding from the... the source code so you can find, oh, this code is using those modules, is importing those things. And those are things that you can do relatively easily in Elixir. We can most likely improve that. So I feel like we have the knife and the cheese. It's just a matter of somebody going and cutting the cheese. I feel like the foundation is there in terms of having this information structured, but somebody needs to feed it somewhere. But again, we can go back. Maybe it's a corporate size.
ChatGPT indexed HexPM already. Not sure, right? Maybe it has done that. I don't know if I can send a letter to somebody, hey, please index my website. Or maybe it's a matter of, so one of the things is that Redwonk, they have released twice a year kind of a graph plotting GitHub against Stack Overflow. And I think they're having the most popular languages according to GitHub and Stack Overflow. And then there's a linear thing in the middle. And it's very funny because Elixir is high on the GitHub side, but quite low on the Stack Overflow side.
And one of the reasons for that is because we have always had the Alexer forum. So that may be one of the things. Where's the knowledge? Where's the back and forth in the community? Yeah, the knowledge is in the forum. Is that thing being indexed? Because we know Stack Overflow is.
Right. And so, and ironically, that's one of the reasons I think I may be misquoting that RedMonk, they are considering removing Stack Overflow from their plots because it's prone, like, I think it has been losing relevance in the last years. Right. But, you know, maybe in the effort of trying to have a closer community where everybody can engage with each other, where I'm active in the forum and I'll probably not have this patience if I was doing Stack Overflow. Right. We created our community at a special place, but it's not known. So, yeah. So I think, I think it's still like too many.
too many unknowns. But I think at the core, at the core, we unwillingly did a good job because we were worried about documentation being accessible, documentation being first class. So we did that and that can be, and we promote people to write documentation, lots of documentation, right? So there is a lot there. And yeah, and maybe the RAG is going to be the thing that is going to be enough. That's one of the hopes, right? Going back to, oh, we want everybody to be able to use this. If RAG is good enough, then a lot of people will be able to augment their ecosystems without depending on OpenAI or whatever. But we are still evaluating. When we talked about sort of the,
the long-term future of Elixir, artificial intelligence, and that sort of larger topic of, you know, how long will be relevant and, you know, can AI generate it well, that whole conversation. This makes me think of this, you know, necessity to not have a black box that is whatever AI is. Because just like you said, who do I send a letter to, to index my stuff so that my very relevant language today remains relevant tomorrow because tomorrow says AI will continue to be more and more relevant to developers in their journey to develop, right? So who do we send a letter to? How do we know? Well, currently the status quo of AI.
is for the most part a black box. Obviously, open source LLMs and indexes have become more and more pushed because of this challenge. But I think this illustrates and highlights really the long-term challenge because even you can't say for sure why what wasn't indexed was indexed for the Elixir corpus, whether that's the forums, whether that's the documentation, the hex documentation or whatever. It's unclear to someone like you how to enable ChatGPT or the likes to better support Elixir assistance for developers using those things to use the tooling. And that's just not cool because long-term we need to have...
inroads into those places so that we can be part of the future if AI is predicting how we'll get to the future. Yeah, and I think, yeah, it's too early. I think we're going to improve a lot. I was listening to a podcast today where Sam Altman, he was saying like, they improved Chagipity 3 about 40 in the orders of magnitude in terms of size, performance, and things like that since they started. I think 10 times for Chagipity, three and a half. And I think open source is going to catch up, I think, and I think that's the hope. But yeah, it's also like we go back to this when we are thinking about Livebook because what I want is for open source chewing.
But when I'm building a feature for Livebook, I need to build the best feature for the users. And when I can use ChatGPT 4, and I can immediately see the results, and they're really, really good. I can use other tools off the shelf. They're not as good. So we are a small company. We're doing open source. So my options, if I have to choose for my users, is going to be ChatGPT 4, because it gives me the best result.
for the least amount of effort, right? I just, it's there. And this is like, so we're back in like about my indecision about investing in this stuff. Is that because I want open source, right? I want things to be open source. But right now, the quickest return of investment is GPT. And then I am in this contradiction space, right? But yeah, and it's just, I think it's just patience. We have to be patient. And, you know, I think probably in one year. And the whole thing is like, it's crazy to think about is that this thing has been happening for a year only, right? It appears that this thing has been out for so long, but it's a year. And I think like, if I'm back on the show in a year, we may potentially be having a very different conversation. So yeah, we'll see.
Do you have any fear about this? Like, even as you respond to that, you sort of had some, I wouldn't say like trepidation in your voice, but you sort of had some uncertainty. Do you have any like fear and uncertainty and doubt the FUD that people sort of pass around? Do you have any fear about this? No, not really. In the sense that I consider myself like very lucky, very fortunate or whatever, or blessed, whatever you want to say it. I think maybe it's a, I'm not being overconfident here, but more like thankful that I think whatever happens to me, it's going to be fine. I truly believe that what's going to make Elixir survive is the community more, you know, than whatever technological changes. Unless there's something very...
I talked to my father about this, about investments, right? So like when Bitcoin wasn't crazy, right? And then my father's like, oh, have you heard like about this thing that if you put your money there, like people got this huge return. And then I always told him, father, if we got to know about it, it's because it's too late. You know, it's like, or something happens, right? It's like, oh, father, like if something happens, it's because like, if something goes this bad, it's because it's going to be bad for everybody. So like, don't try to fight it, right? So again, like, unless there's a very major change, I think I'll be fine.
right? So I'm not worried about me in a sense. I always think more about, it's more about ideals, you know? Again, I like to say, well, me 10 years ago, that's where my trepidation is if things go like closed source, you know? And those things, they happen by, we don't see the results. Like, I think another polemic topic about this, it's like, hey, I use Chrome. As soon as Chrome came out, today I don't use Chrome anymore. But as soon as Chrome came out, I immediately swapped to Chrome.
right? And if I had known that this would lead to a point where, you know, Google is in this position where it has a lot of control over the browser, over the web, right? And over how we use the internet, like 10 years ago, I would probably not have used Crop if I could have seen it, right? So I think that's where my trepidation comes from of like things being closed source, like the developer experience. So today, another example today, like Elixir was the first programming language that GitHub had like the new navigation, code navigation things that were provided by the community.
So there were some programming languages, and there still are, where they have very good navigation and exploration on GitHub UI. And the path for that, to get that feature, to get that behavior was, and I'm very thankful that the GitHub team, they discuss with us and allow us to do that, but that's closed source. And GitHub plays a major role over how developers use. So it all comes back to this idea of if you want to provide a good experience for your users, how much of that is behind something closed source that you have no control, and you are depending on somebody paying attention. to you, like, or you having a contact or, you know, me having a name because I was very active in the Rails community that GitHub uses like 10 years ago, right? Those are the things that, but I, like, I feel lucky, you know, but it worries me, right? Like how much is being closed, right? How much is going to be out of our control? And then the trepidation, I guess, is like, what does that matter for the small Jose out there, right? Let's start building his thing today and they won't be able to.
Well, you killed the vibe there, Jose. Oh, thank you. That's me at parties. Not invited. Just kidding. Oh, funny. All right. Well, let's... Should we try to close on an up note? On a high note? On a buffer? What's that? Wow. I had no idea. I think we should end it right there, Adam. Don't you think? We ended on a high note. This cheese and knife tactic there. I love it. Do you want higher? I don't think it would be good for the listeners. I think that was plenty high enough for me. Adam, were you satisfied with that? That was a high note. Literally. Yes. And I dig it.
Thanks for listening to this very special episode of Practical AI. These were just extracted segments of much longer conversations. If you want to hear more, there's a link in your show notes to each of the episodes featured here. Thanks again to our partners, Fly.io, to our Beat Freakin' Residence, Breakmaster Cylinder, and to our friends at Sentry. Save yourself $100 on the team plan and use code CHANGELOG while signing up. That's all for now. Chris and Daniel return on the next one.