Hello and welcome to another episode of the Bike Shed, a weekly show from your friends at thought about developing great software. This week, we're doing a special episode live at Rails World at this really cool booth sponsored by Buzzsprout in the middle of the conference floor. I'm joined here today by- Rachel Wright-Mann, but more commonly known as Chail Codes Online. And I'm Joel Kinville. And together we're here to share a little bit of what we've learned along the way.
So Rachel, what's new in your world? I just give a talk about generators at this very conference if you can believe it. Yeah, that was intense and exciting and delightful. Congratulations on the talk. It's a big deal and now you get to maybe let the adrenaline go down a bit.
Take a deep breath, calm down a little. So your talk was on the topic of generators. What got you excited to pitch that particular topic to the conference? Yeah, so the last several, gosh, it's been months. I don't want to count how many.
I have been focused on building out generators for Ruby events. At Ruby events, we store all of our event data using GIML, which means that I'm kind of using AI to try and create some automation. And I need to make it more, what's the word, like reliable? I need it to not lie about the events. And so I've been using generators and building out a workflow to do that.
And that's kind of been, so I stream on Twitch and I have spent the last several months building this out on stream. And so I finally got to talk about all of the work I've done and the results that I've seen. And it was incredible. So you mentioned Ruby events. For the audience members who might not be familiar with that project, can you give us the quick elevator pitch?
Oh my gosh, the Ruby events.org is on a mission to index all Ruby events. So we capture events, we capture meetups. If you're interested, if you're attending an event, you can mark that you're attending. You can favorite some of your favorite people. So for example, I have favor to you.
I have favor to review. And we can see which events we're going to when we look at those events. There's other features as well, like in constantly updating list of CFPs. And I think the most shocking thing about Ruby events is that we actually have very little automation. Like it's very much, we see their event updates.
We're subscribed to different events. We're watching the events on Blue Sky. Sometimes the organizers themselves are contributing. Those are my favorites. When an organizer shows up and they're able to contribute those events.
And a lot of the tooling that I've been building out is to make that possible. So for example, Deckon Queen, which is coming up in as an India, first came on our radar because the organizers submitted that event. And then we've been responsible for kind of checking the website, updating it and managing that. Something I understood from your talk is that you don't store all this information about events in a database somewhere. This is a GitHub repo of YAML files that you manually have to update or maybe open source contributors can add their own as well.
Is that correct? Absolutely. We love open source contributors. And I've worked a little bit on the documentation to try and build out and make it easier for them to do that. And yeah, it's a database would be easy because then we could just subscribe to updates with webhooks, right?
But we need something in the mix that can make commits. Fortunately, a tool has recently come up that does that for us, which is AI. So you mentioned AI here is the idea to move away from you as a maintainer having to manually create these events in the GitHub repo. There's a mix of that happening already. So for example, the Rails world CFP was me seeing that CFP pop up in like a Slack or a website or like in blue sky.
And then firing off an agent session to actually build it. But I'm quite frequently like actually opening my, like opening open code and having it go check the website, building it out and then confirming that the website actually matches the YAML file and like the event that we've generated. I see, so you're having an agent go and effectively scrape the conference website to build out the details. Yeah, yeah, yeah. Sometimes I have to copy paste.
So here you go, you should have a script on these schedules pages. So I had to like copy paste the schedule in order to get it to format correctly. Otherwise it wanted to say that there were only 30 minute ones. And the verification step is like a really key one. So like I didn't really get to this in the top.
But one of the things that I've been struggling with lately is ensuring that my PRs have screenshots updated so that I can review it from that PR stage. So then I have screenshots, I don't trust it, which means that I have to go home, open my laptop, hold down the code and actually check it against the event. I think that's something that the development world isn't really leaning into quite as much yet, but I'm seeing a little bit of that in a few different places. The idea that the LM can bake in various artifacts that make it easier to review. And you have no idea.
One of my previous companies used review apps and I miss them so much. Like the classic Heroku review apps? Yes, exactly. Can you imagine you have like six agent sessions go off and then you go and you check your review apps and you're like looking at the PRs and seeing if it worked? I love that.
Unfortunately, not the world we live in today. But screenshots are pretty good start. So one of the directions you've gone in has been really hard on to generators. And that's been basically the topic of your talk. I feel like generators are kind of the OG Rails feature, right?
They're sort of what was shown off in some of the early Israel demos as isn't this cool Rails comes as generators. What made you pick generators as the feature that you wanted to lean on to sort of augment your workflow here? So generators to me are undeniably a major aspect of Rails's success. Like if we're talking about generating code that matches conventions, having a generator where you can just say here are the parameters, go build the rest of the endpoint is absolutely vital. And so I think that it was part of the lifeblood of Rails.
So I completely agree with you there that that was like Rails's magic thing. As for why I picked it, I was sitting there and I was like, I need to see a lie. I went to Rails Girls, some photo and nobody there had tokens. Or if they did, I didn't meet them. I met people who were working in a cloud space because they were working on a Windows computer.
I met people who were just trying to run it locally. And so what the generators allowed me to do was call the generators and the CLI without having to rely on those tokens. I think at some point we're gonna see token costs increase. And when that happens, I wanna make sure that I have a fall back, especially because there was actually kind of a big. So I was using GitHub Co-pilot and I was using the agent sessions.
And then recently, I don't remember which month, they changed how they calculate usage. So before I got 300 requests total and I was like, oh, I get 300 event updates. Maybe there's a couple of revisions there that I have to do, but then they changed it to calculating based on AI credits. And I think I had like a thousand and I tried one PR and it was 220 AI credits. It's gonna burn through fast.
Absolutely. And I'm hoping that Luna helps with that. And I'm definitely testing that route and like using those smaller models. That was one of the things that we talked about in the talk, was being able to use smaller models with better tools because that's gonna allow me to be more token efficient and bring those costs down to something that I can reasonably support with an open source project. Because it's my own usage, it's my own money, it's my own time.
Right. Yeah. Yeah, no, the LMS aspect of it and the token cost definitely matters. In my experience, I found that sometimes my LLM tasks get better when I extract part of them out of the LLM and into something more deterministic, like a script, like a generator, even just a template so that it's not building HTML for me. And at that point, the speed goes up because some of these things are actually kind of slow with an LLM and the token cost is drops.
I completely agree and you're talking about that tomorrow, right? That's right. I'm talking about harness engineering on Rails, which is all about finding different ways to bring in signals into your system, your agent, such that it will help steer it towards the right outcome. Part of what's there is sometimes moving some of that judgment out of the LLM and into something deterministic. Then I do have a moment where I talk about generators because they are really great for that and they're like you said, a foundational piece of rails.
No, I completely agree with you. And I actually saw something really interesting. So the very last section of the talk, I go into like each of the dev experience changes I made and how it affected token utilization and overall cost and the error rate as well. Because accuracy, you know, we're in index. Accuracy is the most important part for me.
I wanna make sure that we're putting the right schedule out there. Cost or being some cheap doesn't really matter if you're wrong. Yes, cheap doesn't really matter when you're wrong. But also one of the interesting things I saw is as I pushed it more into the generators, I saw the overall cost decrease. And I think some of it was moving from right tokens to read tokens.
I can't prove that yet, because OpenCote just lumped them as tokens. But like some of the cost changes that I saw showed that using generators, using the MCT resulted in lower costs at even sometimes relatively similar tokens. One of the things that you mentioned was that you can do more with a smaller model when you've got strong tooling. Is that something that you've seen elsewhere outside of this particular project? Gosh, yeah, absolutely.
I just feel like when I'm doing logs and things like that, I also feel like so many companies have these little custom, we use factories this way, our tests look like this. We can figure this way and I can't help but sitting there thinking, like if you could use a generator to create your company's code, your best practices, your standards, you're going, uh-huh, you're like, yeah, that's gonna come up tomorrow. One thing that's interesting about generators is it's the thing you learn is a brand new Rails developer. Then maybe you rails generate resource a few times. And I feel like eventually most Rails devs kind of use them as training wheels and then move on to something else and write it by hand.
Do you tend to use a lot of Rails generators when you write new code? Yeah, you know a lot of generators. Well also like, when you say like right at by hand, do you really mean copy and paste an existing example? Sometimes, but for me oftentimes I'll just blank file controller inherits from an application controller. Blank files terrify me.
Like that's the thing that generators always get me over. And maybe that's why I lean on them so much. So I look at a blank file and I'm just like, I don't know what to put here. But if you give me something to edit, something to modify, something that is imperfect but works, gosh, that's right, that's right, I love working in that space. I'm curious since you said you lean a lot on generators.
Do you tend to write a lot of your own custom ones or maybe even modify the ones that are stuck with Rails or do you just run like regular Rails generate resource? Movie events was the first time I started playing with custom generators. Okay. It is the project that like spawns this for me. And I see so many opportunities for it at work now.
Like we have a whole bunch of scrapers. And I just sit there and I'm like, we could generate a scraper to our latest standard instead of relying on like the previous examples and everything else that we've built. Instead of looking at outdated code, you'd be like, no, this is the standard. This is the best practice, build this every time. And this becomes really important with an LLM because an LLM looks at your existing code and wants to really match yesterday's template or best practices.
And if you've changed that, it's really hard to steer it away from that old code that we no longer use. Absolutely right. It is so big in the world of LLMs. But I think it's also big in the world of like humans. Like if I'm not using a generator, I'm copy pasting from an existing example.
And there's no telling which one I pick. You're kind of doing the same thing that an LLM would or I guess the LLM is doing the same thing that you would. Yep, you're right. I think there's a lot of knowledge that's kind of tacit on a team, especially if you're a smaller team and everyone talks to each other. So that we kind of paper over some of these things where just everyone knows because we had that one meeting or we happen to see that PR and as long as everybody sees it, we're probably okay.
And so we just all hold this extra knowledge in our head as a sort of unspoken rule. And something that I'm finding that LLMs are doing is that they're forcing a lot of those unspoken rules to become actually encoded into our systems. And that's great for the LLM, but it's also great for our teams. If you didn't say it, I was gonna say that exact same thing. So I'm a mom, I'm a two kids.
I'm an open source maintainer. I stream every Sunday. I am exhausted. And then I do all these talks too. And I'm tired all the time.
And like I cannot remember how sort of a works to use F.create factories, like all of the little details and like all of our custom little pieces, like the names of methods. In my 20s, I did fine with that. I could keep it all in my head. I could memorize all the things. I could be like, oh, I'm reviewing this PR.
These are all the things that need to happen. I am so relieved to have these skills to rely on. And like even before LLM speaking of thing, I started having to like write everything down to remember it. That documentation is so valuable. Cause it's like how much did you forget between the last time you did this and now?
Especially if it's something that you're not doing on the daily, if it's a, oh, I create a new endpoint here once a month or in your case, I create a new event. But the scheme has changed since last time because another contributor made some changes. How are you going to remember? Yeah. Well, and it helps to standardize things as well.
So like one of the things that we're working with right now is the Japanese community and figuring out how to represent those events. One of the challenges is like with Kanji and Herakana, like we can't cleanly transliterate them. And so keeping track of which parts we can support it in and which parts we can't support it in yet. Yeah, it's really key. One thing I'm kind of excited about for LLM's is a sort of rise in developer tooling.
This is almost like this burden that we've born as community developers where there's certain costs to building developer tooling. And we don't 100% need it because we can rely on that tacit knowledge, but LLM's can't do that. And so now we're forced to build that tooling and that tooling's also cheaper to build because we have LLM's. And hey, that's great for them, but also it's greater for us. Like now we get that tooling that we've kind of always wanted.
It is so cheap to do the right thing now. That's a tagline right there. I feel like a lot of times with AI, there's this push more and more and more faster, faster, faster. And I'm like, it is so cheap for me to build the right thing, to do a refactor. If I have a massive change that needs to cascade through the system, I can write one example and then verify as I go through fixing Rubocop or Herbalinter rules, you can just write a skill for that.
You can have six PRs in a day and you can have them all cleanly separated out. I love doing one Rubocop or world per PR. Yes, 100%. Right? And that's so easy now.
I'm a real stickler for breaking up your work into iterative chunks where each small piece is individually shippable. And I think that that is a skill that is only more valuable in the AJI. Absolutely. I see this push for people to be like, no, I'm gonna do a 35-LPR and everything. I'm like, okay, maybe you write that, but it's so easy for you to just be like here, the separate PRs that I would like this to be broken up into.
And then it's easier to review and you can review it in more depth. And so can the person who's looking at your code. If we're spending more time reviewing, let's shrink the PRs and make the review cycle easier. And the important thing though, is not just smaller PRs, it is individually scoped PRs. If you've got a bunch of little PRs, but I need to read the other ones and we're gonna understand this one, you failed at your job.
You're right. And I think a classic case of that is the sort of like horizontal versus vertical slices. If you're saying, oh, I'm gonna ship a PR, that's got a model, and then I'm gonna ship a PR that's got a controller. And then a ship PR that's got a views, at that point, no, this is a shippable independently. Anybody that's reviewing needs to see the other pieces to understand how it works.
And it just has a tingles mess. I'm curious, have you seen like the spider way of breaking down tasks? I've not heard of this methodology. Tell me more. Okay, I think it's spike, interface.
Nope, don't remember all of them. That's okay. Well, go, go, go, go, go, go, go, go, go, later. It's an acronym, it's delightful. I learned how to do it a while ago.
Don't remember the acronym right now. So I'm telling a path to completion. If you were doing a login screen or something like that, the first phase would just be successful. And then it would be like, what are the different paths that you take? And every single opposite condition would be a separate PR.
And so you get the full path, but you also get much smaller PRs that iterate and modify it. Yeah, but I think we've gotten really far off topic now. That's a fun of a podcast, right? The conversation goes where it goes. No, that's true, that's true.
Also, I don't know if our audience can hear this, but there's a lot of background music that's just popped off. We are in this sort of greenhouse in the middle of the vendor hall at Rails World, sponsored by Buzzsprout. And it's a vibe. It's gorgeous, the glass walls. You can see everybody moving around.
Like you're right about the greenhouse cleaning, there's like plants everywhere. The entire venue here is gorgeous. They have string lights and barrels and doors and like green astrotour for old out, I think. It's a lot of fun. It is, it's so much fun.
Well, and I'm so honored to be here on the bike shed. I've listened to this podcast for years. And so it's really exciting for me to finally be on the show and for it to be here. And like the ultimate kind of podcast experience is just delightful and wild to be. Before we wrap up, I'm curious.
If you're thinking about how to be more ambitious with Rails generators, what are some of the next things you're willing to try? How are you trying to sort of push the envelope in terms of how you use generators to automate your workflow? There's this really interesting thing you can do where you can use say to like ask a question of the person who's like working through the generator. And so one of the things that I want to do is I want to make it possible for a contributor to have a like little conversation and like answer questions from the generator and then it generates the file at the end. Ah.
So almost like what we're starting to get used to with an LLM where you have like ask me questions before you do the thing, but old school built into a generator. Old school built into Thor like OG original conversational CLI stuff. Turns out that maybe the people writing some of that original tooling knew what they were doing. Do you feel like with AI, all of the old things that we learned, small PR is easy to review. Like there's been the recent thing about standard technical English that's cropped up.
Like what we're really finding is that software engineering practices continue to apply in the world of AI. Timeless. And in some ways almost doubly valuable. You were mentioning the breaking things down into smaller pieces. AI is really bad when it tries to do a little bit of everything.
But when you scope it down to one little vertical slice, all of a sudden it's really good. You were mentioning that thing about smaller models and better tools. That's amazing what you can do without just trying to brute forces with intelligence. Yeah. And having the tooling reinforced it is really helpful.
And I feel like with the larger models, they're trying to take on some of the scopes. What feels like my purview? Where it's like, I know what I want you to do. I would like you to do the things that I have asked you to do in the order that I have asked you to do them. And it's like, I'm going to run a web search and like see if I can find the organizer for this and like go search for all this stuff and like search these websites.
And I'm like, no, that doesn't do this to me. Especially for a really scoped task, like what you're doing. Looking for an intelligence that is going to own the task and figure out how to get it done and all that. You have a pretty set script you wanted to follow. Yeah.
And like how many of our tasks look like that? How many of them are at additional logging and metrics to this, track the values? See like what does the bullet think of this or a prose pie and implement those? Like when it comes to the Rubocop rules we mentioned earlier, those are so, you know what I mean? Like there's a ton of breaking down large and challenging tasks into small, easy ones that we have been doing for like the history of software engineering.
Like I feel like my team lead skills are used all the time and like breaking down work and understanding how to make it simple. If I'm not generating a whole application. With the Rails generator? With the Rails generator. With the Rails new generator.
Ah! But yeah, if I'm not doing that, then do I need a large or one? Really, very by task. It does. And I'll acknowledge that.
I think on that note, let's wrap up. OK. It has been such an absolute delight and honor to be on the bike shed and thank you so much for having me. Oh, thank you for joining your live at Rails World.