Hello and welcome to another episode of The Bike Shed, a podcast from your friends at ThoughtBot about developing great software. I'm Joel Kinville. And I'm Ajis Slater, and together we're here to share a bit of what we learned along the way. So Ajis, what's new in your world? Recently, at least as the time of recording, Google has released a new Gemini model, 3.5, and for a time before that, 3.1 was in preview and then recently released as well.
And the work that I'm doing with my client right now is doing a lot of retrieval assisted generation. So people will ask questions to this chatbot and will draw on the expertise of our clients team through that, through that RAG generation to answer those questions. And so we've been evaluating are these new Gemini models better for what we're doing, right? What is it going to deliver more on the metrics that are important for what our application is doing? And you'd think that with the newer models coming out, they're going to get better and faster, latency is going to go down and different things like that. But this was an interesting exploration
of how the choices that the producers of these models and the trade-offs that they're taking from going from one version to the next might not actually line up with what your particular application is looking for, because we found in the metrics that we cared about, these models were all performing worse. They weren't meeting the needs of what our application was looking for. So latency first went down a little bit, that's good, but then 3.5 it went through the roof. And so we could just rule that one out right off the bat. But a lot of the other things like tool calling and consistency and remembering its context or remembering the earlier parts of the session sort of went down. And they don't, as in Google, they don't really release what those
trade-offs are very meaningfully. So you kind of have to guess at it and sort of a lot of the different guesses across the internet. I mean, they range the gamut, right? But it might be that they're trying to cut down on compute or something like this. And so they're doing different sort of caching strategies with context. And there's a lot of different ways that they could have gone and done
that. But essentially what it has done has made it better for a lot of use cases but worse for our use cases. So we wanted to fully explore if there was something that we could do with the levers that we have control over to tune that and get it into where we need. But ultimately, it ended up being that Gemini 2.5, which we're currently using as the main model for this use case, is at a sweet spot right now. It's getting what our customers and our clients need from the application. And I thought
that was just a really interesting kind of proof or example that the trade-offs that the model producers are making are not necessarily always going to be improving or better in the newer versions for what your application needs to accomplish. Right, understanding what your needs are. And then what is this sort of general purpose product that you're using? Is it always important to balance? I imagine that for you, there's probably also an interesting situation where there's almost like a coevolution going on there between the models and the code that you're writing, the prompts, the harness, all of that, where if you are sort of tightly optimizing for the behavior that you want for your code against one model and then the model changes, then all of your prompt and code might not work as well with the new model. And so you might see a decrease. And it's possible that
you could get something similar with the new model, but you would have to redo the prompts and perhaps some of the code stuff as well. Yeah, we experiment enough across the range of the major models that are out there that we feel like we've got a pretty good prompt that works across the board and haven't over optimized all of that that much. But yeah, that's definitely something that you need to consider at any point along the way with something like this. Do you have like a fancy like eval suite or something like that? What do you mean by that in particular?
Yeah, so it's a we have evaluating, I guess, a model integration or prompt or something like that. And so what you do is you try to create somewhat more consistent environment and then you might run, let's say the same prompt against three different models and see which ones give you the best results. And some of that might be deterministic, some of that might have some evaluation that is based on an LLM, but it's almost like a test suite for it could be for a prompt, for a model, for a skill, things like that. That's what I was assuming you were asking, but I didn't know if that was a particular terminology that I just wasn't aware of. Yeah, we do. There is a consistent set of
questions that we kind of run through that are based on not only like synthetically generated questions, but also things that come from real production conversations that we can then with or both through a couple, like you were saying through an LLM sort of looking at it, but also just with humanize, see like is this more of the voice that we want to be doing and like the quality and we have different qualitative metrics that we're looking at through to those. So yeah, it's been interesting. It's been fun. What about yourself? What's new in your world?
So a couple of months ago, I was in North Carolina for Blue Ridge Ruby. I gave a talk there called state is the first decision you never made, talking a little bit about how the HTTP protocol is a stateless protocol. You don't carry any information from one request to another. And yet the experiences that we build are stateful across multiple requests. And the example that I gave was a shopping card experience. And so how do we as developers try to think a little bit more consciously about
how to maintain state, how to thread state from one request to another? And how do we make that decision something that we consciously do with the trade offs in mind rather than letting that just be downstream of some other decision? So just like, Oh, I'm hyped about reacting to react, that's going to have impacts on how you manage state. And ideally, the state and its trade off should be the technical thing that's driving your decision, rather than excited about frameworks. So it's a little bit technical back to basics, understanding the web and a little bit a call to sort of self reflection for developers when we're thinking about how to navigate trade offs, when we make technical decisions.
Nice. To do take a sort of wide approach of evaluating trade offs and that sort of thing, or do you have some sort of heuristics that you use to determine some of those upfront decisions about state? So in the talk itself, I did not give any like, here's what you should do, or let's walk through a particular example. It was more about creating that awareness. I did talk about a lot of dimensions you might want to consider when you're thinking about the trade offs. So obviously there's some
things around like security that you might want to think of. But even in terms of like the behavior of the app, you might want to think about resumability as a possible for someone to let's say start shopping and continue shopping later, potentially in another browser tab. Or what if they want to start shopping on their laptop and then they leave the home and they want to finish shopping while they're standing in line at the grocery store or something like that. And now they want to do it on their phone. Does their shopping cart move from one to the other?
Is it possible with the architecture that you've created? Do you ever need shareability? Maybe in a sort of family setting, I start shopping and then I want my partner to contribute to the cart or maybe I want them to take it and then like finish it. Can I share my cart with someone else? There's just a lot of little things like that that you've got to think about in terms of state and sort of how you want that to live, how to be managed that will have impacts on the technical choices that you make. There's a lot of fun things you can do if your state is all local in
the browser. But that has impacts on, let's say, shareability because it's all local in your browser. Can I send it to someone else? Can I send it to my phone? That might be a little bit hard.
Anyway, a lot of these things stack. You can guess and a lot of these mechanisms. But understanding and their trade-offs should be an important part of decisions that you make. Yeah, absolutely. Just from those couple of examples there, I could see myself making a decision or two that would have put me on the back foot trying to add it on top of one of those features that you're just talking about. So that's a really good point.
The thing that's new in my world with all of this, because it's a little bit old, the videos are just unpublished. Any of our listeners who are interested in digging a little deeper in this concept, the videos are published, we'll link to it in the show notes. One fun thing about this talk as well is that it's lifted from a chapter of my in progress book, the Web as a Platform, where I talk all about the paradox of state on the Web. So that was a fun, taking content I had in one form and then converting it into another. Since we've talked about the approach or process of making a talk or pedagogy or different things like that, can you just briefly talk a little bit about what was that like taking something that you had meant in created for a written medium and turning it into a talk?
For me, I think a lot of it was finding the narrative. What is the story that I'm going to tell? I think in a book, sometimes you want to be a little bit more complete. You want to make sure you handle a lot of different cases or you show a lot of different things. Whereas in a talk, you want to demonstrate, make a point, sort of peak curiosity and then move on. I think I tried
to pull back a little bit with some meta points as well. What I was trying to do with this, I had the material, but I wanted to take people who were already, for the most part, familiar with how the Web works and sort of allow them to reengage, reconnect with some of the basics and then maybe get them to zoom out just a little bit to get a mind-blown moment or, oh, I never considered this in this fashion or didn't see how this connected to a broader set of ideas. So one thing that we explored is the idea that state can be an emergent property of a system without memory. Because one way that you can create state on the Web is to just have the client and server sort of echo their data back and forth. And it doesn't even matter if you've scaled horizontally and you've got a bunch of different servers who have no context for the previous conversation because the entire history of the conversation, if you will, if you're doing a multi-step form is echoed back to the next server who's going to have that whole conversation. And so just by echoing back and forth, you can cause
something that feels stateful to emerge even though neither side is keeping memory. That's really interesting. Emergent properties of things is one of those topics that kind of shows up in a lot of different places. And I always really love seeing it emerge from whatever I'm talking about or trying to understand. Yeah, that's really neat.
I do feel like emergent properties often are like a hint that I've stumbled across something deep. The idea or the study of emergent properties or emergence in and of itself is such an interesting one. And it sounds like it would be like a meta study, but it's not necessarily because it is actually it's another property or it's another attribute of a system that although you may not have been looking at it in the first place is still as much of a property, even if sometimes you can't point to it. It's like, this is why this emerges. Whereas in your example, you can, but there are plenty of other studies of emergents that are they are this other thing kind of only visible when you look at the whole in a different way. What we wanted to talk about a little bit today was
graph and tree structures. And I was going to make a connection about the sort of emergent properties and attributes and qualities that you can find from a largely connected group of data or behaviors or relationships. And we'll just we'll just leave the listener to kind of insert their own transition there as they can. Yeah, so I've worked a lot with graphs and trees recently. I feel like they're sort of present in the background, but as web developers, we don't often have to do a lot of like traditional graph and tree work. It's not in the way that you would from like you would
expect from a data structures and algorithm class. But I got a chance to actually do real graph and tree work because I was in a situation where we had just a bunch of entities where you could sort of like have variants of them that were connected to each other. So like a contracting system where there's like some contracting that can happen sort of recursively for different things. And so you want to like get a sense of like the project of the whole, but then also like different pieces within it. And then like depending on where you are in that tree or potentially graph, you might have different abilities to see other parts of the project. But you might also want to do queries like
for a sub tree or sub graph, like what is the total cost or things like that? Oh, that's really interesting. One of the sort of core thing that we're manipulating in a in a Rails app. And so the question becomes we have this sort of pseudo organization. How do we save that to a database so that we can do more traditional cloud operations on it? And it turns out that the answer is maybe not as straightforward
as you think. Have you ever worked with trying to serialize a graph or tree to a database? I think the times that I can think of where I have at least with similar sort of relationships that you were just talking about there, it gets really complicated to reason about pretty quickly if you have these sort of sub joins and like recursive joins that can all kind of be into the same table or like types of table that those kind of relationships get really tricky in a relational system. And so I've been fortunate, I think every time that I've had to crack a problem like that, we've been able to kind of simplify it down a little bit or like change the way that we were thinking about it to get ourselves out of having to represent it that way. But certainly there were plenty of different problems and sounds like the one that you were recently working on wasn't something that you could abstract in a different way to kind of get those gains to fit more readily into the relational structure. So I'm really curious how you ended up representing that or working with that.
So what a little bit of a journey? You know, first thing that comes to my mind, I hear, oh, I've got a structure, I've got to save graph data in order to query it with graph algorithms. What about graph database? That is the wrong answer. Okay.
Almost certainly don't want a graph database. My understanding is that you typically want a graph database when you're mostly doing graph queries rather than just because the data structure you're saving is a graph. In particular, when you've got a lot of different types of entities with different types of connections, and then you might want to find a path from one node to another across a variety of types of connections or things like that. Or something that's a bit more structural like an organization. That's not really a good fit for a graph database. The classic way of
representing a graph is in a JC list. And so you basically just have a list of what all the edges are going to be in a table. So you say node A connects to node B, node B connects to node C, and so on. If you've ever written like mermaid syntax or graph is, that's effectively what you're doing when you're just doing all the pairs of nodes. The annoying thing for querying is that if you ever want to ask questions like this node, a child, or an ancestor, or a descendant of this other node, now you're writing recursive queries. And postgres has recursive queries and they're relatively nice.
But they're a little bit more complicated. They're not a thing that like ActiveRecord sort of has built in, kind of just a little bit clunkier than traditional SQL. So you might want to look at some sort of alternative ways of representing graphs. The nice thing about the adjacency table though is that your data is all sort of in a normalized form. So every sort of graph, every sort of true fact about your graph exists only once. So if your graph structure ever changes,
you only need to delete one row and you're good. You can't get yourself into an inconsistent state where there are two rows that are making contradictory claims or things like that. Okay. Or other formats that may be easier to query are not quite so normalized and so might potentially run into that thing where you have to, when you're trying to change one thing in your graph, you need to change two or three rows and keep them in sync. Do you have something that's maybe a little more specific, an example of those two differences in structure?
Yeah. So the adjacency list is really just Sounds like a join table in a lot of ways. Okay. You could think of it that way. I think that you would often do with something like that.
That's a this is an aside. This represents the structure, not the data. So in this case, all of our contractors all would have a contractor's table. They're not on this graph thing. And then you'd have just the organization or structure relationships or whatever as a separate table that just connects all the all the things. So in that sense, it is like a join table, specifically outlining
that the pairs of values for a graph rather than a just generic sort of many to many association. To your earlier question, though, there's another approach that's called a closure table. And what it does is it basically, this table will hold all of the sort of pair combinations between any two nodes that have a path between each other. So you might have like parent child, grandparent child, grandparent parent, all of those exist and they'll have a depth to them. So this is sort of a it's not quite combinatorial explosion, but it definitely grows a lot larger as a table. And you will need to change multiple rows if you ever modify some of the
relationships. Sure. So if, see, if I'm thinking about this, right, this would be if you have A to B to C, then you would have a row for A to B and B to C, and then another row for A to C, because A connects to C through B. And if you ever broke like B to C, you would also have to change that A to C correctly as well. Okay, I see we're going it's not entirely right. But you can
a little bit think of it as storing paths through the graph. And rather than storing segments, sometimes you might be storing groups of segments sort of as one connection. Right. Right. If there is a path from A node to another, then you might store a row for that, even though you don't, even though they're not like directly one next to another.
Sure. Yeah. I see how that would be helpful, because then you're not having to find each segment every time you need to know is a connected to C. You know, that's short enough. But if your path is made up of a lot more segments, then it's going to be a lot easier to find that one connection than it is to have to search through all of those connections to see if you can trace that path. And because of the utility store depth with each one, so you say there is a path
between these two and the depth is so many. So now you can ask things like, is this thing a direct child versus is this thing a descendant of? Okay. And you can do it all just like regular where clauses. So it allows for more classic active record style queries, but with some of those caveats. Yeah, a little more overhead, but it's a lot easier to work within an active record
mental model. Yeah, I guess it's a little bit more overhead. It's a little bit more complicated on right, but it might be a little bit easier on read. Yeah, exactly. That's a good way for it.
It's also more expensive in space because you're now for every connection in your graph, you've got perhaps multiple rows. So your table will grow faster. In this particular project that we did, we actually did a combination of both. We had the classic J&C list as a source of truth. And then we had the closure table as a sort of almost like a projection of that data that made it easier to query. Okay. And some of our logic was sometimes more complex. We would do a combination of both
tables. So you would say effectively say, give me a subtree to say, give me all the anybody that's downstream of this one contractor. And that is a simple query with a where clause in the closure table. And then we say, all right, now let's convert those into the equivalent nodes from the adjacency, this sort of relationship graph. And we'll do some more querying there. Because our graph is a
little bit more complex, it wasn't just connections. We had a little bit of metadata on a connection. There were different types of connections because of like different contract types effectively. So when you're saying really, I want all the anything like downstream of this one organization, but only of a particular contract type, and that lives on the adjacency table, because that's the one that has the sort of source of truth for what the relationships are, you have to sort of query across both tables. Now, maybe forgive me a little bit here, but it sounds like with the splitting of these ways that you're kind of recording this data and having to then combine those back together sometimes, but not all of the times. And the way that you're describing these linkages,
having a little bit of metadata or like different kinds of edges between nodes, where was the line that it was too far to actually take this into a graph style data structure? Because it sounds like you're solving for a lot of these things that you were earlier saying was something that's holding you back, or like not the right use case to go to a graph database or something like that. I think our sort of query shape was very static. And that we've got just this organization, and we're going to search through it. And in case we're going to filter by relationship type, and that's really the only dimension that is going to matter at sort of a graph level, sort of position in the graph, and then relationship type. Anything else is just standard stuff. Contrast
this with there's an article I read from Untapped, which is the sort of beer craft beer social network. And what they do is they they have a graph database that they use for analysis. And so what they'll have is they'll have a bunch of nodes that are people that are places that are beers, and then they create all sorts of connections between them of different types. So, Ajhi rated this beer. Joelle went to this location. Ajhi and Joelle are friends. And then what you can
do is with all of that, we're all sorts of really interesting sort of analytical queries on this. So you can say with two people who don't seem to know each other, what are the, what is the shortest path between them? Or how many different paths are there between them? Okay. And now you're like, oh, well, these two people should maybe get to know each other because they both like a lot of the same beers, they hang out in the same places, and their social network is only like one person apart. They
both have a mutual friend. Those are the sorts of things you can surface with a graph database, where it's really more about the different paths between things than it is about sort of more static just saying filter the tree. Okay. All right. Yeah. I can see what you're saying. You've got a
little more room to sort of improvise and reason about or find emergent properties of a graph when you've got it in this sort of more traditional graph mode, as opposed to knowing that there are a couple of different operations that you need to work on, and then that overhead of a different structure and a whole different, potentially even just a whole different database just for this part of your data set can be a lot more complicated than what at the surface seems like. Maybe it's a complicated setup, but it's a little bit feels like Senate and forget it once you've got these couple of modes of looking over your data. Yeah. Yeah. And the sort of types of connections and things are fairly static as well. Something I do want to just reemphasize is that both of these
tables, the closure and the adjacency list, don't hold the actual data for the nodes. They are only about the relationships. So we have a separate table for the contractors. And these tables are really just about finding out who connects to who. So at the end of the day, when we do our like fancy query, we're doing joins and we're doing all this stuff. What we would like to get back is a
list of contractors who match this particular set of conditions. We're just using these other tables to contextualize which ones we want. Okay. And so then are these tables also going to be what's giving you say associations or things like this? Is this what you're using for that kind of what you might think of as like traditional active record or relational database joins? Or is this
purely used for that other type of query that you're just talking about? It can behave effectively like an association in a lot of ways. We're not doing actual associations, but what you could do is sort of hide all of that between behind some sort of like more domain models or domain methods that look and feel like they would be associations or just actions you might want to take on a thing. And it'll just sort of behave that way. So maybe you have a contractor dot subcontractors method.
And the implementation of it is it's doing all these joins and things. But from a user perspective, or from at least the person who's calling that method, it feels more or less the same as it would a regular active record method. Yeah, absolutely. I guess that is a lesson that I pulled away from that is that you really want strong sense of domain modeling there. Like what are, what is a contractor and then what are the like questions or commands you want to give to a contractor object? And like there's a lot of implementation stuff that has to do with graphs, but a developer
who's interacting with a contractor to build a feature doesn't really want to know about the graph that should stay hidden. That's an implementation detail and it should not leak into other layers of your app. Especially like the controller layer. The controller should not know that you're doing stuff with graphs. It should just say, hey, we've got subcontractors and we want to do something with them or we want to do this particular type of query and the model can expose that. Right. Yeah,
that's another thing that I love about the convention over configuration and that sort of approach that rails development will have is that if you can have this sometimes wildly different implementation, but it can still feel like something that's familiar and it can still work the same way. Like you're saying that developer that comes along and needs to tap into this thing doesn't need to know the implementation of it, but it can interact with that model the same way that might with other associations and things and get the same sort of behaviors out. And everything will just be sort of understood there right at the right at their fingertips. Now rails maybe pushes you in that way, but it's not given. That was the thing I had to like really actively work on. And sometimes it's not obvious. I sort of felt like those details were constantly trying to leak
out and I had to sort of take active time to sort of think about what I'm actually doing here and to really separate the sort of what is the semantics or domain behavior happening here versus the implementation and to try really hard to hide that. I think that's the fun part of building an abstraction for me a lot of the times is thinking about like when can you do that and when are you helping something and also like you're just talking about potentially is there ever a time when it might need to leak out because they're fundamentally doing something different and it shouldn't look exactly the same way because then they're going to reason about it wrong. One thing that I did think about in terms of implementation is could I get away with a tree?
So trees are a subset of graphs. They're a type of graph just with strict rules about them. Every node has a single parent whereas a graph you can have multiple parents. And it turns out that that like stronger set of constraints actually opens up a lot more specialized. Both algorithms for traversing but also representations when you see relies that are simpler and easier to work with because you have those more constraints. So that sort of paradox of saying sometimes
more constraints allows for richer expression. 100% creativity happens in a box. And one that I think is really cool that actually Postgres supports out of the box it's called L tree and effectively what you do is every row in your table represents a path from the current node you're representing to the root. So if you've got A's the root B is the child C is the grandchild then the path that gets stored on the row for C is A dot B dot C. And the fun thing with the fact that you're a tree a truth that comes out of that is that for every node in the tree there is a unique path back up to the root which is not true of graphs.
Right. Because you're not going to have any loops in a tree as opposed to a graph. Correct. And now because you sort of have those paths that are one to one to your sort of node data one thing that you can do is decide I'm not going to do a separate table for my node data like my contractors and a separate table for their relationships. I can put it all in one table.
So just say all of my contractors are going to have a an extra column that is just their path. It is going to be an L tree type in Postgres. And that's just going to be like typically like a concatenation of their could be their primary key could be some other identifier. And then Postgres knows how to do all the recursive stuff or whatever to like actually doesn't even do recursive stuff it just can do some string matching. And then it can very easily just look at that and say is this thing a descendant of or give me all the descendants of this node. And there are a few sort of built-in
operators that Postgres gives you for that. So that you're never sort of personally going through some strings slicing or anything like that. Isn't Postgres great? Postgres. But yeah if you can get away with a tree a lot of your life just gets easier. Graphs are really cool. But they have a lot
of fair bitches. Yeah, absolutely. Absolutely. Yeah myself not having a computer science background coming through a bootcamp and things. A lot of what might be these sort of fundamental data structure ideas and knowledge really only kind of come up as I encounter them as a particular use for what I'm working on. So as a lot of these are kind of sprinkled throughout my career of
learning about them. They weren't this sort of upfront learning and knowing about these tools. But it's been sort of peeling back layers or learning about like diving deeper into the tree of knowledge about these things as they become useful in a particular situation. And yeah so a different kind of look at how all of these things work together. But every time that I encounter one of these it's really impressive. It's really amazing to me how many problems can be reduced into or
like fit really neatly into a tree or graph data structure that you wouldn't or not even actually the data structure but just like a mental model that kind of works out this way that like can help you reason about or solve something more readily than if you were in a maybe less structured way of relating concepts together. So I really do like uncovering a new kind of tree or a new way of looking at a graph every time it comes up. So yeah I think a classic one for us programmers would be the fact that the languages that we write we see them as just a stream of text in a file.
But when your interpreter or compiler is working with it it's actually turning those into a tree and abstract syntax tree. And that's how it operates on it. Yeah I think that's one of those places that for a lot of boot camp grads that having to think about how a compiler interpreter all of that works is not something that we have done until again some either curiosity gets under our hat or some problem of significant enough proportion makes us look at it. But it inspired a little bit maybe by the 500 episode we had recently I do remember that I think the first time that I was learning about like abstract syntax trees. Wow is that hard for my mouth to make those sounds came from early on episodes of the bike shed listening to Sage and Derek talk about I'm sure it was something having to do with Rust.
Probably. I do want to give a shout out to Philip Otter, Matos Richard. I think you give a talk at Tropical on Rails that sort of was exploring the sort of how Ruby's interpreter works. And talking through the abstract syntax trees and sort of the different steps of the pipeline for compilation because yes there is a compile step in Ruby even though it's a dynamic language. So if that's a thing that our listeners if you're ever sort of wondered like I wish I had an accessible way of understanding like some of this magic behind how languages work that talk is worth watching. That's a link to it in the show notes.
That's a great talk. Wow we talked so much here about contracting and how to represent these sort of data structures in our go to of Postgres. There is so much actually to cover in this we might have to come back and revisit some other aspects of graphs and trees if people are interested. So let us know if you've enjoyed this conversation because there's a ton more to talk about here including there isn't a video for it but I will make a video for it of my recent talk at Haggis Ruby where I dealt into procedural generation of maps using Dragon Ruby and there's a ton of like graph traversal ideas in that whole world too that I got to explore a little bit in my preparation for that talk and a lot of resources that can go along with that as well. But once again we've gone really nerdy for quite a while so I'm looking at the time here and wondering do you think this is a good time to wrap it up? I think so we will revisit graphs because I think both
of us really like them there's so many cool things you can do with them. What is the best data structuring lines of graph? But I guess maybe for this particular episode I think where we found our center of gravity is the idea of how do you serialize or how do you represent graphs at rest? Yeah absolutely without having to break out of relational data this is right on that note shall we wrap up? Let's do it.