Skip to content
496: Test suite performance episode artwork

Mar 3, 2026 43 min

496: Test suite performance

from The Bike Shed Hosted by thoughtbot

Joël and Sally cover all the bases as they look at improving their test suite performances times. Our hosts lay out some spicy takes on various different test suites, comparing the key differences across the different forms of testing, where you might encounter pitfalls in each method, and how to make the most of each test. — Interested in exploring different test suites to see if they could improve your projects? Check out these articles on everything our hosts discussed today, as well as Joël’s talk on slow tests. Avoiding Factory Bot - Why Factories? - Parallelisation in Testing - Joël’s Talk Your hosts for this episode have been thoughtbot’s own Joël Quenneville and Sally Hall. If you would like to support the show, head over to our GitHub page, or check out our website. Got a question or comment about the show? Why not write to our hosts: [email protected] This has been a thoughtbot podcast. Stay up to date by following us on social media - YouTube - LinkedIn - Mastodon - BlueSky © 2026 thoughtbot, inc.

Episode details come from the podcast's RSS feed.

Embed this episode

Ready to play

496: Test suite performance

0:00 43:37
of matches

Transcript Auto-generated, may contain errors

Hello and welcome to another episode of the Bike Shed, a weekly podcast from your friends at the thoughtbot about developing great software. I'm Joel Kenville. And I'm Sally Hall, and together we're here to share a bit of what we've learned along the way. So Sally, what's new in your world? I had a fun time this weekend. My brother, who's one of my close friends and lives nearby,

asked me if I could help him with his portfolio website. He's also in tech, but he's a designer, and has just decided like, hey, I haven't touched this in a long time and I should freshen it up. And asked me for feedback. I gave him feedback and then ended up like wandering into his GitHub repo and looking at the code itself and giving him feedback. And then at some point he was like, I don't think that matters. I'm not trying to get it. I'm not a designer who codes.

I'm just trying to display the work I've done. It doesn't matter how well the website works. If it works. And I was like, but it does matter. I do care about these things.

Do you have a deployment pipeline? Right? Like your commit messages are weird. You might want to read this and squash some of this together. I think one of the things I said to him, I was like, you should add DS store to your git ignore. And he was like, what? But then, you know,

I started reading through the case studies and things like that on his website. And I'm like, okay, I don't know what any of this means. And it all looks very impressive. So good job. Uh huh. My mom thinks we have the same job, but we really, really don't.

Is it bad that when you first said portfolio, I was thinking like stock portfolio? Oh gosh. Although why would you ever want to website for that? Unless you're talking something. And why would you come to me for that? Website? Yes. Stock portfolio? No.

So yes, development and design or specialty. And you know, they're planning people to do both, but neither of us are really those people. What about you? What's new in your world? I've been on a project where I've had to do a pretty significant refactor of a subsystem with some really gnarly code that had been sort of evolving over time. So bits and pieces have been

built it on. And I've been really leaning heavily on LLMs to try to explain why choices were made. And they're surprisingly good at code archaeology. So you're asking it why things were made, like asking it to explain to you past decisions that other people made? Yes, specifically by using the git log. Okay. So I'll say, tell me who, tell me the why and give me a commit shot.

And it will do that to a certain amount of depth as well. So you know, it'll think for a little bit and it'll spit out a report and say, well, it was initially introduced to solve this problem. There was a race condition here. And so this kind of hacky fix was added in. And then somebody else was making a change over here and they merged it with this thing. And so now the like, hat has to play

sort of double duty because it fixes a whole here, but also is load bearing over there. And then somebody wanted to extract that into a shared module and it moved and it'll give me like the three commits. And I was like, that was really useful context. And it's the kind of thing that I could have figured out on my own because it's just running git commands and scrolling through some cruft that doesn't matter. But an agent in my code editor will do it near instantly. And so it took a

thing that was valuable, but that I didn't do as much as I should, because it wasn't worth the bother to now something that I rely on much more frequently. Hmm, interesting. I can be such a cranky person sometimes because I'm just like, but I love doing that myself. No one's making a not do that myself. I think for me, it was a little bit of a chore. The main thing is it's slow, right? I'm trying to

get context to make a decision. And now I'm going to spend the next 15, 20 minutes digging through the git log trying to understand something instead of getting a report within 30 seconds and then being able to make the decision, which is the thing I wanted to do anyway. Do you feel like there was any anything missing from what your LL and AD that would have been there if you had done it yourself or any downsides to not having done it yourself that you noticed? Not really. I feel like the LL and I actually did a more thorough job than I probably would have done myself. It went deeper and got more context. It was relevant, whereas I might have stopped at

that first commit that I found. It also did a better job, I think, following things through different file moves and things like that. I feel like I got a better outcome, at least on this particular refactor project. Maybe that's a thing to check in on at the end of the end of the season. So maybe that's a season three kind of thing, but hey, now that you've been using this practice semirregularly, do you find that has the accuracy and the like thoroughness that you need or is it like, oh, it was useful on this one project, but overall, not the best technique.

Do you then verify that what it's telling you is real or have you gotten to a point where you just trust it? I mostly trust it. Occasionally I'll do like a take the commit shot and do a big git show, that kind of thing, if I want to dig into something particular, but I've not had any case in the case where I have verified it, it's all been accurate. So I've not really run into anything where it's hallucinated commits or anything like that. That's good. It sounds useful for you.

And my guess is because it's working with a very small sort of set of data that's like already there, it's much less likely to want to hallucinate something. I can see that. Yeah. So yeah, I think the interaction of git and git history and LLMs is an area that I'm trying to tap a little bit more. So I think there's a lot of value there, a lot of value that I get as a person, but that's still a little bit sometimes slow and cumbersome. And I think that I can get a lot more value by

plugging more machines into the git log because it's a really rich source of context. Assuming you're not my brother and you know how to make these things git commit. Yes, yes, that is true. I would say I always write pretty in-depth git commit messages for humans to read. Yeah. It also turns out that commit messages that are great for humans to read are also really good

for LLMs. Another technique that I've used is if I want to load context into a session, sometimes I will just point it to a past commit and say read this commit. This will give you all the session that you need and then we'll talk about what I want to do. And because I've written my commit in such a way that another human could read that and get all the context, it also works really well for automated tooling. Yeah, I was wondering whether you write commit messages differently using these tools, but it sounds like you don't. No, no, it's really, I think what's good for humans is

typically good for an LLM. Any sort of like, why do we make this decision? What is this weird choice that was made in the code or why did I structure things in a certain way? Any gotchas, any maybe benchmarks I ran on change, alternative decisions that were evaluated but were we chose to not take that sort of thing. And AI is able to analyze a different sort of get a sense of what's happened in the diff better than a human can. I can stare at 1,000 lines

of diff and it'll take me a long time to get a sense of what's happening. And so I need, I often rely on the commit message to give me a little bit of a sense of the what in addition to the why, whereas I think the LLM really cares mostly about the why. And the human cares mostly about the why anyway. So making sure that's included in that and the commit message, I think, is what has the highest impact. And we'll have the highest impact on your colleagues. Yeah, it's the thing

that I push for the most on colleagues and clients is the, you didn't, you have a change, you didn't tell me what you did, you didn't tell me why you did it or why you went down this path and not another one. Can you please explain that decision? Yeah, those are so crucial. I find that, you know, a good commit message has prevented me from reintroducing a bug a lot. You know, like, why are they doing this this way? This seems like a very strange way to approach this.

And then you go to the commit history and you're like, oh, I see that they listed up all the things I thought they should have done instead as things they tried and didn't work and here's why. Uh huh. Well, would you like to talk about test suite performance? No, it's a topic you enjoy, it's a topic I enjoy. Yeah, yeah, I've done a decent amount of work on projects that required better performance for the test suites. I give a talk at RailsConf several years ago titled your test suite

is making too many database calls that was all about a particular aspect of performance, done a decent amount of work with parallelization as well. So I've touched a bunch of different aspects of performance. What are some areas of test performance that you've had to interact with recently? The test performance issue that I have dealt with the most tends to be like system tests. Assistant test is inherently going to be slower than a unit test, but sometimes we need them. And

so like when do I write it? How do I write it? Why do I write it? How do I make it less painful? Are you talking specifically typey bar driven? For the most part, yeah, that's where most of my

experiences. How do you feel about the testing pyramid as a concept? I think it makes a lot of sense. I'm trying to remember exactly how that works. It's the idea of like, it's like if we compare it to the food pyramid, like system tests would be like desserts. Right, right. I do think

that makes sense. I think I definitely, you know, many, many years ago, system tests were often the first tests I wrote. And I think that is less the case now. Do you find yourself writing kind of an outside in approach writing system tests first and then letting that push you towards unit tests or do you tend to sort of go inside out starting with unit tests and then working your way up to system? I think I do build. I think it kind of, I think it also depends on like, am I working on a whole

new feature? Right. In that case, I might do outside in more because like, what is this user interaction going to look like? Or am I writing a test because I'm fixing a bug and want to prevent depression? That's fair. What about you? I do also a mix. If I am sort of doing classic TDD, I tend

to prefer outside in, especially for new feature work. So sort of with the broad description of like, what are the parameters within which I am trying to build this feature? And then within that I can play around and come up with an implementation that does what I want and maybe parts that implementation need extra tests for themselves. So I do really like that approach. But these days sometimes I find myself testing after. So that can be sort of a little bit of whatever.

I find that, yeah, the idea of like testing a little after these days, I feel like sometimes that feels like a spicy take, I don't think it really should be. But I think the thing that most often will lead me to switch to like TDD if I haven't been doing it otherwise is going like, man, I keep doing these same like four manual setup things to see where my work is and to like work through it. I should really just put this in a test that I can run. The right to test this, I'm like, set up and then put a binding pry in it so that you can get exactly the state that you need to like manually check some things, even just the debugging technique. I really do like that. Yeah, that's a good one. What is your spiciest test performance take?

Because I know you've got some. I would say that it's almost like a Jason performance take. But one of the big causes of slowness in a test suite is going to be database rights. Typically, this is going to be caused by using factory bot. I think most people don't really understand the value of factory bot or how to use it. And so oftentimes,

end up making the wrong trade offs when it comes to performance in their tests, but also end up making tests that are really, really slow. I'm wondering if I am guilty of somebody straight offs. Give me an example. Let me put that question back to you. What would you say is the value of factory bot to you as a developer? Why do you like that tool? I like it because it allows me to focus on

the bits of data that I actually care about when I'm writing a test. So if I need to create a new user and I really just need that user so that I can log into something, it's annoying to me to be like, okay, well, a user needs to have a first name and a last name and a valid email and a valid phone number and all of this other stuff. And having to write all of that out myself every time is so tedious and makes the code so messy and can make it harder for someone reading the test to know what matters here. Right. So I really like using a factory to be like, I'm creating a user. The user's role is admin.

I'm not specifying anything else because nothing else matters. What matters about this user is the fact that they are in admin. Yeah. So it's an emphasis on locality. You want your setup data to be in your test, but you also have a certain like minimalism where you're trying to only have the data that's relevant to the test show up in your setup and nothing else that's just noise. Right. I think

another thing that I reach for factory bot for that makes that I suspect might be some of what you think people are leveraging the wrong parts of it is same situation. Okay, I need a user so that I can test this, but all users have to belong to an organization. And then there might be like another layer of that. Right. So every time you instantiate a user, you have to also instantiate an organization. Right.

And that is annoying. And I don't care about the organization. So I'm relevant for this part. So if I have a factory that does that for me, then I don't have to remember that in every single time run my test and have it be like, this user's not valid because this organization is there and remember to do that myself. But I think because we've talked about this before enough, because I've done this to myself, that is where it gets messy and slow sometimes.

Because every time you're creating a record, you're going out to the database and all of those round trips to the database add up so that can get expensive. A mildly spicy opinion is that your factories at least the base ones should only have the minimum number of fields required in order to define on there for that record to be valid, to be savable to the database. I agree with that. I don't think that's terribly spicy. That is official factory bot recommendations. Yeah. In practice, most code bases I've worked on

don't do that. Do you consider if you're making a factory for a model that has a belongs to association? Is that associated record part of what you consider the base, like, can you just exist for this to be valid? Can you save it to the database? No. If Rails will like yell at you for

not doing that, then I will probably put it on the factory. If Rails will let me get away for by not setting that, then I will try to not set that. So the user needs an organization because validation will fail without it and it won't save. It goes in the base factory. If a user needs an organization because nothing in the code makes sense without being able to refer to the user's organization, but for whatever reason, the user is still valid without one, which will pretend like that's a normal situation, then that doesn't go in the base. Correct. So the base is really the minimal set of values

that will validly save to the database. Yeah. I think I have that type of approach. I feel confident about that. That makes sense.

The idea here being that the part of the purpose of factories is like you said, it's hiding that noise of all the things that are not relevant. And let's say you didn't need to save to the database. Let's say you just needed a user in memory. You don't need a factory for that because you can just set user.new and set the one thing that you care about, maybe the admin flag. And then everything else doesn't matter.

It's not relevant to this test and you can execute your test. The reason you care about all this other stuff that's irrelevant to your test is because if your record needs to be saved to the database, active record will enforce validations, or maybe your database has not null constraints or foreign keys. And you cannot save to the database, even though those things are not relevant. And so what the factory is doing is it's handling all the irrelevant, but somehow also required stuff so that your test can only focus on the relevant things. I feel like approaching factories this way can also highlight sometimes errors in your validation.

Right. I've told active record that every user needs to have an email and I'm writing this test. And I want to create a user without an email. It won't let me. Right. Maybe that means that this

actually should not be a required field. Or sort of post validation because you can create an instant in memory. And it's almost anytime you're creating just an instant in memory, you almost never need a factory. There are like the build and build stubbed strategies that you can use if you need things like associations and all that build out. But for the most part, if you are testing an object and you're not persisting into the database, you don't need a factory.

I feel like I tend to default to if that factory exists using build or build stuff anyway. What are your thoughts on that? Like if we're not saving it to the database, then we don't have that particular performance hit. Depending on the strategy you use, there can be some like downstream things where it does save to the database, especially with the build strategy. So it's not quite as clean as you need.

But also it is a little bit more extra overhead for things that if you don't care about, then there's no sense in doing. Now what you might want something like that is say, you want a user instant in memory, not safe to the database, but it does need to pass validations. The fact that it passes validation is not super relevant to the thing you're trying to test, but ActiveRecord will get into the way. And so maybe then you'd use something like a build strategy. That's how I would tend to approach it. But I tend to view factory bot as a sort of

a tool I pull out when there is a justified need for it. But my ideal would be just calling user.new in the test every time. Yeah, so it's a tool you use, but it's not like when you keep it hand, not your favorite knife. It ends up being a little bit bad just because in a Rails app, the most frequent thing you're going to have to do is act on records that are saved in the database. And so that is something that I keep going back to. But if I can get away without

it, I would absolutely use just a direct call to the constructor in my test. I feel like a third person in this conversation might go, but what about fixtures? And so I'm going to pretend that they're a person that go, but what about fixtures? So fixtures are a little bit of a different philosophy for testing. But they can fill the same role as a factory. Yes, yes, there's sort of the

alternative to a factory. I should clarify that a lot of people associate fixtures with mini-test, because that's the default Rails stack. And I sort of like alternate stack that a lot of people like to use is RSpec plus factories. But those are all those pieces are all interchangeable. You can do RSpec with fixtures, you can do mini-test with factories. There's not necessarily

a tie-in with those two. Fixtures are just out of the box. They're faster. They are faster than factories and will almost always win any benchmark you go for. So if you were trying to maximize the speed of your test suite, fixtures are the way to go. But I feel like they're less good when you're

working on the test in the moment, for me, because now you've got two different files you have to deal with, or you're looking in different places. And you no longer have everything that you need to know about your test defined in the test. What's interesting there is if you're talking about the native and mystery guest data that will impact the outcome of your test, but that doesn't exist anywhere in the test file. It's a little bit of that spooky action at a distance. Where fixtures are also interesting is that the reason that they're fast is that they basically pre-write all of your data to the database. So all the data you need for all of your tests gets pre-written once before the test

suite runs. And so you only make one trip to the database, or maybe one per table or whatever it ends up being. And so that's really, really fast compared to making five queries per test or something like that across 10,000 tests. But what it does mean is that your tests are now no longer running with the minimal data set that you need in order to make that test pass. So your test isn't just saying there's an admin user and there's a normal user and they interact in this way. There's

20 different regular users and three or four different admins and maybe someone with some other intermediate role all present in that data set. And that may or may not impact the result of what your test does. So to be clear, I probably can't on one hand the number of times I've used fixtures. They're not really a part of my workflow at all. But the things that pop into my head when I think about working with them are things like say you're on a big team, right? And you're using fixtures.

And as I'm writing a test, I'm realizing like, I actually need this user to be a little bit different. So my options are change the user during the test as part of like part of test setup being like user.update name is actually Joelle and that's Sally. Change the fixture or add a different record to the fixture and use that one instead. My first gut would be like, well, this isn't a huge change. And I don't want to add a whole second record just for this. And it's basically

the same as existing one anyway. So I'll just, you know, update this attribute to the fixture and it'll be fine. And then I break all these other tests because none of them, you know, they will ring with one expectation for fixtures. So then I add my own second object in there. And now we've got 14 different people have all added variations of an admin user that could actually all just be the same record. Right, right. Your fixture files need a certain amount of

curation, which is certainly also true of factories. But factories at the very least you can have for your base factories, like pretty hard and fast rule that idea of like the only things they get set in your factory file are on your base factory are anything that's required in order to pass validation. And so that's like a hard and fast rule that you can sort of apply. I'm thinking lean doesn't require a lot of different judgment from different people on your team, as opposed to what doesn't count as a fixture requires a little bit more judgment. And at what point do you need to add something that was supposed to mind find something there?

Yeah, yeah. I think what happens with fixtures is they're a little bit of a shared story that your team tells. So you know that your sort of base layer has two admins and you know, maybe you name them admin one admin to maybe they are, you know, Joel and Sally, maybe even in this sort of shared story that you have, there's a bit of a persona to each of them and you're like, Oh, well, Joel is a really strict one who like bands everybody and Sally is the one who is more open to things or, you know, there's things like that to become a little bit part of the shared lore and can't act as a short hand in your tests if your team all sort of is in on the shared story. Yeah, because you read it, you're like, Oh, yeah, well, right, right, this test and somebody gets banned by Joel and then, you know, there's an appeals process and here's how it works. And you can just sort out a glance immediately like download a lot of information. Yeah, but it's higher sort of global context.

You have to know all of that. Yeah. And so now we're sort of moving out of just what is valuable for individuals or that a technical layer and talking a little bit about how do these technical systems interact with your team composition. Yeah, and different team cultures and makeups will definitely point towards different directions. Like if you've got a large team that changes a lot, it's not going to be reasonable to expect everybody to maintain the context of that story as they building. And people will be trying to all sort of modify that shared story and

diffusing that those updates to the shared story are a little bit longer. The large team team and a lot of churn, a team that is growing rapidly might struggle a little bit more with something like fixtures. Or a team where you know that one of their one of the places they need to improve is communication around something like this. Yeah, a small ish stable team with good communication patterns will probably get a lot of value out of fixtures because that shared story does have advantages. A way that you can maybe sort of short circuit some of that onboarding cost or churn cost is by using your test fixtures as your development data. And so you just sort of reset your development

environment every day or every other day just to the latest whatever is in that fixture file. And so now you can sort of explore that shared story as part of your clicking around in the browser day to day. And so it helps the team stay up to date with that shared story. So in that situation, your fixtures are playing the role of what could otherwise be your factories, but you can also use them as an alternative to your like seeds that are v-file where you're seeding the database with 70 million individual database interactions for your development environment. Yes, these are specifically development. Yeah, development. Yes. Yeah.

That's another classic mistake I think people do is they'll put a lot of like dev stuff in the seeds file and that's really meant for data that's required across all environments. So, right. So yeah, I really mean more like whatever you're using for your often we will use something like a seeds file for like a development setup. And you can use the same things for your tests and your development setup instead of having to maintain two different sets there. Exactly. Here's my spicy take. Not every test that you write while you're developing needs to

say in the test suite. Okay. So use tests, especially in a TD flow or even in a sort of test after kind of flow. I think both. But yeah, I think it's more often in a TD flow. Sometimes I will do this

when I'm testing a bug fix and I want to test a couple variants of it repeatedly to make sure that I'm hitting edge cases that don't necessarily need to always be tested in the test suite, especially if we're talking about a system test or something, right? So there's this one very strange case where a person does a series of actions and the JavaScript does something unexpected and it's a bug. And I want to make sure I can reproduce that and fix it. But and so I'm going to write a system test that does exactly that series of things as part of my work. But at the end of it, really what I end up changing is like, I don't know, like a small method on a model. Sometimes I find

that at the end of the day, really what you want to the test you want to keep is the unit test for that model and not this complicated and weird user flow system test, which is not truly what the bug is. It was just how we discover the bug or the context in which it happened. Yeah. I don't know. Does that make sense? Yeah. Honestly, every bit of code in your code base is maintenance cost and tests

have to run and they add some runtime to your CI. Yeah, if it's not providing enough value to balance its cost, then you might consider cutting it. Yeah. I'm curious to get your thoughts here. If I have this sort of little bit of a mental model that I'd like to get your thoughts on about facts, reason, fixtures, the idea is that testing with a fixture approach is a little bit of an integration test mindset. You're creating a sort of complete world with sort of, let's call it realistic data for a

full system. And then you're just running a bunch of tests inside that world, as opposed to more classic factory based approach, oftentimes emphasizes creating the minimal set of data that can recreate the conditions under which you were trying to prove something as true or false. And that's a little bit more of a unit testers mindset. Do you think it's very characterized the two sort of data tools is almost coming with philosophies that are a little bit different? One is more of integration testers, philosophy, and the other one's more of a unit testers philosophy?

I think it's a valid angle to view those through. And I don't think you're saying this, but I don't think that that's like a hard and fast if you're using fixtures, you're automatically you're the best you're writing or more like integration tests, or if you're using factories to test you're writing or more like unit tests. But it feels like a good shorthand mental model. So you asked me about the testing period, but I haven't heard your thoughts on the testing pyramid. The sense that I'm getting in this conversation is that you tend to lean heavier on the integration test or the system test than the unit test. Is that accurate?

No, actually, I do generally follow the classic testing pyramid. Okay. Yeah. I do tend to work sure from the top down so that like outside in approach classically, but I will have much more code in, I try to push things typically into models, and then unit test those. But I do like to open up by setting the parameters with a system test.

Recently, I've been doing a lot more tests that are code that is in a system where I'm building just an API. And so those situations I tend to do more like requests specs. And so those, I'm not necessarily spinning up Kapi Bora or like you're needing a browser equivalent. I'm just saying if I send each to be requested this endpoint, does it respond with this? Right.

So I've not tended to be in that top part of the pyramid or the very point of it as much recently as I have in the past. With your outside in approach, do you ever, you know, you start with a system test and as you develop and you've got these other tests going, do you ever then look back on it and realize that system test isn't actually valuable to keep? I have occasionally deleted system tests, especially if they're for like really sort of esoteric paths or things that I felt like wasn't providing a lot of value. Yeah. I want to balance that coverage, right? And that's always sort of

the challenge in that at the end of the day, what's important is not the Commonwealth well tested your units are it's that the software behaves as a whole correctly. Right. You can have a whole bunch of perfectly behaved units that do not cooperate. But having well tested units sometimes makes it easier to reuse pieces or catches bugs or allows you to test edge cases that are just too expensive to test at the integration level, or where there's just too much combinatorial explosion to test at the integration level. And that's the whole idea of the testing pyramid is you can have that volume of coverage at the unit level. I know that some people in JavaScript community prefer a testing trophy

over a testing pyramid. Oh, what does that look like? The idea there, I think, is it's a little bit more like a pinched middle, where you have a decent amount of unit tests and then a decent amount of end tests and maybe not so many mid level, like integrating a subsystem test. That makes sense because in theory, what you're doing in those middle tests would either be covered by the end tests or not really be adding value beyond what you've got in the unit tests in those cases. I think it might be a little bit also because some of those middle tests are really annoying to write. That is true. And so like if you've got just like utility functions, you can write

unit tests for them relatively easily. And it's a little bit more set up, but you can have a test suite that like pops up in a browser and clicks around and doesn't interactions. And that has a lot of value. But testing at the maybe component layer in React can get annoying because you're having like stuff about all these things. Yeah, I feel like I'm having flashbacks to trying to write tests of a component layer and react and experiencing exactly this.

I do want to take a couple of minutes just to talk about the idea of parallelization because that's a classic way of increasing your test speed. Have you done any work of parallelizing a test suite? I haven't. I have worked in test suites that someone else has paralyzed and I've enjoyed the benefits of that having happened, but I have not actually done that myself yet. So I'm curious to hear how you approach that. So if you're using any test on a modern version of Rails,

you get parallelization out of the box. You might need to fill in a couple of configs. I forget exactly how that works. But it is built with parallelization in mind. Our spec doesn't quite work that way out of the box. Typically you have to pull in a third party gem, the parallel test gem, and then it

will. There's a few different strategies that can use to parallelize your test suite. What's interesting is that parallelization is a more complicated problem than initially seems, in part because you need to figure out how to chunk your test suite. Lines of code are not a great metric for what is going to be fast and what's going to be slow. What you might end up with is a paralyzed test suite where one sort of parallel process is taking 10 minutes and your other five each ran for one minute each and then just sit around waiting for that first one to finish. So sure, you shaved a

few minutes off by passing some data or some work to those extra parallel tasks. But if most of the work is sitting on that one sort of unbalanced process, you're not getting the full value. Yeah. So are there tools that you can use to help you figure out how to chunk that up? So parallel test by default, it comes with use values out of the box. One of the things

you can do is use a timing file. So you run your test suite and it will record for each spec file how long it took to run in a sort of metadata file somewhere that it saves. And then the next run can read that and do the math and say, well, unbalanced, a file is probably going to run it about the same amount of time as it did last time. And maybe you made some modifications so that files can be a little bit off. But on average, that ends up being a pretty good estimation.

So we can use that to split up the work. And that works decently well. If you've made modifications, that can be a problem. And if you have what I've sometimes heard referred to as a tentpole files, so specs that are really, really slow, those are typically your integration tests, that might be a problem as well, especially if it's a real outlier. Because ideally what you want to do is take your total test suite time, divide it by the number of workers you have and each worker takes one end of your test suite time. You've got five workers you want each to take about

one fifth of the time. So let's say you've got a 10 minute test suite, every worker should process about two minutes worth of work. Well, what happens if one of your test files takes five minutes all by itself? Yeah, you're not going to get down to two minutes. Yeah. Question here, as someone who hasn't

done this, is a test file the smallest unit at which you can split up this work into parallel streams? So that's where the interesting question is, is the chunking things up. The easiest thing to do is chunk by file. Once you start chunking smaller than that, I think you start having to like play less into just giving file paths to processes and having to like read and interpret those files to get like Ruby objects from our spec and then pass the examples around so it can get more complex. So in general, you're probably going to be working at the file level.

You're probably working at the file level, although there are some things you can do to work at the let's say the example level. We've also been talking about sort of pre-splitting, using a timing file, something like that. You can also put all of your tests or all the files in a queue, sort them roughly by the amount of time. Maybe you have a timing file still, sort them all by roughly slow us to shortest, put them all in a queue and then every worker just pulls from the queue. Yeah, that makes sense.

And so that way, nobody's waiting doing nothing while somebody else has a big backlog work to do. You only get to take one file at a time, you process that and if it takes you longer than expected, one great. The other workers will pick up the other things. And so that ends up being more of a self-balancing process. So there are tools you can run to sort of have your own queue system. There are third-party services that will do that for you. Like Knapsack Pro is the sort of

more famous one. It just manages a queue for you where you're at a like kind of timing file and all that. So you just tell it, hey, I've got so many workers, so many test files and it'll assign files to your workers. That can be nice too because it's easy to sort of scale up and down. When you are parallelizing test suite, do you tend to sort of do the same approach for when you're running test locally and when you're running them on CI or are there variations in sort of what you think about for setting up parallelization for those two contexts?

I tend to try to do something similar when running local versus CI. At least I'm running a full suite for running a single example or just one file that I don't know. You can't parallelize one thing. And it's just not worth the overhead, right? Yeah. Which is another thing. Part of what is expensive with your test suite is overhead.

And so your Rails app takes a while to boot. It might take five seconds, 10 seconds, depending on the app. If it's a very large one, I've seen a minute plus to boot a Rails app. Every worker that you have is going to have to pay that cost. And so if you say, well, I've got 10 minutes worth of tests plus one minute of boot time, that's 11 minutes of total work. Split that across five workers, I'm not going to do 11 divided by five.

What I'm actually going to do is say they all pay that one minute overhead. And now each of them gets to have one fifth of that 10 minute test time. So total runtime is going to be three minutes per worker, one minute of boot time plus two minutes of test time for total of 15 minutes of total execution time across all threads, which means that you're going to start seeing some pretty heavy diminishing returns for your test suite as you start throwing more workers at this. And especially if you're saying, well, I'm not just going from two to three workers, but I'm going to go to 16 or 32 once your test suite time starts going well below the Rails boot time, then you're sort of like every worker is adding just a tiny little bit of increase to your test time while paying the full boot time. So again, with the example of the one minute boot time, now you've got something where adding a worker is like shortening your actual test run maybe by 10 seconds, but you're paying 60 seconds for the boot time. And so adding another worker doesn't

really move the needle much. And so that might be a time where you say I'm not throwing more processing power at it. I want to look at reducing my Rails boot time. So you might do some more lazy loading. There's a variety of techniques that you can use to make that better.

Yeah, that's a whole other episode. But there's an interesting map where you've got to do there where it's the answer to a slow test suite can't just be through parallelization at it. Yeah, like many of our problems, there's not a one-size-fits-fits-all solution here. One thing that will help you with your paralyzed test suite as well, make the total test time shorter. Right? That's another lever you get to move. Maybe by making fewer total database calls,

whether it's by improving your factories, switching to fixtures, using a combo fixtures factory approach, because there are a few strategies for doing that that we didn't have time to cover. But yeah, I would say in general, the levers you have to play with are parallelization, total test suite time, and sort of overhead. So any sort of extra startup cost you have to pay. Yeah, makes sense. So you pull on those three levers and the balance that fits with the trade-offs of your team and your app, and that puts you well on the way to improving your test suite times.

I feel ready. On that note, shall we wrap up? Yeah, let's do it.

Frequently asked questions

How long is this episode of The Bike Shed?

This episode is 43 minutes long.

When was this The Bike Shed episode published?

This episode was published on March 3, 2026.

Is there a transcript available for this episode?

Yes, a full transcript is available for this episode. You can read the complete transcript on the episode page.

Can I download this The Bike Shed episode?

Yes. Use the download control on the episode player to save the publisher-provided media file.