Autoscaling is a simple concept. Automatically add resources when needed, and automatically shut them down to avoid paying for excess capacity. How hard could that be? If you've set up Autoscaling yourself, you know it's not that easy, especially using the native Autoscalers on platforms like AWS and Heroku. That's why you need JudoScale.
JudoScale is Autoscaling as a service, and they make Autoscaling simple and easy, as it should be. You can use JudoScale on AWS, Heroku, render, fly.io, and more. It's free for low traffic apps, and unlimited plans start at $25 per month. Autoscale on easy mode at judoscale.com. Hello, and welcome to another episode of the Bike Shed, a weekly podcast from your friends at ThoughtBot about developing great software.
I'm Sally Hall. And I'm Joel Kinville, and together we're here to share a bit of what we've learned along the way. So Joel, what's new in your world? So this is one of those things that's new-ish recently. Both you and I were at ThoughtBot's summit, which is our sort of off-site event where we get all of us who work remotely and spend a week doing things in person.
And this was held in Amsterdam, a city that is famous for its canals. So after the summit, I had an extra day to myself to explore the city, and I visited the Canal Museum in Amsterdam. It's one of those places that it's not like a big fancy museum. It's just sort of one of the classic canal row houses and then you go in. And the way that they've set up their displays is it's a lot of interactive dioramas.
They'll have a diorama of the city, and they'll have projectors, and they can sort of project video on top of that, or highlights in certain areas. So it's a very sort of multimedia show kind of thing. I've seen a couple of these in a few different museums, and particularly for smaller museums, I just absolutely love this type of display. I love a small niche museum, man. That sounds really fun.
What was your favorite thing you learned about Canal? The thing that was really interesting to me is how a lot of the canals were sort of developed in the early modern period. So after Amsterdam started getting a lot of wealth, they're sort of cramped inside the old medieval city center, and they're like, hey, we've got all these swamps around us. What if we kind of drained them and built these like luxurious neighborhoods that have canals going through them? So it's really sort of a planned expansion of the city where they got more land, built the canals, and then built these row houses that were meant to be for the wealthy along them.
And in a weird sort of way, to me, that parallels the development of Boston's Back Bay neighborhood, which is a similar thing where colonial era of Boston is a sort of cramp on this peninsula. There's this sort of stagnant bay behind the peninsula, and at some point, it's the city becoming bigger and wealthier in the 1800s. They just fill it in and build all these fancy row houses for the sort of new money in the city. And now it's a beautiful neighborhood that people like to go and visit as tourists. My question about those, because I have also had the experience of living in a city that used to be a swamp, Charleston, South Carolina.
Do they flood off? And it feels like with the canals, that's a pretty good drainage system. Yes, you have to have the canals for drainage, but also because you're close enough to the sea, you got to watch out for storm surges, things like that. In Boston's case, what they've done is, they've actually put a dam on the river, and there are pumps there so that if ever the tide raises higher than the level of the water in the river, but there's a storm and so the water in the river is looking to overflow into the city, they can just pump it upstream into the ocean. Yeah, I don't know what Charleston built, but when I lived there, the strategy was mostly memorized, which roads between you and where you're going are high enough to probably not be flooded today.
So the more engineered solutions seem pretty cool. And part of the engineering that's really interesting is that both Amsterdam and Boston, I assume perhaps Charleston as well, in order to be more solid in their building in the sort of swamp area, they drive a lot of these wooden piles into the ground, and then they build on top of that. And these are inert as long as the water table is high enough to cover the wood. If the wood ever becomes exposed and there's oxygen, the wood will start to rot and then you sort of get cave-ins. And so part of the issue is not just not letting the water rise into the streets, but you also don't want the water to go down so much that now the groundwater no longer covers these wooden piles in the ground.
Yeah, I've definitely seen beach houses on stilts. So it's interesting to sort of imagine a similar thing, but they're underground. So I got a chance to nerd out on, I guess, a little bit of history, a little bit of architecture, a little bit of geoengineering, how to great time. If anyone's going through Amsterdam and looking for something that's a little bit more low-key, that some of the bigger flashier attractions are in the con museum. So Sally, what's new in your world?
I have been spending some time trying to figure out why some reports that get generated in this app I'm working on were efficient last year unless efficient this year. For context, it's this project where we built an app that gets used for an annual event, but not really used between. And last year, this data import task worked fairly well. And then we made some changes since then to enhance some features and things. And one of the changes we made was case insensitive matching on importing.
So if I import some records that are for the Boston facility and I type Boston in all caps, it should still find the Boston facility, that's just a capital B. The first attempt at addressing that was using post-gresses I like to do a case insensitive comparison, which worked great. It found what we needed to find and didn't worry about capitalization. But then when we got into production and we're trying to import 10,000 records at once, that's a lot of slow-sequel happening there. It's too late for this year, but I'm working on trying to find a way that they'll happen faster next year while keeping the case insensitivity.
And my first attempt is sidetext, I guess, to how you pronounce it, which is the case insensitive text column type for Postgres. So I've just been digging into this, but it seems as though it retains the capitalization of the string you put in there, but always searches case insensitive. OK. Because one of the things that was hard or that slows down the searching using I like is that our indexes are not case insensitive. So we've got all these indexes that did a great job of speeding up the queries we needed, but then we slightly tweaked that query and now the indexes aren't working anymore and we're back to slow land.
I know it's just been interesting. I always get excited when I start to try to solve a problem and then remember that Postgres has already solved it for me. And I can just make it Postgres' problem. Love that. So I'm having a good time with that and a big fan of Postgres.
When you're looking at indexing for something like a like query, is that different than indexing for something like a where or a join? I think so. In fact, we don't actually need the like query. We were just using it to get the case in sensitivity. So I had a mental step between the I like query and the side text, which was just comparing the lowercase version of the what in the database and in the search string.
OK. So I actually haven't done much indexing on to improve like or I like queries. That's really interesting though. I imagine it would be a little more complicated than just a regular index. You know?
Yeah, I was just thinking like off the top of my head, I can't remember if you benefit from regular indexes for like, because you're querying on a sub fragment. Yeah. I don't think you do because I think you would have to index sub fragments as well. It might depend where you put the percent marks too, right? Yeah, that's true.
So if you're sort of indexing, let's say like alphabetically or something, we're like in a tree structure that effectively does that. Like I'm trying. And your percent doesn't cover the beginnings. You say, I want a fragment that starts with this and then ends with my pattern, then maybe you could use regular indexing for that. Yeah.
I think that's something I'd like to spend some more time digging into and thinking about. Index is just feel like a magical tool sometimes. Like, you know, you can over index something. But when you hit the sweet spot of indexing it just right so that the queries are useably fast when they used to not be. That's one of my favorite moments in programming.
There's a thought-on article that I appreciate that goes over a lot of the common index types in Postgres and when you might want to use them. Sometimes you want to specify specifically index with this structure of this algorithm, because you're going to be using it for certain types of queries. And I think that's a great place to start. It's a good thing to just refresh myself on every now and then just read through like, oh, yeah, this index type. Yeah.
One of my favorite gotchas for indexing is when you're doing a multi-column index. So you're indexing on, say, like name and location. The order of those columns and the index matters. It has to be the same as the order of the columns and the query. Because if you switch it, you're not going to get the power of that index.
So this would be the order of the columns in a where and expression or like in a select? I think in the where. OK. So you have to say where column one and column two not where column two and column one. Yeah.
That is a tricky gotcha. Which feels weird. It feels like it should be flippable, but it's not. It's always run explaining your queries, right? Yes.
Explain analyze and look at those query plans. Make sure that you're actually using your indexes. Yep. Sometimes you've got an index and the query planner does not use it. Yeah.
Yeah. And then you're just writing extra stuff to your database for no reason. You just got these almost like ghost indexes that aren't are being used. Ooh. Very spooky.
We are recording this on Halloween. So it is Halloween. It is Halloween. Speaking of which, want to tell some dev horror stories? Love to do that.
I guess we are recording video now. I should have just brought a flashlight and just sat around with the ghost face. I have a lot of random things in my office, but none of them is a Halloween costume close by. Yeah. Do you have a dev horror story you'd like to start with?
Sure. My very first professional job as a developer. I don't remember exactly what I was trying to do, but I feel like I had just learned a way to do like several command line things in order, like chain them together. And was trying to do something to either my local environment or staging, but screwed up the command enough that I ended up just dropping the whole production database. Did it write a passage?
Yeah. Our backup was pretty old. Not super old, but it was like 24 hours old or something, longer than you needed it to be. But we did have good logs. OK, so I ended up spending quite a while with the recreating from logs, recreating from logs every transaction, every database transaction that had occurred since the last backup.
That's a creative solution. But wow, that must have been painful. It was very painful. It was very painful. The reason I can't remember what command I was trying to do is because I never tried it again.
Whatever it was, it was bad and wrong. But I did learn a lot about the importance of both backups and logs. If we had had better backups, more recent backups, it wouldn't have been a huge problem. But if we hadn't had decent logging available, I would never have been able to rebuild all of that. But it still kind of makes my skin crawl when I think about it.
Have you ever dropped a production database since, or is that the one and only time? That was the one and only time I've ever dropped a production database. Not to say I haven't ever occasionally deleted some data, but never on a huge scale. What about you? Have you done the production database wipe Reddit passage?
Sometimes I've managed to avoid it so far. Wow. So I've gotten lucky. That's amazing. I do feel that data-related tasks are always way scarier to me than any code-related tasks.
Oh, yeah. I can push a bug after production. And it's a problem. Maybe some users complain. That's not a good thing.
But we put out a fix. We deploy it. We're good. But you corrupt your data by doing something wrong. You run a bad migration, or you tried to run a manual command or something.
And some things you do might not be recoverable. So I tend to be a little bit paranoid anytime. You're doing some sort of data migration, or even just in planning the schema of your tables, that kind of thing, because it has to be a bit of a measure twice once, because you don't often get a redo the way that you do with code. Yeah. Absolutely.
There's not really the same kind of rollback capacity. I mean, I guess in some ways there are. But in general, you're not going to be able to click a few things and have it go back to the way it was. Like you sometimes can with a bad code push. Make sure you have frequent backups.
Yeah. Yeah. And that you know how to restore them. So I've never dropped a production database. But I have definitely corrupted data before.
Pretty early in my career, I was put in charge of building some internal software that included a time tracking application for the employees of the company to track our time. And that was sort of what we were paid off of. The no pressure. Just like you're getting paid check. So basically, sort of a timer, you know, start, stop.
And then based off a few different things, like there were pay periods. And you could get to track over time, exempt status, all that stuff. My program would capture all of that, print out reports, and that over to accounting. And then they would pay people. Yeah.
I get a message from someone in accounting at some point. It's like, hey, could you make a check your program? Because this person on your report, they're slated to get an awful lot of money. And that doesn't seem right. And sure enough, I.
They looked into my code and there was a bug there. I think it was a combination of bugs where somebody had gotten a small raise on the same pay period that overlapped daylight savings. And so somehow, instead of doing the math correctly, I just double paid this person. And their hours counted twice. Yeah.
And yeah. And so then I get the message from accounting being like, hey, I think your program's wrong. Have you seen the 90s classic office space? No. OK.
Well, part of the plot, not just by a lot, but you've had decades business group of programmers who are disgruntled at their job and find out they're about to get laid off, write a virus to steal round off error when transactions happen to steal the fractions of a penny. Oh, yeah. Yeah. That sort of classic scam. But they screw it up.
And I think it's like a divide by 100 thing or something. And so it's supposed to trickle into their account slowly. And they wake up the next day and they've just got $30,000. So then they panic trying to give the money back without admitting what they did. So it makes me think of that.
So it's good that you had a human in there who could intervene and go, wait a second. This doesn't make sense to me. So I feel like there's a couple kinds of dev horror stories. There's the ones we do to ourselves that we just mentioned, where hopefully more often early in a career than late, we make a sort of disastrous mistake. But I feel like there's also the ones that come from non-developer expectations in your company.
Are you talking things like deadlines, like unreasonable deadlines? Yeah. Or unreasonable expectations. The one that I think of most often is the Thanksgiving I spent trying to help my boss log into the internal system that I didn't understand why he was using on Thanksgiving, going to Turkey. And he knew just enough to delete.
I can't remember exactly what he was having a login problem. And I kept trying to check the status of his user. But he was able to do just enough to make it look like nothing was wrong when there actually was something wrong. So it ended up being a really small actually case sensitivity bug again. But because we were trying to debug it over text him, who knows where, me and my kitchen trying to bake pies on Thanksgiving, it took hours and hours of texting back and forth to figure out what was going on.
And all the time really wanting to yell. Just go spend time with your family. We can fix this on Monday. But I couldn't say that. Yeah.
So that was the Thanksgiving that was ruined by case sensitive user name matching. What should have been a relatively minor bug? It should have. You know, it'd have taken your holiday. It was like cooking with a toddler.
They're just helpful enough to make it so much harder. Now that I would compare that person's intellect to a toddler's, but maybe their real ability was to other level. Right. Right. Is this person trying to go in and do things like the console and that's messing things up?
Yeah, they kept resetting their password via the console. That's tough, right? Because someone's trying to solve the problem themselves and not involve you. Yeah. Which is a great instinct.
Yep. But like I am going to try to be sort of autonomous and follow the problem on my own. But sometimes that can lead you to trouble. Yeah. It's a kind of situation that had it not been a holiday.
And we had both been in the office or in our usual remote work set up. Probably we could have communicated clearly and made it not a problem much faster. But that happening on a holiday was just the perfect storm of. Yeah. The worst trouble shooting.
Have you ever tried to debug an issue in production while switching between your error tracker, log viewer, and APM tool? Maybe you know how painful it is when you can see the error, but can't find the relevant logs. Or when you see slow performance, but it can't connect to the root cause. Scout monitoring is all in one platform, built to make your life easier, with log management, performance tracking, and error monitoring coming soon. All of your data will be together at last.
We'll go from bug to fix in one place. The result? Less context switching, faster debugging, and monitoring that actually helps you ship code instead of distracting from it. No DevOps team? No problem.
Scout works with your existing stack and gives you insights in five minutes, not hours. From bug to fix in one place at ScoutAPM.com. I know it was something I really enjoy doing whenever I work on startup projects is doing things that will empower the more admin team on the client, because they often have very, very poor tooling or no tooling at all. And a lot of things have to make a special request of the dev team to do things manually for them. Or sometimes there are things where they can fix themselves, but it's really painful.
So I worked on a project where instead of an admin panel, there was one giant text input with a massive blob of JSON. Oh, no. And the customer support people got to edit that. Oh, no. This was a project backed by MongoDB.
So basically, it was just you kind of get direct access to the table. Yeah, so there's not even really the same level of validation that you might expect otherwise. There may have been some validation on it, but yeah, it was a little bit dangerous, but also just a horrible experience for the customer service people who had to learn JSON as part of their job to be able to do it properly. And then had to fight with this form that had to, I don't know, how many hundreds of lines a day, son. I took a few days and built them a real admin dashboard.
I was like, hey, you can now edit users and trigger password resets and things like that. And that made their year. Yeah, I bet. Doesn't Amazon's identity management have some fields like that where it's just a big editable blob of JSON? I forget.
Although that's not a good service rep doing that generally. So a little more reasonable. I feel like to me, the whole Amazon ecosystem is a little bit of a horror story in of its own trying to configure it. I am user properly for a project is a labyrinth. I think it's actually illegal to do it correctly the first time.
Like, yeah, you can't know what he ever has. Nobody ever will. And it's the kind of thing that I do infrequently enough that it's just navigating through the UI and figuring exactly how to set up permissions and how can I give myself access to an S3 bucket? I was like, no, that's going to be an afternoon. Yeah, that's what it's going to take.
I had some of that this week trying to navigate. I haven't worked on a project on AWS in probably a year. And I've picked back up this once a year support project this week and was trying to get some metrics on CPU usage and database CPU usage because the big imports with the case insensitive matching were crashing. And I just felt like I was wandering around in like a warehouse full of other warehouses with where everything was mislabeled. And it took me so long, so long.
And I never did manage to really isolate what was causing issues via that and just sort of started going through the like, okay, what has changed since the last time we did this get the strategy of debugging, which was way faster. I'm not just using the tools that you know, I do feel a little bit of a twinge of horror every time I think about the admin panel side of the AWS experience. Yeah, but I think, you know, one devs horror story is another devs exciting project, right? Like I was rambling about my excitement about indexes earlier. That's probably somebody's horror nightmare of a Tuesday, especially someone who may be specialized with more on the front end of the stack.
Yeah. It's just like, uh, databases, I pay other people not to think about them or even someone who's just like really experienced in a very different database type. Another horror story that we're talking about database type was trying to upgrade a Rails app to three versions, one of those kinds of tools that was important, but didn't change enough that it had gotten fairly out of date. But for security reasons needed to be updated and digging into it found that it had been written forever ago when MongoDB was a brand new shiny thing. And there wasn't really an ORM for it for Rails.
So they were the realm cool, which is cool. And it just, you know, never got to the point where the priority was to swap that out or something, you know, more stable. So then we got to tip toe around that while upgrading several Rails versions. And the moment when we discovered that that was what was happening, when we got down into like, Oh, that's what's going on. Was sort of like the, I don't know, like a scary moment in a movie when you suddenly realize like, Oh, it's not all sunshine and birds.
Something dark is happening here. Sometimes those are fun on like a debugging mission where you're like, you trick tracking out a weird bug and you keep going deeper and deeper. Then eventually like you realize what the source of it is. And you're both deeply horrified and also kind of proud and like, wow, it's this really weird thing that was happening that's like hard to find. And it makes it validates the fact that it took me a whole day to track down this thing.
Yeah. Typically with like a one line change. Yeah, that's interesting. Now that I'm thinking about it, as I'm thinking through these horror stories, you know, some of them are mistakes or just like unpleasant situations. But a lot of the horror stories I can think of from my programming experience were also a little bit fun.
You know, I'm not really one for a haunted house or a horror movie in real life, but like a haunted code base. That's a good time kind of or candy. I did a sort of a bug hunt that was a haunted code or test suite where I was trying to figure out why tests were randomly failing. So it's a flaky set of tests. And I'm trying to reproduce them locally and it's I'm struggling to reproduce them.
And then I realize it has something to do with timestamp precision is coming back from Postgres. So when I get the timestamp from Postgres, which I think have a higher precision than time objects from Ruby, or maybe see the other way around, then I would get the potentially flaky tests. Whereas if I was getting them just directly from Ruby, the test would pass consistently. Then started realizing that the test always failed if the sort of sub second like decimal portion ended in certain numbers. Oh, that is wild.
I forget exactly what was going on, but some sort of like we're taking that timestamp and I think we were then turning it into a string to create like a suffix on a file somewhere, but there was a little bit of transformation that happened. And as part of that, I think it was interpreting numbers as octal rather than decimal. Okay. And so it was a weird thing where if it was like cleanly divisible by eight or something, it would truncate things in a different way than the test would fail because it was expecting for the behavior for other numbers. And so that's one of those things where you're just like, wait a minute, it's only getting bad behavior on like sub second fragments that are cleanly divisible by eight and then you're digging into it more like, wait a minute, this is an octal, which sounds like not like if you came to me and said, I have a bug that happens when the decimal portion of this number is divisible by eight, I'd be like, no, you don't.
What is that? You've isolated the wrong thing, but no, it sounds like you did. You know, I was able to like cleanly demonstrate, look, I can consistently cause it to fail and I can consistently cause it to pass. And so yeah, I was able to fix the bug and the test pass consistently after that. Was that a bug that affected users of the app or was it the kind of only really affects the test because of comparison things?
I think it only really affected the test. You might have gotten like a wonky file name because this was like a file name generation thing, but you know, if your file name timestamp had like half as many digits as you expected as a user, you probably wouldn't have cared. Yeah, I imagine that usually would just kind of skim over that and don't really register it. Yeah. Yeah.
So it wasn't a thing that would have caused user complaints, but it was incorrect behavior. Yeah. So it wasn't just the bugs in my test. It was a bug in the code and the tests were catching it. And it was showing up inconsistently again because it was time dependent.
So I feel like it sounds like at least two of our horror stories now have related to the difficulties of handling time, which I think is a favorite topic for both of us. Yes, indeed. I dealt with a similar bug last year. It was, I was trying to fix it on my very last day on this project. And it was a thing where like I probably would have stayed up until midnight and because I didn't want to walk away from that unresolved bug and it ended up being this weird combination of things where we had a controller action that had a like a round hook that temporarily changed the time zone that was used for queries.
Oh, no. Which worked fine until we started needing to group things by date, right? Because there are large chunks of the day, large chunks of time every day where multiple two different time zones are actually on two different dates. Right. So if you're grouping by date, you're going to get a different result.
You know, midnight on the East Coast or one I am on the East Coast on Friday is happening on Thursday in California. And it was so, so frustrating to track down and so confusing to see because we were looking for it was an appointment scheduling situation and we were seeing all of these free times in one view on the app and then you go to be a patient and try to schedule an appointment and there was like half or less of what you expected. And it was just this tiny little bit of like around this one controller action pretend to be in a different time zone. It was horrifying and exciting at the same time. Yeah.
One of those moments. I feel like we've talked a lot about sort of some of our own mistakes, some of the bugs that we've encountered, but we also sort of see quite a variety of projects because of our consulting roles. What would you say is one of the scariest things for you to find when you're onboarding to a new project? No tests. That would have been my answer as well.
And that is terrifying. Yeah. I think no tests, no documentation. And then actually I think the truly scariest thing for me to find when I'm onboarding to a new project is really unhappy developers. The team you're joining, they're all really unhappy and they all kind of hate the code base they're working on regardless of what the problems in that code base are.
It's going to be worse because everybody's so miserable. Definitely. Yeah. The interpersonal and organizational things are always going to have a major impact on the outcome. I was on a project once it was a sort of greenish field project, a very short engagement.
The client came to us, they worked with a different shop who sort of built a prototype slash V1 of this product. And then the client wanted us to sort of take it from like really MVP to some more fleshed outside of features. And this was an express JS project that had no tests and expresses very bare bones. It's kind of the opposite of the rails philosophy of batteries included. So a lot of the things that allow you to move fast in rails just don't exist.
You have to customize them or you have to pull in dependencies. So the code is a bit of a tangled mess. And there were no tests. And so anytime I tried to add a feature to it, things were breaking. And it was one of those situations where it felt like, Oh, this is to step forward one step back.
And we had a very tight deadline because of a tiny engagement sort of thing was like, all right, I got a time box one day to try to back through the lens and tests and get them to work. And I spent that one day and could not get that to work. I was like, I can't justify more time getting tests in here. So I had to finish the project out without tests and that project gave me so much anxiety. Does that still kind of gross you out when you think about it?
Huh, there's a little bit of me that was like, maybe that is the one time where I should have justified a rewrite. Oh, yeah, because the MVP was small enough that I think I could have rewritten it in a sort of test-driven fashion faster than all the other fighting that I did with the framework, probably in rails again, because of the batteries, included approach. That's the kind of thing that you often figure out at the end. Yeah. The things you know at the end that you wish you knew at the beginning.
And, you know, I'm always very skeptical of rewrites. Like we love, especially if you handed over a code base written by another team and you see that and there's a lot of things you don't like and the temptation is always like, Oh, the other team did a bad. We should do a rewrite. And that's bad impulse. We should not do that.
The other team was working on an environment you know nothing about. And once you get even there, all of their terrible decisions might start making sense. But I feel like that was a case where I probably could have justified a rewrite. Yeah. It would have been tight, right?
Because the deadlines were so tight on that that it was a, huh, did I lose so much time fighting the existing code base that I would have been faster to do a rewrite? I don't have the counterfactual. So maybe, you know, every right would have taken twice as long as I expected. And it would in fact not have been faster than it would have been better out doing what I ended up doing. But it really felt like coding with one hand behind my back.
And I felt like I wasn't able to deliver the outcomes I would have liked to have delivered for this client due to that context. How did you handle that? Did you just do a lot of manual testing for every change? A lot of manual testing and a lot of realizing, Oh, this one thing broke something else and then having to go back and try to fix it. Decent amount of also QA being caught by the client, which is you're not always a good look.
Well, that's good. Well, I guess it depends on how closely involved they are and how much you're working together. It's frustrating for the client to be constantly like, Hey, thank you for shipping a feature. But you broke this other part of the app. Like now log in doesn't work.
It's not a good look that you're sort of constantly breaking things. So scary on first approach when you when you see that, but then deeply frustrating to work in as a developer. Oh, I just remembered a really scary thing from before I was technically a developer. I was writing automated QA scripts. And this was long enough ago that it feels a little bit justifiable, but it's not.
We did not have a version control system. Oh, now that's almost scarier than no tests. Oh, yeah. Oh, no, it gets worse. I'm a team in India and we were on the east coast of the US of almost no overlapping time and they would just FTP their code to a server at the end of the day.
And it would just overwrite what was ever, whatever was there yesterday. So there was no change log. There was no version history. There was just you come into work and there's new code there. And good luck figuring out what's different.
I am glad that version control has now just become table stakes for industry. Yeah, the idea of walking into a company and having them actively developing software without version control is still terrifying, but seems really low probability at this point. There's a sense of almost like maybe lack of professionalism for a team that's doing that. And sometimes you work with teams where like you were saying you were not a developer at that time. I guess in this case you weren't the one who was doing that, but you maybe you're just new to coding, you're hacking some things, you're mostly maybe a product person and you're trying to get things done.
And so that's sort of how you set things up. Yeah, it's what you know how to use and it's the tools that you have and are familiar with. Yeah, and you take them to the limit of what you can do and then you bring on professionals that know how to work. So that's no judgment on you, right? But the point where you do move to professionals.
Right. And no judgment on that team. Yeah, it is good to move to a version control. Yeah. Oh, I just thought of a scenario where you might encounter a project with no version control.
Oh, no. Product person, vibe codes, an app for you. Nope. Nope. Just the word vibe codes.
Nope. Having to pick up a vibe coded app and do anything with it is certainly one of my fears. Yeah, maybe that's a horror for the future. I've yet to see that, right? I don't know if this is just a fact or if there's going to be enough of these where there will be a lot of sort of vibe coded projects that have sort of crumbled under the weight of their own complexity to the point where the AI just can't.
And then you go to professional coders to try to fix it for you. And then you realize sort of the tangled web horrors that are underneath there, including potentially no version control. Oof. Yeah. I mean, I've I've definitely encountered the gremlin that is mostly an AI generated pull request, specifically AI generated tests.
Pretty terrifying. It's like there are a lot of lot of things happening in this file and not one of them is testing the code, which is almost impressive. It's like, you know, catching someone who's trying to fail a test to not show that they're smart, but they like they're smart enough to pick the wrong answer for every question. And you're like, actually, I reverse caught you. It takes skill to do it this poorly.
Well, I can imagine in years to come, AI will bring us new and terrifying ways to to scare ourselves in a code. Yeah. But hopefully, you know, some of the old horrors may be fade away, right? Like we're saying, a lot fewer code bases are on version these days. Yeah. For a lot of different project types as well.
It's pretty standard to have tests now. I rarely see untested Rails projects these days. JavaScript world is a little bit more 50 50, but they are adopting type script, which provides a lot of the same benefits. Yeah. Yeah.
I think we are in a lot of ways moving in an integrate way as a discipline. But of course, we also open new horrors to come. Yes. Every horror is a learning opportunity and every new project is opportunity to ruin something in a new way with new technologies and also just, you know, on your journey as a developer. Yeah. Absolutely.
Both of us shared stories of like some of the horrors where things we did when we didn't know better. Yep. I wonder what I don't know now that will horrify me in 10 years. We'll see. Yeah.
So here's a question. Do you think that the horrors just get bigger as a more senior developer because your like scope of impact is bigger? I think they get harder to fix. Oh, you know, like as frustrating as it was to recreate all of the database transactions from that log, it was pretty straightforward. And when I was done, it was done.
But I feel like as a more senior developer, you're making decisions often that touch more things in more complicated ways. And so undoing the mistakes, touches more things in more complicated ways. Like, oh, this took you a couple of days to recreate the database for logs as a junior developer. But as a senior person who pushed for a overcomplicated microservice, microservice architecture is now taking a whole team six months to unwind that or you just have to live with it and your headcount is inflated 50% by what it needs to be for the lifetime of the project. Yep.
Exactly. Sometimes I long for the simpler mistakes, but well, on that note, shall we wrap up? Absolutely. Happy Halloween, everyone.