CICD remains one of the hardest things to get right in software engineering. But why? Rob Eris is a CICD expert, having worked in his field for more than a decade. In the early 2010s, we were teammates on the Skype for Web Team, and then Rob joined Octopus Deploy as one of the first engineers 10 years ago. In today's episode, we cover progressive delivery in practice, can I read deployments blue-green, and why future toggles are often still better?
What is GitOps, and why it's not about Git, and where that everything in Git mindset breaks down? Why should prioritize rollbacks less, and focus on roll forwards? And many more. If you want harder and less about CICD, progressive delivery, and what's coming as AI changes how much code we shipped to production, then this episode is for you. This episode is presented by Intissis.
Verify your systems correctness without human review or traditional integration tests, and avoid bugs or outages. Today's episode will be about CICD. CICD at scale is one of the hardest infrastructure problems to get right, and the teams from they'll know that the details very much matter. This is where I need to mention our season sponsor, WorkOS. WorkOS brings the same rigor as many of us use with CICD at scale to enterprise auth.
SSO, SCIM, RBAC, production-ready, battle-tested, and built to handle real-load and real compliance requirements. To add enterprise auth without the infrastructure project, visit WorkOS.com. Rob, it's awesome to have you here on the podcast. It's good to be here. I'm loving it.
It's been like, what? 11, 12 years? 11, 12 years? Yeah, I think, 2015, 2014, 2015, I think I left UK. And Skype, when there was still Skype, our team somehow inherited the outlook.com plugin, which had 400 million users per month or something like that.
It was crazy to mess up scale. This was an interesting job. Deployments were very much a case of you ship once a week and you have to go to a cardboard, a chain-device report. You have to get signed off in approval. It's always found out really weird.
We're building a piece of software. It runs on the way. We can ship it whenever we want. It was running on Azure at the time. We've got full access to push whenever we want to make these changes through the week.
But we have to hold them back. I guess to Adele, our manager, both of our managers at the time, we worked around the system. When the code was ready, we'd build it and ship it through the week. It was really impressive and proud of this process that the whole team put together where we'd commit the code, the test would run several layers of testing, it would go to staging, et cetera. Then it would get shipped to production.
We're executing a form of continuous delivery at the time. We would then ship ourselves once a week. I can't always like to tell the story that at the time, when we have a build ready to go, we do a form of Canary deployments. This is where you can roll out to a small percentage of your customer base. We always found that the customer base that would be our test objects was New Zealand.
New Zealand was always our canary. A bunch of reasons for that. The first country reached this new date. The first one is to roll out to it into a time. When the command passes, it's like 1 AM, first country is New Zealand.
Exactly. First country, significant signs. They speak English, so if there's any bugs or issues or reports, it's kind of easy to understand. But to be honest, New Zealand is small enough that no one really cared if we shipped a bug and had to fix it quickly. Sorry to all New Zealanders listening.
I think that's a good example of using a continuous delivery technique to ship the code fasten what we otherwise could of if we had these big bang releases. It's all processed, I guess, open my eyes to what progressive delivery, what good CI-CD could be. I guess from there, I spent a few years there. I've been eventually wife and I decided it was time to come home to Australia. Then back in Australia, when I started to work out to talk to the employer?
Eventually, I came back and actually worked at a place with a friend of mine just for a little while just to get back on the feet. I remember that we were using Octopus Deploy there. Octopus Deploy, for those who don't know, is a deployment tool was built and developed originally in Brisbane. It was a strong kind of Brisbane attachment to it. I found out that they were hiring.
I thought, OK, I'm going to give a big eye like CI-CD. I like this space. I think there's a lot of interesting problems in this space. I'm so glad you joined in. At the time, I was employee, I think, number eight or nine or something like that.
It was very much still a bit of a startup culture. Definitely not startup in the sense of Silicon Valley world parties and ridiculous spending. But startup in the sense that everyone who I worked with was an engineer, even Paul Sovell, CEO, he's an engineer. That's where it started from. We'd all be working on code together.
If someone had an idea, you'd have a chat about it and ship it. We were the marketing. We were support. We were going to be of everything. Obviously, the company's growing a lot since then.
The company was focused on deployment. They were focused on deployments. Can we talk a little bit on whenever I think about deployment? I always say CI-CD, continuous integration, continuous delivery. Why was there a focus on deployments?
Is that the same as continuous delivery? Yeah, interesting. So you're right. Quite often people talk about CI and CD as this kind of interchangeable. They're either interchangeable or the word is CI-CD.
It's not just attached itself. It's hard for me to imagine a CD without a CI, continuous integration. That's right. I guess the way to look at it is you've got multiple stages of maturity of software teams as they move on their way from initially CI, which is continuous integration. This is the idea that...
Initially it's YOLO. Initially it's YOLO. Initially it's right. You're only in the long-seas of what's prod or SSA-C-ing? We're used to.
We've all worked in places where we've done that. That's the same. That's why you're right. YOLO is the first stage. So you're not really in the continuous integration.
This is this idea where you want to keep integrating, merging your code changes into a single branch and you want to be continually running test again. Continuous delivery is the next stage where we talk about testing our code and unit test and iteration etc. What you also really need to test is your deployment process itself. So continuous delivery is this idea that you want to make sure that at any point in time when I click the button to deploy, I want to get to production once we get to this place. The next day beyond that, which not all companies necessarily reach, is continuous deployment.
So this is the idea that not only your changes being merged and merged together at the same time and ready to go, but they're also being shipped to production essentially. So the stages we have is first YOLO, then continuous integration, then continuous delivery and continuous deployment. What is the difference between continuous delivery and continuous deployment? The big difference I guess is the question of do your changes go out to production and automated? Does it kind of flow through without any intervention, I guess?
And for continuous delivery, they go out but not necessarily to production, right? That's right. So that's why you'll have environments like Dev Envirom or Testing or staging or whatever. Now, it's possible that some parts of the process may also still be manual. Maybe you only update the test environment once a week so the test can play around with it again.
But the key principle is that you could, you can push it through automatically the whole way through if you want. And what teams would not want to do continuous deployment? Because it seems to me, continuous delivery, you kind of want to get to because then you just get more and more feedback, right? But then it is a kind of good question like, should it go out immediately? This is the question everyone always asks, it's almost ready to go out, why can't we just push it, but I can put it in as engineers.
Let's go. It's ready, right? The reality is, it doesn't really suit every company, right? So it may be the case that some companies really do still have review boards where you need to validate is this where to go out. Particularly if you're in an industry that has a lot of regulation and compliance problems, compliance requirements, and they need to make sure that when it does go out to production, it's done at the right time with the right people available, et cetera, et cetera.
It's not necessarily true to say that everyone should be going to continuous deployment because that's sometimes just not viable for various reasons. But if it at least got to that point where you're continually seeing your changes go through all the testing, you're promoting it through the different environments, which is your therefore testing the process itself. If you can only click that button to go to production once a week or whatever, that's fun. You've done a lot of the hard work. You've mitigated risk, which is what a lot of this process is about, right?
Is feel the pain as soon as possible and de-risk anything that could go wrong right up until that last point. So I know you're deep into CI, CD, or continuous integration, continuous delivery, continuous deployment. You've been doing this for like what templates here is now. But I was pretty surprised to see that when I checked off to this deploy, it's that deployment is just continuous deployment, continuous delivery. But it also says Kubernetes.
How is Kubernetes kind of arrived in the topic of CI, CD, and general infrastructure? What happened there? Kubernetes is the platform of the moment. If we take a bit of a step back, Kubernetes came out of the Google, I guess, they originally had bought, you know, they were using it to host in one of the infrastructure. They ended up releasing Kubernetes partly.
I'm not going to, you know, pretend I can read their minds and know exactly why, but partly as a way of helping to level the playing field between them and some of the other cloud vendors. Yeah, so like before Kubernetes AWS was a clear leader, and I talked with Kat Kow's growth who came to the podcast, who works on the Kubernetes team, and again, she speculated that by releasing Kubernetes, it was a lot easier to move workloads from between AWS and Google Cloud. So it was a really kind of level to playing field. And now there was a reason to choosing Google Cloud was no longer as big of a risk or choosing Azure was not as big a risk and so on. Yeah, that's right.
It kind of made it simple to move between vendors. And so as a customer of one of these platforms, if you wanted to move to AWS and you're using containers, they probably just put a putting in a new place. And so Kubernetes came along at the time when there was a bunch of plays in the field for container orchestration. So you know, you had, you know, Nomad, you got a bunch of other options are out there because a core was a kind of high tower just on the podcast. They built fleet, which was another container orchestration.
And this was all around like 2020, 2013, 2014. And then Kubernetes came out and somehow started to win market share. Yeah. Yeah. I mean, I think the, some of the mechanics that they provided kind of really appealed to engineers and I guess DevOps teams out there.
And eventually, I think particularly because it was so easy to use cross platform and because some of the clapieners then did it on picking it up, it kind of as it ended up now being essentially the winner in the space. I know even back then, even non-contained orchestration tools like Azure service fabric was another kind of attempt to handle the fact that, you know, in this, well, they're one's building microservices and I want to host them in a single platform and how you orchestrate that and do with dependencies, et cetera. But Kubernetes has become the clear winner. And when you say winner, I understand that, for example, when you have a bunch of backend servers on a service, like, you know, we have a website, there's a large backend. Okay.
I'll use Kubernetes for that. But you're talking about your infrastructure, right? Or you're talking about even things like build servers. That's right. So it's kind of funny.
And we talk about Kubernetes being, you know, cloud native. This is what you're, this is the term you always hear. It's cloud native. That's what they say. That's what they say.
And, you know, you look at the vendors picked up. But otherwise, it would potentially be a management team in the opposite, it's just, you know, creating different tools for people, and those networks, and there's a lot of context.
You can talk about the infrastructure. And then we're going to talk about the connection team in the ops teams to just build and define kind of in that declarative fashion that communities provides, exactly what runs and how does it run, et cetera. So it's like this is the one thing is, it's a career in Minecraft, what is it called? And then what's the next thing, is you have a, you know, like in JavaScript, which is, you what infrastructure you want and what you're doing is essentially defining the desired state and then the tool kind of applies it and you know but puppet etc. And so Kubernetes has this similar concept right where you define what you want your infrastructure to look like and the internal Kubernetes controllers are basically insured or whatever you've asked for it always applies so if you say you want you know three replicas of something it will ensure that there's three replicas of something so one of those pods dies for example it will spin another one and so it simplifies this process of being able to find declarative declaratively kind of exactly what what you as a application team need to run your system.
It's fascinating because I always assumed that Kubernetes has won the cloud native space and and the cyber scalars can you tell me a bit more about how it's being used on-prem. I'm sorry because I'm interesting story you must have seen something because you said that you're working with companies who are managing like large on-prem Kubernetes or like interesting situations. Yeah this is one of the nice things about working at a company like Ocarpastro we we talk to and deal with so many different customers and yeah everyone's doing things a little bit different a little bit slightly different needs and requirements and you kind of get exposed to a lot of different you know problems and patterns and it's easy to sometimes to get lost in you know what people are talking about in conferences and everyone's saying it's all that cloud and this is the best practice and should be doing this and the reality is you know everyone's kind of got their own problems and they just want to solve them why they kind of need to solve them and so some of our customers in fact a lot of our customers will run Kubernetes kind of you know quite on premise so for example I was actually talking with one when you say on premise can you just be a bit more clear is this a data center where they're like renting and co-locating is this actually like I have my own data center or is this like I actually have my own machines in my closet.
Yeah yes and yes I guess so even in a closet I mean I was trying to joke there. I'm sure there are still things out there that are running you know the core accounting tools and etc. under Steve's desk but even when we talk about you know small computers some of our customers have Kubernetes clusters basically in their point of sale system so they have hundreds and hundreds of stores and they have little Kubernetes clusters essentially running them in each one's independent. They you know running to their own problems with that because particularly at scale when you've got you know thousands and thousands of clusters and you know these these customers are you know following various get-offs practices etc where they're pulling the actual state from from a Git repository so the Git repository itself becomes the bottleneck or they suck at any throttled and so they have to sort of resort to other mechanics to try to mitigate and work around that. I was talking to another one of our customers actually just the other day at Krigkam there who they are deploying they've got Kubernetes clusters running on research vessels and those research vessels as in boats as in ships.
Yes it is ships. That's right I'm not going to pretend to know exactly what they're doing on those ships we didn't quite get into that detail but they've got Kubernetes clusters out in the open sea right which is apt to give an excuse to name. The problems they run into though are a little bit different right so for them you know those boats might be out at sea for about weeks months or whatever that might be so when you want to do a deployment that the ship's not available so when that ship comes back and deploy it needs to get the update right so they'd be talking to you how you would how you'd achieve this or how that process would work. This is super interesting and I love how you kind of get a peek into so many different type of teams through the fact that you're talking with them with how they do the deployments but you're you probably see some other things that you're doing or things you're struggling with. What are some trends you're seeing across the industry in terms of the this wide range of companies you work from startups to finance companies like these research vessels?
Yeah I guess one of the big trends these days is a lot of focus on GitOps. What is GitOps? That's a good question. Let's take a step back from it. So we mentioned we talked about Kubernetes earlier and we talked about the fact that it's kind of got this internal continuous reconciliation process where you say to the cluster please spin up five pods and it takes that desired state and ensures it always sort of is true in the world.
And so there's a lot of products around that that we're doing some more thing, Terraform does that for infrastructure etc. And a bunch of people started wondering why can't we sort of take that process and pull it back further so that not only is Kubernetes just dealing with desired state but we can pull it sort of directly out of Git. And so I can as an engineer make changes to that that Git definition that desired state and I'd have some process that essentially pushes that to the cluster and ensures that it remains in line with what I'm asking and what I'm expecting. And so the term GitOps was coined by WeaveWorks in 2017 or so. And as a general practice it sort of started picking up the next theme particularly in tandem with Kubernetes because at its core Kubernetes is very declarative.
Later on in early 2020s it was kind of formalized a bit more and there was sort of four key pillars of GitOps. The first being essentially declarative you want your state to be declarative. So this is the idea that you want to define what you want the state of your infrastructure to look like. This is to basically makes things a lot simpler to understand what the state of the world is going to be when the deployment takes place. So if you think about deployments that are a bit more imperative that has sort of a process, the end result is sort of the result of multiple steps.
But when you want to just update some infrastructure that decides state and it works really well at particularly in the Kubernetes space. And then in GitOps the desired state will be just describing like how many nodes I want or like how many I don't know replicas do I want on a database or how many web servers or like low balance or how to be connected that kind of stuff. That's right. Yeah so it's basically a way of being going to say I want my infrastructure to have whatever state it is. And then the GitOps agents, GitOps products basically ensure that remains the case.
So they'll keep applying it to Kubernetes. So you've kind of got this situation where Kubernetes keeps its internal status in sync with reality. And now you've got these GitOps tools that take the declarative configuration instead in sync with what Kubernetes is. So they will take whatever I put in Git and whatever format I use and they can translate it into something that makes sense for Kubernetes and how Kubernetes can apply it. Yeah I mean ideally you want it as close as possible to what sort of Kubernetes is expecting because it allows you.
That's right. So what you're describing there I guess is the continuous reconciliation. And so this is the idea that these GitOps apps will essentially as we said sort of take that state and apply it. And if there's any drift from Kubernetes side. So for example someone you know runs group control, you know, delete pod or delete deployment or whatever case might be because your desired state is now stored in in gear going this case.
That will kind of self repair. The second pillar of GitOps is that the desired state you've sort of defined should be stored somewhere that's immutable and versioned. And so this is the idea that once I say that I want to have this state, I want to have sort of something I can point to a pointer and I might be a tag or a commit share or whatever. And I want to basically use that to define what that actual state should be. And I don't want that to be able to change right because otherwise I kind of defeat half the point.
By having it versioned and immutable, it also makes things like ordering a lot simpler, right? You can see the transition of that that desired state over time. What's interesting though is a lot of people will point to that and go yes versioned and mutable. I know what that is that's Git. I was about to say that because Git gives you definitely gives you versioning or it gives you commit history.
I'm not sure if it gives you're versioning and immutable in the sense that I mean the past can all be changed. That's that's right. Or actually can it because you can rewrite history. You're right. So you can depend on how you sort of configure your GitOps agent.
You certainly can rewrite history if you have a pointing a tag, for example, you can change tags. And so suppose best practices around that I guess kind of wiggle the finger a bit if you're using tags to manage that sort of state. But what's interesting though is really nothing in these pillars and very quickly that the third one being pool versus push. And so this is the idea that you're your GitOps agent will pull the state from GitHub and put it into the cluster and the fourth being continuous reconciliation. But nothing in any of these pillars actually talks about Git.
And I think that the naming of GitOps kind of gets people to already have this expectation that everything has to be in Git. I mean, why would you not have that expectation? That's why I assume that. That's right. I think the problem though is not everything should be in Git.
So you've got this constant kind of conversation within that community about where do you put secrets, for example. So no one can get. We would not know. We know it right as that. Do not put it in Git.
And so that's the thing. So there's been all these solutions to try to put it in Git. So there's things like secret where you can put it in Git. Sounds like a terrible idea. But I guess what's really this is highlighting is the reality that some things don't need to be in Git.
Right. As long as you can have this sort of control over the versioning or the mutability, then that's completely fine. And then the trend around GitOps is what you're seeing that a lot more info teams are moving from a few years ago. They might have just made the finishes for Kubernetes. And now they're moving over to GitOps.
So saying, OK, we'd like to control Infra in a tool in a way that's described as in version of GitOps. Is that the trend or what is the trend around GitOps? I guess it's more just the trend of the growth in general of GitOps in enterprises. Right. So not every company out there is in Kubernetes today.
And as they sort of approach Kubernetes and they're looking at, well, how do I perform the deployment? So how do I manage that process? GitOps becomes the fact that process. And to some extent, it is giving rise to this idea of using it to manage other things outside of Kubernetes. And there are a few examples of projects and experiments that will use things like Terraform and there's a continuous reconciliation service that keeps your actual state outside in sync.
At the moment, it's really about the focus is on, I guess, Kubernetes is the core place to release. And I guess it's more the growth of Kubernetes itself means that GitOps is coming along for the run. And you mentioned enterprises, which means like these large companies with oftentimes thousands of people or unregulated environments. So that's why I think of enterprises. Are you also seeing smaller teams pick up things like GitOps?
Is it like everywhere or is it more? There's certain types of teams that seem to be just more interested in this. That's a good question. So I guess sometimes what we see is a lot of people go to conferences or every blog post and they get it, GitOps is what you should do. So I guess what I want to point out here is GitOps is potentially not necessary for all all like agents, all environmental teams.
There's certainly a bunch of benefits to it. But the reality is there's some things you need to do outside of just GitOps. You might use GitOps principles in part to your process, but some of this absolutism, I think that sometimes exists may not be necessary. So there's often a bunch of other processes you do around your actual sort of, you know, quite a quiet deployment. So things like maybe you run some of tests or maybe you want to send a notification when it's complete or maybe you want to do a database update or something like that.
These kind of steps don't really lend themselves very well to kind of this declarative. Everything is in Git kind of process. Right. And so that's why you get things like I go workflows and roll out some things, come out to try to kind of get up-sify this process. And that works for some people.
But the reality, I guess, is that I think some people get really hung up on this idea that everything is Git. So therefore they've found the tool and so everything is a nail. I think there's a hammer in it. And this is the thing, like talking with customers, when we go through this process of, you know, you can use GitOps in Octopus and we've got a bunch of support for various mechanics that integrate well with Kubernetes and Argo. But there's a bunch of other sort of operations you do around that process that doesn't get.
And when you talk to them about it, you know, they realize that what they're trying to do is ultimately just ship software. So again, that difference between what you hear when you talk about conferences and things where you know, everything is, everything is Git and everything must be, you know, in this particular format or whatever the case might be. The reality is for most customers, they're just trying to ship software. I don't care what name you give it, if it's GitOps and it works end to end and solves everything. Good.
If they want to use GitOps as part of the process, but then have other mechanics that are most imperative, then good. It's just sort of the reality of, you know, there's tens and tens of thousands of companies out there in the world that are doing software delivery. And not all of them at conferences, not all of them at the forefront, I guess. As Rob says, most people don't care whether you call it GitOps or anything else. They just want to ship software and know that it works.
Our presenting sponsor, Anticipist, does exactly that, is less you ship knowing that it works. Anticipist goes beyond code review. It runs your whole system inside a hostile simulation. By doing so, it finds every bug before your users do. And because the simulation is fully deterministic, Anticipist doesn't only find bugs, it gives you a perfect reproduction of every issue.
I know this sounds like science fiction, but it's actually hardcore engineering under the hood. Jane Street, Fly.io, and the EdCity Community Ship Agent Britain code was full confidence because they know it's been verified by Anticipist. To see more case studies and details, head to Anticipist.com slash pragmatic. That's Anticipist.com slash pragmatic. I also want to mention our season sponsor, TurboBuffer.
TurboBuffer is exactly the thing that just works. A veteran full-text search engine built on object storage, fast, cheap, and extremely scalable, no exotic architecture required. Here's something I find interesting. The team is building the smartest AI products out there, cursor, notion, and tropics, and troppic. They all run on TurboBuffer.
But why? Let's think about it. An LOM without context, it can feel pretty dumb. I can still remember shortly after a chat TPT launch, early 2023, how it felt both incredibly smart but also frustratingly stupid. If you ask it a question that was outside of training data, it just made things up.
Fast forward today, and the models and their tools for retrieving context are much better, but hallucinations still happen frequently. Here's a typical way to integrate TurboBuffer with an LOM. Plug it behind your search tools, get fast responses, and relevant context back. The neat thing is how it gives you the perfect blend of low-cost storage, faster, and a bunch of different search tools like Vector, Full-Text, and Filtering. You can afford to index billions of documents, and then you can query it with different tools to get the most relevant handful of documents in milliseconds. This is how your LOM feels smart. It gets the right context really fast without
blowing through a bunch of tokens. Of course, you can use TurboBuffer to search anything, not just code. If you're building AI products, check out TurboBuffer at TurboBuffer.com slash pragmatic. With this, let's get back to Rob and talk about Progressive Delivery. Yeah. Another trend that we talked about just before is the Rise Up platform teams.
Can you talk about what you're seeing? So, platform teams are kind of, I guess in the past several years, they've become this sort of new standard organizational structure to help teams manage their, I guess, deployment workflows, a bunch of infrastructure around it. And it's kind of come out of this evolution of DevOps, right? So, we mentioned for, in the other days, you'd write a bit of code and you'd throw it over to the ops team. So, there's Dev teams and ops teams.
That was like in the 2010s, 2000s. Back in the long, long ago. And then DevOps became the practice everyone realized that actually we want to have the engineering team being involved in and have ownership of that operational process. Idea being fast feedback loops, you're able to, if you feel the pain, you fix that, you ship it. We've all kind of heard that.
And so, a lot of teams, you know, took that to heart. That's good practice. But as things start to scale up, what you'd find is that they would end up being like a DevOps team again. And sometimes, sometimes separate to another ops team. And so, there'd be a separation of development and DevOps and it kind of goes against some of the principles of what DevOps was trying to destroy.
But not only that, these teams then end up having lots of different ways of doing their deployment. So, you know, every single, you know, a whole bunch of application teams and they've got slightly different requirements and all building it from scratch. And so, you'd end up with these teams, either, whether you had the DevOps teams or was still within the application teams, where there was just this, this large number of different ways of doing things, right? And that becomes difficult at scale. So, you know, you can't really move between teams.
And by scale, you mean typically when there's a lot of teams, right? That's the easiest. Yeah, that's right. If you've got lots of lots of teams and each one is kind of owning that process end to end, you know, you sort of get this verification of processes. And not only that, the application teams themselves start kind of getting this context overload, right? They now need to
think about what's best practices of the different cloud tools have a bigger level. Devs rarely wants to configure the deployment scripts and test them and testing it's hard. You can't stop unit testable. So, it's now a different job. I remember when I was on earlier teams where typically on a mobile team, you have a mobile team of five people and one of them had to kind of specialize in Jenkins configurations because Jenkins is oftentimes the, or used to be the mobile CI CD. And it's kind of like half a person dedicated to that. And it was more like,
you know, like with the draw stick on who's going to do it? Because we want to build some... Right, you just want to focus on running code. And so, if you're spending a bunch of your time sort of managing infrastructure and pipelines and things, that's no fun for anyone. And so, platform teams have come about as a new way of solving that problem, where it's different to kind of decide you're a dev ops team or an all-process. They more sort of define best practices and they
provide a ideal self-service mechanism where application teams can essentially use often what's called as an RDP and an internal development portal. And they'll be able to essentially self-service and maybe they want to spin up a new project and they're able to use a template for the platform team that's generated. And so, the platform team are able to sort of create these standards throughout the company. And they can be responsible for the definitions of those processes and the best practices and how to achieve that. But the ownership of the actual running operational element is still within the teams. Right, so they still get those benefits of dev ops being close to the real code
and feeling the pain of the problem and etc. etc. But they don't need to spend all that time becoming experts in all the different ways you can deploy the software they've got. And so, this has become really common now where particularly as you sort of get to a larger size, platform teams are a great way of solving that problem. Now, that's not to say that every company everywhere should have a platform team. I mean, if you're a small company sometimes you've just got the
app statement and they're sort of doing, you know, quite a bit dev ops. But this is certainly something that as you start seeing larger organizations with multiple teams and multiple projects, these platform teams are way of basically bringing some sanity and control and focus, I guess, to the whole space. One trend across the industry, of course, is AI. Everyone's, it's hard to see any teams where devs are not using AI agents specifically to code, product managers will be using these things. And of course, we have a lot more code produced as a result. When it comes to
CI CD systems, what are you seeing changing there because of AI? This is the elephant in the room, as AI affecting the reality is, I think, to be honest, it's still very early. I think what will happen is the impacts of CI CD are really tightly coupled to how development teams end up using AI. So there's going to be some sort of, I guess, a lagging process there. But we're finding a lot of people, a lot of teams don't use AI in the development process. And so we're starting this
process of going out and looking and talking to customers and learning what's the way that they're handling AI in their teams and their application teams, and then how we can best leverage the CI side to support that. But then, in addition to that, use AI within the pipeline itself, again, in the right place. So one of the things we've been, I think, pretty keen on at Octopus is this idea that, we'll probably one of the few companies that didn't have AI class at all over it. We try to be very, that's what gets the sales. You stand out now these days. That's right, but not having AI.
We've got AI in Octopus, but what we've been trying to do is think about how do we actually use it in a way that's actually useful for our customers, for engineers, etc. And so we've been slowly adding capabilities within Octopus to provide AI support, whether it's a MCP server, whether it's a recovery agent that can review logs and tasks and that sort of thing. But that's within the product itself. Some of the bigger changes will depend on, like I said, how our actual application teams use AI. What I think we're talking about, we'll find, is there's going to be a lot more velocity. I think
that's one of the big changes, right? There's going to be a lot more code coming through. I think one of the questions is, okay, what does that mean for your pipeline? One of the things you often talk about when there's a human element to the pipeline is speeding up the cycle to get that feedback quicker. If you got engineers sitting there waiting for their codes to run tests so they can get back to it and fix it, the shoulder and shoulder you can make that feedback loop, the better it becomes, because I don't need context, et cetera. I think in a world where the majority of the code is being
developed by AI, that becomes perhaps less important. If you can kick out your build and test process, and it takes 30 minutes versus 20 minutes, does it really matter if the engineers already gone, moved on to the next problem and be the actual AI agent themselves can babysit the process and review the problem that came up and issue when you fix. I guess there'll be a de-emphasis, I think, on some of the speed of the pipeline itself, and more on decreasing risk, the risk that comes from having AI items generate code. And so exactly what that process looks like, I guess it remains to be seen, I think what we'll see a lot more use of is things like progressive delivery and I think particularly feature toggles are going to be a really common tool in the tool belt of application teams, partly because it allows you to ship that code as fast as you can, as fast as you want, but manage the rollout of the actual features, et cetera, changes sort of independent of the deployment, so decouples your deployment from your release.
And so in a world where we've got a lot more AI agents generating code and being involved in perhaps part of the build process, those agents themselves being able to use toggles to react to it quickly, I think then become a lot more important than perhaps what we see today. Can we talk about progressive delivery, what it is, and what are the most common ways to, you know, like to de-risk getting your code or your software out there? The progressive delivery is the next evolution beyond continuous delivery. So, you know, with continuous delivery, it's a side of it. You know, I made a change to the system and I want to ship it to the level of, typically, if it gets to production, sort of in one hit, right?
With progressive delivery, what you're trying to do is basically release those changes in a little bit more of a controlled way, typically through things like a canary deployment, so this is where you might deploy just some subset of your instances there out there. So, what is a canary? What is a canary? Canary deployment is this is New Zealand, basically, New Zealand's an actinary. So, this is, as we said before, this idea where you select some subset of your customer base or whatever that might be, and you would typically route traffic to a new instance. So, you ship, you know, you've got version one running and you want to release
version two, you essentially ship version two side by side and you might use, you know, most common one would be some sort of network traffic manager to route some percentage of your traffic to that new instance. And you gradually roll that up, typically, you know, as you do this process, probably you should have a fairly mature observability mechanism in place to see that you can roll up or roll down. And I guess this all thing comes from a canary in a core mine, right? That's right. Yeah. So, the idea being that, you know, in the old days when you'd be in a
core mine, being away and you would release, you know, all sorts of toxic fumes and things like that. Canaries were a lot more sensitive to it. So, they have a little canary in a cage. And if that canary sort of died, I guess, I think the canaries, as I understand, they were like tripping and then, okay, I was not tripping while it also died. Oh, okay. So, all right. Same
same ending, but just, you know, nice way to go out. These get out. Yeah. So, it's this idea that you get that advanced warning, I guess, that, you know, rather than you getting knocked out by the toxic assets, et cetera, you know, you can get out of this. So, it's that same principle, I guess, brought to the software. There's various other mechanisms like blue green deployments.
So, you've got your first version, this, you're receiving traffic and your second version is up and running and you can now do some tests against it, validate it. Maybe you've got sort of the, you know, the IP details that access it directly. You can basically validate it. It's working. Sometimes there may be a way of avoiding cold starts and things because that process may need to, you know, initialize a bunch of stuff. But then when you've sort of done that validation
and you're ready, you can essentially swap traffic around. So, all the traffic goes the other. In some ways, it's like doing it canary, but straight to 100% but you're doing a bunch of validation, sort of on the side before it actually reaches customers. It might be probably the more useful progressability strategy is featured goggles. So, this is the idea that you've got some feature files or like, feature goggles. Yeah. That's something. Oh, yeah. Often used interchangeably.
So, this is the idea that you've got, you know, some sort of variable in your system and it's linked to typically some sort of external service. And through the state of that particular variable being sort of true or false on or off, you can essentially have different code paths essentially take effect. And there's a bunch of benefits that feature toggles have over a secondary release. It's particularly for application delivery where, you know, your unit of change with a feature toggle is very granular. It can be, you know, single lines of code. And so,
everything else remains the same and all you're doing is tweaking that single line of code. With a canary or any sort of version sort of delivery or deployment mechanism, your unit of change is the entire app. So, if you've had, you know, 20 commits since the last sort of release went out, then you're essentially testing all 20 things in that one hit. Your ability to sort of target the actual customers, it's a lot more precise when using feature toggles. So, you can, you know, use all sorts of complex rules and say that, I don't know, everyone from Germany who has this particular product in their basket has this kind of experience. And that's, you know,
really hard to do via network traffic rules, right? The other is your ability then to actually roll back. So, to roll back from a canary, hopefully you're still in the process where you're sort of going through that canary processing and roll it back. That can take, you know, minutes, maybe you have to redeploy the whole old version. That could be minutes or more with a feature toggle, you know, you know, then seconds that's pressing a button that happens immediately. Not only that,
but you've got more control, I guess, on when you sort of do that. So, with a deployment you're doing via a standard version release, you're sort of tied to when that deployment takes place. Because when it takes place, that's when essentially on your feature is available. And as an application team, that means you need to know about exactly when it's taking place and make sure you watch in the logs at that point. Maybe you and 10 other teams who are shipping things at the same time are all doing the same thing. Whereas with feature flags, you've basically got control over when
that takes place. And you might ship the actual, you know, the assemblies and all sorts of things on the Monday, but you release your feature on Tuesday when you come in and you've got the logs ready and you you've got the next step. So, it really makes things a lot easier to decouple releasing a feature from deploying software. You know, version deployments through Canary, etc. They're really useful, particularly if you're doing infrastructure type changes where there is no kind of application toggle that's this relevant there, but you want to violate some changes to infrastructure or your process. Or potentially, you know, things like things that will involve schema changes and
schema changes that are the big, big, similar, schema change. This is the big problem in any like to be fair in any progressive delivery. And this is why the question always is, are you ready for progressive delivery? To do schema changes, I guess this is the point that application teams need to be really mature. And I mean, mature in terms of, you know, not telling silly jokes, but mature in terms of understand all the problems are in place with this and know how to release these sort of changes in a gradual controlled fashion and do it over multiple stages that, you know, ironically, it's actually quite hard for us. And Octopus with us, our software is
both SAS hosts. So we have a SaaS offering that customers can use and we have on-premise version. And it's kind of, because we have both sides, we kind of have the best and worst of both worlds. In the cloud system, if you've got a SaaS product, you have complete control over what versions go away. So if you want to do expand and contract, you can state to our process. You know that it's all
been updated before you come to the next stage. On the other hand, for a self-hosted application, where they go and they still look on their own infrastructure somewhere, you don't know what version they're running and what they're coming from. So they might upgrade from version one, straight to version six. And so you're not really forcing them to go through that expansion track phase. On the other hand, they've got a lot more control over when they upgrade.
And so you can kind of be a little bit more deliberate about, you know, making sure that they do back ups before they change and maybe they can manage that migration and accept a little bit more downtime during migrations and updates and things like that, then would actually be, you know, accept the limit in a SAS product. So one thing about, you know, we talk about progressive delivery and you're kind of doing this to avoid surprises. You know, if a regression goes out, a new bug or something doesn't work, you kind of want to catch it early. Hopefully only a few customers have experience that, or even if it's not 100%, you kind of, and you have a way to go back to all you do is you, if it's a feature flag, you hide it, if it's a carrier deployment, you go back to the other one. But there's also this thing where like when things do go wrong, at some point, you want to do a rollback. Can we talk about how have you seen rollbacks done well, and what does it take to
actually have a real rollback strategy? A bunch of people talk about CICD, some people talk about feature flags. I don't hear too much chatter about rollbacks. Yeah, rollbacks. This is always a spicy one. We get a lot of customers who say, why don't you
have a rollback button? They want to roll things back. Why can't we roll things back? As in the deployment software, like Octopus or anything else, they like, okay, if it can deploy, I want to like do, and like just do a rollback. Sorry, how hard could it be? Just do what you've all got?
How could it be? Tell me. Well, this is the problem, right? So in a completely stateless system, that's, you know, pretty straightforward. If you've got a completely stateless system and, you know, this is something that get up so it's really good that way, you'll have that definition. So
it's something repo, if it's completely stateless, you can do a, a get revert and push it and it'll go back. The reality is for most systems out there, you've probably got some state. Sleeping databases. Staping databases. It could be, you know, any sort of information that you can't necessarily just kind of undo, I guess, because if you roll back in there, you've got your code talking with the schema, the database that's not in sync, you can provide schema, my, if you've got a schema migration, let's say, in a normal deployment, you can provide alongside that a secondary sort of anti-migration that kind of undoes the change. But again, that's not always possible.
You need to do what you're going to do with that. We've gotten pretty far in basically trying to advise customers that you never want to avoid ever talking about rollback, it's always roll forward. So if there's a bug, roll forward, get a change in as soon as possible, this is where fast feedback loops are important, right? You know, this is about the hot fix process is a four, right? Like so we all know that in a standard process, you want to go dev-staging prod and maybe it's, maybe you've got, you know, approval processes and so on, etc. But if you've got a significant bug that you need to kind of
quite unquite rollback, sometimes the safest thing to do is actually make a hot fix to that version and push it out as quick as possible. And your bottom, like, might be the build pipeline or whatever, but depending on the viewer appetite for risk there, you can resolve that sort of a lot quicker. Now, obviously, if the failure itself is just from some mechanism in the deployment process itself or somewhere further down that chain, then your time to recover is just going to be a lot quicker. But it's this idea that, you know, if I've got a failure in version two, my rollback isn't going to version one, it's to go to version three and make sure I've got that fix in version three. It's the sort of thing that, you know, when we talk to customers and someone go, yeah, we roll back, you know, roll back all the time if there's a problem. And when you ask them, what do you do if
you've got a schema change, that kind of stop and realize that they've never, that it's just sheer luck that they've never sort of run into that, right? Is it fair to say that you want to roll forward if it involves business logic or something that is not stateless? Because if it is stateless or if it's application logic, you know, you have a code that says if this else then and you realize there's a bug there, you can just revert it as long as it doesn't touch the schema or the data. Yeah, I mean, in an ideal world, you're reverting is through a feature flag, right? You're essentially creating by changing the code path.
And this is why I always say feature flags are kind of a nice tool to use for doing this progress delivery, because, you know, it's just as easy as to roll out a feature, you can typically roll it back now, you're still going to have some of those problems with schema issues, etc. If you're making a change, you've got parts of your code path that expect one and not the other, you're going to need to account for that. But you can even account for that inside the feature flag. That's right. Yeah, so that's the way you sort of ideally manage that. So within regards to which
part to get down the feature flag, it's kind of self-consistent with whatever version of the actual database game that's out there. So I guess the more feature flags you use, the fewer surprises you might have, but it's a bit of extra workable to build and also to remove. You get stuck with still feature flags all across your code base when you start using it. Well, I saw this at Uber. Yes, yes, 100 times yes. So when you're adding a feature toggle to your map itself,
so we obviously use feature levels in our code quite a lot. And we use open feature as like the framework, the SDK to interact with it. But we essentially have built a wrapper around it, where the toggle itself within the code is sort of, we provide some details about which team owns it, and that team sets an expiry on it. Now, the expiry itself, when that time passes, nothing bad will happen. But three parts of the CI process is if that time is passed, we can send an notification to that team. So hey, it looks like the toggle is no longer used. So the specific mechanics don't
matter as much, but it's more a matter of making sure that if you're adding feature toggles, it's really easy to forget about it, because you start rolling it out and you kind of forget about it. And you want to keep it in there just in case for a while, in case you need to roll it back. Having the ability to understand how the Matull has been there is kind of a key part of helping to maintain that hijing. Now, the reality is even the doctorate is we've got a bunch of, I know I've got a bunch of net that I'm sure if I was to log in, I'd probably get a bunch of notifications to remove. We use that gardening metaphor in code, right? This is one of those
sort of operations, this is weeding, right? You need to just kind of keep on top of it. There are some mechanisms around, even in the AI side, which will ideally, if you're using feature toggles, you've probably got a bunch of observability and metrics and logging around it. And there are some tools out there that will allow you to keep track of when the last time a toggle was kind of evaluated. And that kind of gives you that signal. Similarly, you might remove it
from the code, because typically when you want to remove a feature tolling, you want to remove the code first before you touch your actual sort of toggle system. And so having a mechanism, so that once you remove it from the code, it might take two weeks before it makes all the way out into production. So you don't want to delete it before then. By that time, you've kind of forgotten about the fact you removed it. And so having mechanisms that will keep track of that change, I guess, going through the system. And when it reaches the environment where production
where it's actually being used can kind of go, OK, that code's gone out, that's removed the toggle, it's fine and safe to actually remove the configuration. Because you've got that feature to toggle information in two places, right? You've got it in the code and you've got it in your platform. Can we talk about how developing environments evolve? We talked about CICT, but I'm interested more in, you know, you go from like you have one environment, later you might have staging or something. And what evolution have you seen across all the
teams that you work with, all these hundreds or thousands of teams? Yeah, I'm not sure if there is one particular pattern there. I mean, I think most common is dev test prod. So these three different environments? Yeah, and I mean, even that, I think it's probably a gross simplification of all the back and forth. And dev meaning my local machine.
Dev in the case of CD is often like the first point of integration. So it's kind of test often customers will keep tests kind of reasonably in sync with let's say production or some sort of sanitize data source. So that way that whether it's the QA testers or the product team or whatever they can go and review the code, dev is almost like the first point of integration that is it actually is a deployment process just at it's core actually working, resuming fundamentally broken at all. I think more and more now we're finding that dev is less useful in that respect. And what we're seeing is more the growth of things like a thermal environment. And so this is the
idea that, you know, I as an engineer, I'm running some sort of feature on a feature branch. And I want to kind of evaluate that it's actually doing what it's what we're expecting to do. But not only that, I want the rest of my team to be able to see it working. And you know, if I've got running on my machine, it's not exactly easy to give other people access, I guess. And then I want to, I may want to completely context change move on to something something completely different. So a
family of a family of a family of environments is this idea from my branch pre merge, I want to spin up a whole environment, essentially from scratch, ideally with whatever dependencies required to sort of run this particular component of building. And I want to basically deploy my app into that as if it was a normal for fledged environment. As once that's available, I want to have access to a web app, maybe it gives me the URL and I can put it around it and I'll be able to evaluate. And then the moment I kind of merge that PR, tear it down again. It's quite common to have multiple test environments, because, you know, I've got a lot of stuff going through my pipeline and I've got three testers, so I'd have three environments. So I can all sort of have one at once or often
we'll see a single test environment and a bunch of tests and they will kind of need to collaborate to see who's got access to the system at the moment, etc. Whereas with the firmware environment, it doesn't roll off a tongue. With a firmware environment, you can essentially have a full fledged deployment per feature. And so again, that's about speeding up that feedback process, right? You know, all these browsers are all about speeding up that feedback process to get the catch those values or issues or bugs or whatever sooner.
There was a time a few years ago where cloud development environments were really talked about a lot, which was the idea that the developer you have an environment spin up in the cloud, you're, let's say, your visual studio code connects to it or maybe just log in online and it spins up all the dependencies, oftentimes, don't look containers, which reminds me of this as well. And there's also like preview environments, but somehow it feels that both that discussion and this one kind of died down. Maybe it's a I, maybe it's something else, but I mean, I told you, is there right we have containers that you can package things together. I'm sure it depends, but it's all doable. Yeah, it does get tricky. This is one of those sort of things that's really easy to talk about
for simple cases. It can get tricky when, you know, what if I've got more than just a single app in my kind of quote, quote environment and how do I make sure it's got all the data I need to value? So it can get tricky. So you know, if you have a bunch of services that have state. That's right. Exactly. So there are sort of complications that it does bring, but the,
I guess the benefits that you get as an application team, particularly, you know, application team where you've still got engineers writing code is sort of speeding up that feedback process, I guess. Well, now with AI agents everywhere, that's even better because in the sense that if one of the best ways to validate, you know, we have code reviews and AI generates and you look at the code, but isn't not better to just confirm that this thing works, especially when it has a UI. That's right. I think even in that world where you've got AI agents kind of building the code and validating the code, any sort of scenario where you want the AI agent to kind of validate what it's done, you're essentially talking about a family environment, even if it's not exposed to people, because it's doing its own testing and poking around in whatever shape or form that it's doing. That still is, I guess, one of these kind of environments rides it. It's a family
that's up, you've got some sort of provisioning process, and then ideally wants the jobs down, you kind of tear it down. I'm interested in learning more about the reality of operating a large infrastructure platform. And you know, one big one you're working on is actually Octopus deploys SAS offering. How does that look like? And what are the challenges of, you know, like running something where you're running all of these deploy processes, all these CD, you probably have a bunch of different things.
What does it look like? So at the moment, I'm not on the team that builds that, but I can give some of the context, I guess, from history and context there. Originally, when we first sort of decided to sort of provide a Octopus SAS offering, I think it was 2020 or something like that, it was all VMs. So every customer would basically get a VM spun up and we would have a virtual machine, and the Octopus self-installed, we'd basically get installed onto that VM and they'd get a whole VM for running workloads on, etc. And that was very much not cost effective. It was costing us something like 100 bucks per customer
per month and they were paying $20 a month or whatever it was. But this process was more an experiment to see was there a demand. And his credit pool was happy to sort of pass out the credit card to kind of go through this process to see that is this actually the direction we want to go? Is there something that's going to turn into a viable sort of direction for the company? Because it's a big step right going from building software that you kind of hand out and people then download and manage themselves.
It was a pretty much self-hulstered background on your own infrastructure, right? Exactly. Yeah, that's right. And so the demand was there. So not long after that first experiment, we basically started from scratch again. And I worked with a couple of the other engineers back then
to start building it on Kubernetes. And so Octopus itself in that space, we have what we call kind of a reef. So what you find is everything in Octopus, we've always got Octopus or nautical kind of names around it. So a reef is basically a way of it. It's this cell-based architecture where contains all the resources that are needed for that particular customer instance. Well,
some of it's shared, but it's kind of broken down into individual cells. And so a reef will contain the cluster and Azure database, etc. And each customer instance is running now in a pod in that cluster. And so as part of that project, that was when I think I was working on converting it, so I could run on Linux and inside containers. And someone else was building the dynamic work infrastructure. So there were a couple of us that kind of just got in and really just got it
up and running so that we could kind of start moving forward and I guess stop losing money. Fast forward to today, now there's an entire team that's kind of backs that and we've got several thousand customers on it. And we run many, many thousands of deployments every month. And so now what we're trying to do is there's a project at the moment to basically make the Octopus deployment process itself more resilient. So what that means is at the moment when a deployment kicks off, a bunch of the process as it's kind of an imperative set of steps, a bunch of that is stored in memory, which means that whenever we want to do an upgrade, we need to essentially stop running task for some period of time so we can kill their instance and spin another one back up. Octopus itself at
the moment doesn't sort of have zero downtime between upgrades. So there's a bit of downtime between that. I want to reduce that and get that as close to as close to zero as possible with the realization that you know, going from downtime of five minutes to one, that's just work, right? That's moving things around. You can maybe change architecture. Going from 10 seconds to zero is a much bigger shift. I'm
not sure if everyone will get there, but yeah, there's definitely this big effort at the moment to make the whole process a lot more resilient to basically improve and reducing the amount of downtime that takes place. So we can kind of perform upgrades quicker, etc. One interesting thing you do is you have a SaaS, but you also have an on-prem offering. What are interesting engineering challenges that come from that? A lot of companies have decided to just like honestly just move to SaaS because now they control everything centrally. I think sure that this or maybe they're doing it
which is a well known one, but clearly it's just a lot more work and a lot more headache to have both. Yeah, and we touched on one of the big problems here a little earlier is that when we want to push out any updates, because we control all process, we can push it out. So we have a gradual rollout process there because each customer is on their own instance. We can deploy each one individually. And that may take a few days till it's a roll out of change. On-prem though is kind of another
matter. So I was speaking to some of the stats around this a little while ago and found it took about 200 days, for on average 50% of our customers on-prem to get let's say let's say you can change today, it takes about 200 days for one average 50%. But then there's kind of like almost an exponential decay there where it takes 400 and something days for 75% to get it. So just it's kind of this curve where we've got customers that are still running versions of octopus from five, six, seven years ago. And so whenever we ship and you change, we need to basically make sure octopus will work from version 2023.1 to 2026.4. And so there's a bunch more baggage I guess that we have
in terms of like scheme upgrades and making sure that that whole process actually is achievable. But why do you do it? A lot of startups will be like screw it. Let's not support all versions. This even happens on mobile. What's the benefit? And this feels like you're kind of
swinging against the crowd with this one. The majority of our customers are still on-prem. And so this is, you know, you're talking about banks, financial institutions, governments, things like that where they want full control over the system. They want to run it on their own. They may use their own cloud or whatever to run it, but they want to manage the whole process and be in control of let's say, upgrades or downtime or things like that. So it's certainly not, it's certainly not
uncommon. And I don't think that's going away anytime soon. As for the upgrade support, we're kind of going through this process. Actually, in the past couple years, we've been getting a lot more, I guess, confident with deprecating features and things like that and just kind of cutting loose old capabilities. And then part of that has come from fully embracing feature doubles as part of that process. I think we're getting a little bit braver in terms of, you know,
removing capabilities that perhaps all the customers may miss. But I don't think that in the long term, suppose it will kind of go away. This is one of the things, again, where I think it's really common to hear, you know, everything's in the cloud role in the cloud. Again, the reality is there's a lot of companies out there where for them it just doesn't make sense or it's not viable or it's not, you know, it doesn't meet compliance requirements or whatever the case may be. Also, it's kind of a reminder, I think that you actually might have a lot of this competition if you build infrastructure software that also runs on-prem because it sounds like there's a demand where companies are like, we want to give you money in order for us to run on-prem.
And I'm sure some of them would do SaaS if there's no alternative, but for SaaS, it's easier to build anyway. So there'll be more competition. So if you're an entrepreneur or if you're software engineer thinking to do a business or start a business, it might give you an edge. Yeah, that's right. It sounds like a lot of your customers, you know, the ones who have not a grade of your software for, let's say, five years on one end, you can say, oh my gosh, what are they doing? But they might just be happy with it. And if they keep paying you as a business,
those are some of your most loyal customers. You see what I mean? That's right. And this is the thing. I mean, I remember when I worked in previous you talked about, or any of us who have any other sort of software that you've got running, potentially you've got running, likely, if it just works, why touch it, I guess.
And so it's kind of the bane of our existence because of noises that we want to ship with features and give them all these great new things. But on the flip side, particularly for something as critical as their deployment system, a lot of customers, once they've got it running, they kind of step away and go, okay, let's just let it let it be. And it keeps happening with AI as well in the sense that, for example, I just read the cursor, their latest coding model. It's updated, like, I think every five hours, which is amazing. It keeps getting better. However, there are customers who, once you have an LLM and it works for you,
you kind of tuned it, you have the instructions, great. But oftentimes what happens in new version comes out of a model or major version and it stops working. And I assume that there will be more teams, companies, businesses, where like, look, it will be worth money to kind of pin this thing or to run it on my own in front and just have it stay as is. And then I will decide when I want to change it as long as it, you know, if it's if it ain't broken, don't fix it. That's right. And I think to octopus is credit, I think we have a really good history at sort of
helping customers even when they're kind of on those other things. Sometimes to the extent of wanting to support team just, they don't all the instant, like, tell them to get a fixed upgrade. But support team are, you know, second to none in terms of their willingness to help. And as you said, if they're willing to pay us, who am I to say no? Yeah, I mean, it's a business strategy. I
think it's just a nice reminder that it's not just one size. And like, even though I think SaaS is eating the world and we're hearing and we're seeing it, it's nice to see that it's not just that. It's closing. If I'm a software engineer and I would like to move beyond continuous delivery, continuous deployment and go into progressive delivery, what pointers can you give me? Yeah, I guess just start with something. I start with adding one feature toggle. It may be scary at
first to kind of go, oh, it's in production. If I toggle this, I'm going to break something production. It's nice and comfortable to know that you're kind of well to the left of the running systems. And if you should code everything, it'll be caught by the test. But if I toggle it, it'll happen. It's kind of like a drug, right? Once you start doing it, you don't want to stop. And that's
why we've got this this hiding problem for things like these struggles, right? It's really easy to add them and actually end up with the opposite problem of how do you control yourself? How do you stop? So I'd say just kind of start doing it, add one and keep an eye on kind of as you roll it out and you look at the results from it. And the reality is, you know, I should feature the fun feature to do. And it's one thing to ship something and turn on a feature and go, okay, cool, customers have it.
It's a very different thing when you do the opposite of your ship, something and there's a problem and you can reach immediately for the toggle and switch it back off. You know, the amount of times you kind of pass, you have this kind of panic of, I don't know what's going wrong. And particularly when you're in that state, you know, maybe you've got called up at 2am because you've got an oncall and you know, you don't know what the next step is to do when you kind of got a panic mind and should I build a new thing or do I somehow force a redeployment? So having the capability of being able to flick that switch just allows you then calm right down and go, okay, I've stemmed the bleeding. Now come back and reanalyze it and understand what's wrong. So having that capability, once you've sort of
experienced that and realize the value that not just rolling things out but I guess rolling that individual feature back off, yeah, you'll want to use it for everything. What's one of the books you would recommend and why? I'll give two kind of, I guess technical ones and more of a fun Phoenix project is still for me. A good one. This is one that I can see, you know, you kind of remember that we got that in in Skype. This was one that Abdullah kind of gave to everyone. Or manager gave it to
everyone. Yeah, and you know, it's, you know, parts of it may be a little bit outdated and you know, some of the practices have changed a little bit at its core. This idea of as an engineer being involved in that whole sort of operation side of your what you're shipping and the value that gives to not just the company but to you is amazing. So I think that book has kind of a core, when it's one of those core foundation ones, it sets the, sets the context for everything we talk about today. And the other one from a more, I guess, organizational and communication side of things, radical candor by Kim Scott, basically to communicate more efficiently with more compassion with your peers and other people around you. It's, you know, really common, you know, I'm an engineer. So I know
sometimes it's really, you can't look back on what you said and you feel like, okay, maybe I can be a little bit blunt. Whereas radical candor teaches is to think about, you know, you want to have those communications that are both sharing that you're caring and empathetic, but also direct and, you know, the benefits of that and kind of the inverse of that where, you know, you perhaps, like I said, you're very blunt. You sort of being honest about it, but you're missing that empathy. So I found that book really useful and interesting as, I guess not even just as an engineer, but as a person working with other people. From the more fun side, basically anything by Greg Ian, he's an Australian sci-fi author. He writes some pretty crazy, mind-bending, hard sci-fi, so if you really into that, I'd say
read like Diispora or Charles Letter, there's sort of books that actually took a second read to get through and he's a mathematician as well. So, you know, he's got a whole bunch of background and mathematics on why a certain part of his story goes away. He's your own entire story on the premise of what if the speed of light wasn't absolute or something like that. This one premise kind of breaks out into it. And then this is what happens to energy and therefore molecules work like this and as a, you know, I'm a tech node, that sort of science stuff. It really appeals.
So you say when sci-fi, there's some science involved, that's actually way more fun. Rob, thanks very much. Thank you, Greg. What an interesting conversation. I hope you enjoyed having someone like Rob who has been building and thinking about CI-CD at scale for a decade.
It was such a fun blast from the past story. As you talked about how it's Sky, our team basically did continuous delivery years before most of the industry caught up and how we did it by quietly shipping new builds from New Zealand every week using this as our Canary country. It's a reminder that a lot of modern software practices were already being run in the wild bikes. We just wanted to ship software faster and our Change Advisor board would allow us to do so. One other thing I took a note is Rob's take on rollbacks. Lots of engineering teams talk about
rollbacks as of their safety. But the moment you have a database schema change in the mix, what's a team head? Rob's advice is to roll forward, not back, and use feature toggles as a way to turn features off or on. This is also a reminder that investing feature flags is usually really helpful. But if you have feature flags, be sure to clean them after them, after you roll them out, otherwise they become a big mess. Finally, a part where I learned something new was UNGITOPS.
I've always assumed that GitOPS was about well-get. But as Rob pointed out, none of the four actual pillars of GitOPS required Git at all. The four pillars are number one, the Claridev, number two version and immutable, number three, pulled, not pushed, number four continuously reconciled. But then GitOPS has caused the whole industry to get a bit dogmatic about putting everything into Git repo, even though they're secrets which absolutely should not be there. Rob's take is that most teams just want to ship software. If GitOPS helps with that part,
great. But if a more practical process works better, just use that. Do check out the show notes below for a related the Prime and Engine deep dives on backend technologies and other related topics. If you enjoyed this podcast, please subscribe to our favorite podcast platform and on YouTube. Especially thank you if you also are already on the show. Thanks and see you in the next one.