Welcome to Cybersecurity Unplugged, the Cyber Theory Podcast, where we explore issues that matter in the world of cybersecurity. Good day, everyone. It's the Steve King. I'm the managing director at Cyber Theory.
With me today is Craig Box, who's the vice president of open source and community at Armel. Craig was formerly the Kubernetes and advocacy lead at Google Cloud. He is responsible for Armel's open source Kubernetes security platform called Cubescape and for managing all relationships with a developer and open source community. Craig joined Google in 2014 just as Kubernetes was about to be released and has acted as a vital link between Google's Kubernetes development team and the wider community of developers who use the software.
Most recently, he was deeply involved in Istio, the leading open source service mesh, acting as an Istio steering committee member and driving Istio towards its recent acceptance as a CNC up incubation project. Craig is also the founder of the popular Kubernetes podcast from Google, the largest and most popular podcast about Kubernetes and a major source for relevant news and updates in the industry. So welcome, Craig. I'm glad you could join us today.
Thank you very much, Steve. One of the unfortunate things about leaving Google was that I had to leave the podcast behind, so it's always a pleasure to be behind the microphone again. Well, good. Even though the microphone is reversed, but that's good.
So let's just jump in here. And I think my first question relates to the capacity, I guess, of current CISOs to deal with what is, I think, a fairly complex technology. And from where I stand over here, it looks like we're, you know, complexity has been, if you follow me at all the complexities, it's been my deal for a while. And we've rushed to board this whole digitization movement in the process.
We didn't learn much about, I don't think about the technologies that are required for hybrid cloud, its computing containers, et cetera. And I usually use the idea that if I hold 10 CISOs, I think I wouldn't get any hands in terms of, you know, explain how Kubernetes works. You know, so I guess the question is that as a community, if we spent enough time learning about Kubernetes as we should. No, I think it's way too complicated and we should throw away the digitalization aspect, depending on paper.
There you go. It's perfect. You're the first guy to agree with me. No.
No, Kubernetes has this challenge and it's sort of a word with two meanings. And the easiest analogy I can go to it is it's a bit like Linux. Linux is a kernel and there are a million different ways you can compile and tune a Linux kernel. But in general, over the course of the past 20 years, we've moved away from having to care too much about that.
We have a vendor who gives us a Linux distribution and they've made a bunch of choices for us. And now the things we need to worry about are more of the interfaces between our applications and how they relate to each other running on top of Linux. There will be some people, especially running at larger scale who need to care about that layer underneath. But on the whole, some third party looks after that for us.
If there's a vulnerability in the kernel, we look to them to patch it in good time. And that to some degree is what we need to think of when we talk about Kubernetes. There is a wildly successful open source project called Kubernetes. But when people talk about that or Cloud Native, they are really talking about running their applications in a particular way using this engine.
But they have so many more things than the engine itself to worry about. If you look at the early books and deployment guidelines that talk about Kubernetes security, they say, well, here are all these parameters you can configure for when you install the agents on the nodes. And here's what you need to do to make sure that the API server isn't available to the Internet and so on. A bunch of very trivial things, which if you get your Kubernetes from Google, Amazon, Microsoft, whatever on-premises, whether you prefer, they look after for you.
You don't really need to worry about that. We recommend that you install some tooling that verifies that. This is a trust but verify model. But ultimately, the thing that you are enabling by installing something like this is the ability to load and run applications, Linux applications to bring it back to our layer stack for before, on many computers in real time, by API.
So you now have the ability to turn on how many replicas you need for a particular thing. That means that some bad actor could come along and use that same API to turn on as many Bitcoin miners or city at home, or whatever the problem is. So you do have to think more about the interfaces between your code and those systems, and you need to worry about the ability that these still are just applications that we're running, but you now have so many knobs that you can turn in the way that they all work together. I think the explosion of edges and the graph, now you've got these moving parts.
That is the piece we need to worry about as a community. Have we spent enough time learning it? We are still finding out what happens, and I think the right thing to do is to codify these best practices in tooling rather than make them something that you have to worry about that you just trust that as we do with software, we run static analysis tools, we look at the software we've written with other software. We should look to do that without cloud environments where we can.
Yeah, sure. My understanding and tracking Kubernetes is that Kubernetes has some sort of native challenges, security being one of them. And I think that owes to its complexity and the sort of native vulnerabilities. I want the best examples, I guess, of a Kubernetes break-in, was that Tesla crypto-jacking attack.
To get people to understand what is required in removing sort of fiction from fact, how do we do that? Is that education? Is that best practices? What's your view?
I think there's a combination of things there. The thing that Kubernetes does is it makes it easy to do things that were more complicated before or took more steps. Cloud is a large factor of this beforehand. The API to getting another machine was writing a check and sending an email off to Dell and waiting six weeks.
So cloud has made it possible to do those things a lot quicker. And then Kubernetes has really automated it to the point where some action in your environment, so more traffic hitting application, for example, can cause your system to scale up and then can ask your cloud load balancer. It is still a complicated piece of software. The distributed systems are complicated.
Kubernetes is a complicated example of that. It has places where things can go wrong. So a lot of effort in the first few years of Kubernetes went to looking at the differences between the sort of academic, single user internal environment at Google where things came from and saying, right now we need to deal with multi-tenancy. We need to deal with the ideas of many particularly groups or users wanting to access this.
It wasn't really designed to do that, and some of that was added on by Teams at Red Hat and some of it is possible in the way that the container boundaries work on top of Linux where you can isolate workloads. Some of it is still an honesty policy. If I request a certain amount of memory, it's like, well, if I use more than that, the system has to kill me. There's no real destruction saying, sorry, CPU cycles, not memory.
So the Tesla crypto jacking thing was an application that was installed in Kubernetes environment. There was a dashboard which was provided early in the Kubernetes lifecycle to do visualization of running work and then let you also deploy things and make changes. In order for that to run, it needed to have a service account from the Kubernetes environment with permission to do those things. So you say, hey, I want to run a thing.
It needs to have permission to act and do this. So this is a very powerful piece of software which you could just deploy on your cluster, misconfigure it, make a mistake as the operator, and say this is publicly available to be in there. That is what I believe happened there. And then you're effectively unlocking a door and saying, here's a way for anyone to run any kind of things that they want to inside this environment.
And I guess from Tesla's perspective, they need to be happy that the people who discovered it might not necessarily have known where they were. They had known they were inside Tesla. They might have tried to do something more interesting than just Minecraft account. Sure.
As you described that, it occurs to me that configurations always difficult and it's getting more and more difficult with hybrid cloud and more sophisticated, if you will, container technology like Kubernetes. And as that happens, our traditional networking seems to be sort of not very compatible with that growth pattern, if you will. And, you know, a hybrid and multi-tenancy are part of that. When the deployment is going across multiple clouds, it seems like we're, and it seems like with the move toward the fourth industrial revolution, if you will, we're heading more and more that way every day.
I know you guys talk a lot about the surface mesh and as a potential solution. Could you explain how that might work in sort of an optimal world? The model inside the individual Kubernetes cluster is very much built around the idea that every workload, every pod set of containers can talk to any other pod. And that's a very, again, academic early internet model, the idea that every node on the network has in the dress and that you're all able to speak to each other and so on.
Works well when you're inside a single company, gets harder when you start dealing with cross-cloud boundaries and almost feels like it was designed for a world where IPv6 is going to exist and be a thing. I'm not sure if that's the fourth industrial revolution you're glued into there, but we still wait with beta breath. Ultimately, we want to get business value and that may involve running a service where our users are and the users are now quite commonly worldwide. So there are a lot of people who are looking at Kubernetes to do some sort of hybrid cloud environment, running across multiple vendors, running across on-premise on.
So there are a couple of different things we need to deal with here. The first is the connectivity between the clusters. We can either have some sort of overlay network or something that makes it possible to treat them all like they are one big network. Quite commonly, you'll still have something that has gateways and these services know how to wrap to each other and we do want to keep them logically independent of each other.
A good pattern here is to say, right, we're going to have everything that's deployed within one availability zone, if you will, everything close to each other will do replication within that because we know we're in the same data center. It's all very quick. And then we need to say, right, in the event of a failure of that data center, we're failing over to another one and the replication between those may happen asynchronously. So you need to have some idea of the topology of the network for that to work.
This is all very early stage community stuff as to how this will happen. The content of service mesh came along a little bit afterwards and said, well, we need to do a couple of different things. First of all, we would like to be able to do micro segmentation. And what I mean by that is to say, we've got all of these different things that previously ran on their own computers with their own IP addresses that we could do firewalling against, for example.
Nowadays, we may have 10 replicas of a particular service. The only thing that should be allowed to speak to is service B. And those service B replicas could be running on any machine and cluster or conceivably anywhere other clusters that we connected to around the world. So we need to be able to save this group of things by whichever parameters we define, are allowed to talk to other ones.
And we have those parameters because Kubernetes deployed the thing. So what we can then do is create a consistent identity source, give them all an idea and a certificate from that because we deploy them, we know who they are, and then grant them permission based on their identity, not based on where they are in the network and which machine they happen to be in, which provided they're running on. But the identity that we gave them, then we're able to say, you're identity A, you're allowed to talk to anyone running identity B, wherever they happen to be. The way that service mesh achieves this is normally with a thing called a side car where we deploy every application, alongside that we deploy a little routing agent that gets those identities that intercepts all the traffic and routes between them.
And it's able to make decisions that sort of a software defined networking agent, if you will, but it operates at the application level. In this year, we're looking now at ways we can improve by not having to deploy an agent alongside every workload, but having something that runs just on the node and does lower-level networking. It's very hard to share that high-level networking where you're actually decoding application packets and so on in a multi-tenant secure way. So we're splitting that out and saying we're just going to do the layer four stuff at the node level.
And then you can deploy sidecars for workloads where you want the extra protection. And effectively, now we're able to validate both ends of a connection. We can say we know the identity of both ends. We know that this isn't someone who, once upon a time, stole that certificate that got an out-to-date certificate.
That solves a sort of security, like a traffic in motion problem. We know that everything on the wire is encrypted with that. If we tie that in with Kubernetes primitives to be able to do security over data at risk, then we have some idea that we are probably meeting some criteria that our compliance team would like us to meet. Yeah.
Okay. I hope you can unfree this question. But do you guys see yourselves as sort of part of a zero trust solution because of the micro segmentation that you described? Yeah.
I do think that that's a common category that's applied here and in large part what people think of a zero trust today is derived from some of the ways that Google runs production environments and that obviously led to Kubernetes and SDO and some of the projects that came out of Google to allow people to run them the same way. The idea there is it doesn't matter what the node is you're running on. It could be adversarial to some degree. You don't have to be in a secure place.
It's not about where you happen to be. I'm on a trusted node therefore I'm allowed to exit something else. It's about the identity that you have. It's issued to you.
You can run anywhere and you can use that identity to connect services outside the network as well. So you can have something that validates your certificates. A public service or a SAS environment or something that you need to connect to. So it really does help you build a zero trust system.
There are some other pieces you need to build yourself. But we want to get to a point where if someone is able to exploit something in the middle of your network and we're no longer dealing with the whole motor around the castle approach, you may be running something like an image rendering technology or things that resize process images. They are a very common attack factor. Yeah.
You hear that especially in mobile phones or that the more modern iPhone hacks have come about due to a bug and processing JPEGs and coded in PDFs or something like that. So if you have to do some processing where you accept user data, you end up with someone sending you a crafted file and then they get to exploit some service. They are now five or six levels deep in your microservices stack. They're not just breaking in through the file or turn up a middle of the network.
And so you need to make sure that they can't get access to your payment service for example. There's no reason your image resizing service should be able to access your payment service. And that's really where the whole microsigmentation comes in. You need to deal with the fact that people can pop up in the middle far more easily than they could in the castle in most approach.
Yeah, sure. You correctly or appropriately describe this as a sort of complex distributed network. What happens with 5G in your estimation? I mean, it's whether it's let's call it 100x, you know, factoring speed.
What's the effect going to be on the temporal based reception, if you will, issues that Kubernetes expects or any container expects as a part of the architecture? There's a couple of different ways that Kubernetes and 5G overlap. You can think of 5G as a way of ensuring everything is connected. It's kind of like Wi-Fi in your house today.
Now it's easy to assume that my thermometer and my TV all have IP addresses and they can all talk about themselves when I'm out of a house. But 5G I think will mean that that's true of everything everywhere. So cars, obviously mobile phone you've already got. But anything that's traveling around, there are a couple of different aspects to this that matter, like the mobility of things, like you mentioned, there is my thermometer in the most part doesn't come off my wall.
Whereas 5G devices move around and they get routed to whichever base station makes sense for them. There will be a little bit of a challenge, which I'm not close enough to the wire here. Just an interesting side note. My father has worked in telephony since 50 years, I think, coming up now.
And it's always interesting seeing how his world in mind has overlapped and then moved apart and so on. There was a period where everything started being digitized and moving to IP in the 2000s and so on. And then it kind of moved away again. And now it's time to talk about Kubernetes actually powering some of these 5G systems.
So I'm not 100% sure of how it all works. But I can imagine the number of devices and the fact things come and go and so on is leading the 5G spec to require a bunch of functions, that things like tracking where your phone is and how to build it and what it's access permissions and sessions and so on should be. If the system is smart enough to make sure that you're always given the same IP address as you move around, then you may end up with a situation where everything's okay. But you need to move to that more modern identity based on something you have and whether it be touching touch on your phone or whether it's just the certificate of the phone, for example, for an IoT device.
All that is really doing then is making sure that there are a lot more, it is making there be a lot more devices out there. And Kubernetes is being used inside 5G networks to power those functions because you need to be able to run a lot of low impact virtual functions near these base stations and page and so on. And then you also need to deal with the fact there will be a lot more devices on the Internet. So bring on IPV6 again.
So I still think that there is a disconnect and this was going to happen anyway. And Kubernetes is a nice way of being able to manage things that are running at scale and have a lot more going on than they used to. But you could have had one without me. Okay.
I'm happy to buy that. I know that also storage has been an issue with Kubernetes. I think one of those reasons is that what the entire storage infrastructure is managed without relying on any cloud resources? Is that the case?
Or how does a practitioner avoid that sort of trap? Is that a configuration thing? It can be. You can have storage of above and below Kubernetes, which is a complicated thing to understand.
So let's look at one at a time. You have Linux processes running inside containers and so on. They are used to running on a disk. So those things need to have storage available to them.
And Kubernetes is an API provider is able to answer a request from your storage for your storage needs. So you say, I'm going to run a database node. I need a 60 gig local disk. It doesn't need to be high availability because I'm going to just be using this as a brief replica, for example.
But you say, please provision me 60 gigs wherever I happen to land. You might land on a physical server that has some storage available or you might have that mapped over the network. But the idea is that you use a consistent language to define what it is that you want. And you might also have classes.
You can say, I want this to be high speed or low speed low cost, for example. And then you can have that deployed for you. It's a lot easier if you're running on a cloud because you can just call an API and say, attach a 60 gig disk. But you can do the same with on-premises.
If you are trying to provide that service, if you're on-premises and you want to provide the facility for someone to say, hey, give me 60 gigs of disk and have it automatically through network attachment, there are a bunch of different NFS onward solutions that allow you to do this. And a lot of them now run on top of Kubernetes. So this is where it's confusing because it's storage above and below. You can run a storage system and you can have Kubernetes handle the high availability of that system and replication.
All these things that people previously had to build into their own application. You can outsource that to the Kubernetes lab. So a lot of storage stacks from vendors these days, a lot of modern databases and so on, are using Kubernetes as their replication lab. You go to Oracle nowadays and it's surprised me to see the other day they have a Kubernetes operator.
This is how you go about deploying Oracle today. The assumption is that any stateful service you're going to deploy on top of Kubernetes. And that means they don't need to worry too much about the different APIs that are provided by different things. It obviously does matter a lot.
The fact that things come and go in Kubernetes. You don't really want your storage nodes just to randomly disappear or be rescheduled. So you have to build a system that's aware of this and it's designed for that. It's a bit harder to do NFS out of the box.
But something like Min.io which provides something that looks a bit like Amazon S3. That's built with replication in mind. And so you can have a Kubernetes cluster providing storage and then you can have that storage mounted by a different Kubernetes cluster or by the same one. So again, if you just think of Kubernetes as a system that provides things, it's like saying, well, I have a Linux powered NetApp system or VSD powered NetApp system that makes storage available and it's available to my other Linux powered systems.
When it comes to security of those things, it's all the same stuff again. It's like the fact that we made it very easy to map and mount those storage volumes. We need to make sure that we're currently grounding commissions to people. And that is one of the biggest areas that people have to watch out for is that if you, again, have something like the Tesla situation where you have a service account that has permission to do anything, if you have someone get access to that account by taking control of some software that has access to it through some means, then they can then pivot into doing whatever else that they want.
And that's where we're seeing a lot of people say, hey, I need to look at the complexity of all of the role accounts and the access granted those role accounts and where they need tools like Kubernetes, they can come in and say, there's a lot going on here. Prioritize the things that matter to me. Show me the vulnerabilities that might matter and how I can go about remediating them. Yeah.
The more you talk the more I listen and look at the stuff, it feels to me like it's actually whatever you want. Let's call them Kubernetes or super smart containers or something. It's almost become operating system infrastructure like a standalone beast. It's like the meta presence for running your entire IT factory.
It is exactly that. It is a cloud provider that is quite commonly run on top of another cloud provider. And you will need someone in your team who understands the interfaces between the two. But if you have that figured out, then most people should only have to deal with the Kubernetes.
The whole idea of this is that you can say to your development team or people who are deploying platform applications for the people who actually want to run them. It doesn't matter that you're running on Amazon or Google or whatever. We have extracted that away. They've all got storage available.
We've just changed the class to say, hey, if you're on the Google Cloud, you've given some Google flavored storage and so on. And you do need a different skill set. I'm not 100% sure that you need the skill set that we're training people for. How to set a Kubernetes environment up from scratch.
Most people are going to deal with it a layer above that. They're going to deal with how do I configure the Kubernetes knobs provided by the vendor? And then how am I going to provide a platform to my developers so that they don't have to care about any of this? Well, that assumes that the person you were just describing understands Kubernetes to begin with.
And again, I kind of go back to that point where I'm really not sure or maybe it's I'm pretty sure that most of the folks leading sub security today don't. Maybe the current CIO world, maybe the IT folks really do. I don't know what your experience has been in that regard. It is a different set of nouns and verbs on a similar kind of problem to cloud.
So you can map that thing. But another piece of the challenge is that Kubernetes and cloud before it, to a large degree, have to overuse a term and shift everything left. The idea that developers now can just deploy things. They don't have to go through processes.
They don't have to be signed off by executives. It now makes it a thing that developers are dealing with. And the challenge is how to make that group of people care more about security where it's slowing them down from the thing that they can otherwise achieve. And that was true when it was, I could deploy a VM with a credit card and your internal secure infrastructure is going to take me six weeks to provision.
And eventually those worlds got bought together. The CIOs found a way to sit guard rails up and have people be able to deploy in cloud infrastructure. And in the large part, that is a default these days. People are expected to use some sort of cloud infrastructure unless a pricing need moves them out of it.
But even on-premises stuff has been provided with cloud-like APIs. So there is a shared responsibility here and that your vendor should help you understand this. Your developers will know this. I think the choice there is to say, well, how do we define guard rails and set up the environment so that the developers don't want to end up on the front page of the New York Times anymore than the CIO does for the wrong reasons.
The incentives there, I think, are well aligned. It's just a case of exposing them correctly. My experience has been that, and this is a large healthcare company, that the engineering folks essentially won that battle period, end of story. You say combined, I say engineering said, hey, screw you.
I'm over here. I'm going to do this thing. You figure out how to make that your world comfortable with it. That was the impression, the very clear impression that I had in dealing with those guys.
And it's great to hear your optimism for this because not understanding this at the level that you do, it's very easy to look over here and look in the history of technology as we've seen it and say, oh, my God, this is another. We're designing and building this airplane that's already taken off on its way over the cliff. If you think about the airplane as an analogy to some degree, the pilot knows how to control the airplane, but they don't know how it's built. They trust that there is a group of people there who understand and who do the maintenance and so on, and they don't fly the plane until such time as the ground crew have signed off and everything's good.
And then as a passenger on the plane, I don't have to think about it at all. I trust that the processes work well and accepting the very rare cases where it doesn't. Aviation tends to be very safe. And the processes have made that the case and have led to the point where everyone heads a skill set that works.
I talk to people who touch individual parts of the class of 40 hours a week very, very specifically, and they run rings around me in that one particular area because that's the area that they're focused on. So I think through this, everyone needs to get an idea of which pieces they need to care about, which pieces are relevant to them, and how we can get enough information, how the CSO can say, right, my engineers have made this choice for me to disagree. How am I going to become comfortable with that? I'm going to get to the point where I trust them as the ground crew and that they've built something.
And that might be, again, a trust by verify thing that the engineering team provides a dashboard and says, we are running these checks and we're signed off against these things. And it is like the list that's presented to the pilot before they fly. Yeah. It's the same engineering crew that's using open source APIs that are impossible to vet in terms of what their dependencies are in the wild.
Engineers love new things. Pardon me? Engineers love new things. Yeah, I know.
Terrific. It's good for them. I'm glad we're able to provide new things for them. I guess the last question I have for you, Craig.
We could probably talk for another hour and a half because it's been great. I appreciate you explaining it at the level of a six year old, which is kind of what I am. What would you suggest that a CISO do to plan and prepare for a Kubernetes deployment? Yeah.
You need to be very deliberate about it and you need to realize that it doesn't look like the thing you were used to in the past, even if the thing was just regular cloud stuff. The idea that things will be shared that were not previously shared is scary to some people and that causes people to make design decisions about how things will run. You need to have a threat model that you're comfortable with. It might be that I'm willing to have different customers who are running multiple tenants in my application effectively that you're willing to let them share certain services on a machine.
But the ultimate thing to think about in the generic Kubernetes environment is there is still one kernel underneath each machine that is shared between them. And in the most part, we trust that that is a safe boundary, but every now and then there's an exploit which proves that that's not the case. So what would happen in the event that some user could do something in my environment to get access to other things running on the machine? That's kind of the most common container escape concerning we need to think about.
And some people like to set up completely different nodes for different customers and maybe for untrusted workloads that makes sense. Google came along and built a Linux emulator, which was a way of being able to securely run things to give another sandbox for generic processes so that if we're running more than one user stuff on a single machine, they would now have to escape two different sandboxes to get into the kernel. So that was a trade-off that they considered acceptable and that's an open source project that you can run. I've seen similar kind of stuff come out of Amazon and the micro VM approach with Firecracker saying, all right, well, we need to think about the fact that multiple people are going to run things in this environment.
The academic ideal when Kubernetes was created was you just have one giant cluster and you throw things out of it and you schedule everything and it will optimize everything so much and you're able to pack so much more. But the reality is that it is going to, it's going to concern people and plan how you want things to be set up, all the observability tooling. There's so much stuff is exposed by modern open source tooling that you can scrape with something like Prometheus and then you can get visibility into what things are running where and you need a way of alerting on that, you need a way of knowing what's going on in your environment, you need some sort of guardrails, you need policy engines, you might want to consider a service mesh, you definitely want to have something that's handling deployment to not just humans, so some sort of continuous integration or get ops if you fancy the modern buzzword of being able to deploy code that will then get processed and rolled out to the clusters by something. There's so many different pieces here that you should probably spend three or four months beforehand understanding what you want to do before you've gone out and built the production release system.
Yeah, that makes sense to me, for sure. Well, Craig, thank you. This has been great, as I said, and folks, this is Craig Box again, the Vice President of Open Source and Community for Armou and is responsible for their Kubernetes security platform called Cubescape, and for managing all of that developer relationships and the whole open source community around them. So I appreciate you taking the time today, Craig.
This is informative and instructive and maybe we can catch up again in four or five months and kind of see what's been going on. Thank you, Steve. I really appreciate it. Good.
And thank you to our audience for spending a half an hour or so with us, and hopefully we've learned a lot about Kubernetes and enough to at least get you sort of started on the education side, so we will see you again next time. And I'm Steve King, your host, signing off. Thank you for joining us for another episode of Cybersecurity Unplugged. You can connect with us on LinkedIn or Facebook at cybertheory or send us an email at social at cybertheory.io.
For more information about the podcast, visit cybertheory.io forward slash podcast. Until next week, thanks again.