Microsoft released a threat matrix for Kubernetes based on the MITRE ATT&CK **framework. There are multiple ways to establish persistence in a Kubernetes cluster but in this article, we will deep dive into the technique involving malicious admission controllers.
Kubernetes Admission Controllers are elements that intercept Kubernetes API requests and can modify, accept, or deny them. This operation occurs before the persistence of a resource in etcd. Today, we will focus on webhooks. A webhook is basically an HTTP server that receives the kube API request and sends a response to it. There are 2 types of webhooks :
Now, imagine that you are an attacker and you compromised a Kubernetes cluster. Your next step would be to deploy backdoors in order to sell it or to come back if your initial access is discovered. A simple way to do this would be to deploy a backdoored container. You can do this manually, with a deployment, for example. But there is a big chance of attracting the attention of a cluster user.
As myself being a Kubernetes user, kubectl get pods is probably one of the commands I use the most. And this kind of deployment is suspicious because it has been created by someone from outside. Let's try something clever: what if a cluster user is himself deploying backdoor for you? One way to achieve this would be to install a mutating webhook in the cluster.
Now, we won't create any pod manually. Instead, we are going to tell the cluster that each time a new pod is created, we want to inject a backdoor into it. As we saw earlier, the mutating webhook is a perfect candidate for this. Let's use the sidecar injector from this repo to inject a backdoored container in all pods.
After the installation of this webhook, every pod of your Kubernetes cluster will have a backdoored sidecar container. So the attacker will be able to come back to your Kubernetes cluster through any of the pod deployed. Of course, it is very noisy to inject a backdoor into every pod.
We can modify the mutating webhook to adjust the injection rate or target some specific pods to get something less obvious. This technique has a lot of advantages for the attacker. You can deploy whatever you want as a sidecar container from backdoor to cryptominers. But the most important interest is stealth. The cluster is itself deploying backdoor without your intervention.
We can also imagine a system where the webhook is pointing to an external server. In this case, the attacker can change the behavior of the webhook on the fly. He can remain silent for months and activate the payload only when he needs it. That’s why detecting this kind of attack can be really hard.
A good RBAC is not easy to set up and maintain in your Kubernetes cluster and it is not always enough to prevent this kind of attack. Indeed, this attack can be deployed by a malicious cluster user, who can create a webhook from scratch.
There is no perfect way to protect yourself against this. You can’t deactivate webhooks because they are used by a lot of other elements such as the nginx-ingress controller. You can’t really rely on validating webhook because the user able to create a mutating webhook is probably also able to modify validating webhook. Surveillance is probably the best way to deal with this kind of attack. Tools like Falco can notify you when it detects malicious behavior in legit pods which gives you a clue that your Kubernetes cluster has been compromised. In this case, inspecting webhooks might be interesting. You can also try to add a notification every time a mutating webhook is created but it might be really noisy.
The strength of this attack might also be a good way to detect it. The attacker introduces a differential between what we want to deploy and what is really deployed by your Kubernetes cluster. This differential can be handled by the gitops approach. Indeed, by using tools like ArgoCD, you can check if what you have in your Kubernetes cluster is actually what you asked to deploy from your repo. With this mechanism, no mutating webhook would be able to modify your pods to install backdoors in it; otherwise, ArgoCD will correct the deployment.