14 January 2025
Drawing upon our extensive Cloud security expertise, we've put together a set of helpful articles covering security best practices we think are really important on different subjects. In today's article, we will introduce 5 best practices for having secure CI/CD runners in Kubernetes.
How can my CI/CD configuration in Kubernetes be a vector attack?
A misconfigured self-hosted CI/CD setup in Kubernetes can be a vector for attacks by allowing unauthorized access to your cluster through poorly configured runners or job pods. This can lead to data leakage or app disruptions. Properly securing your CI/CD configuration is essential to prevent these risks.
What are the best practices for Kubernetes CI/CD?
First, you must ensure that your runners have the appropriate privileges. Then, you must implement the runners in separate nodes as your main app nodes for each environment. You must also add CI/CD scanning to your code and docker images. Finally, you should create multiple runners adapted to each environment's needs.
My runners have low privileges
Recommendation: Separation of Privileges
One fundamental practice in Kubernetes CI/CD is to limit the privileges your runners and jobs pods will have on the cluster.
Running privileged pods may allow compromised dev machines to gain access to the node clusters, for example.
According to the principle of least privileges, each runner must also have only the permissions needed for its stage.
Usually, your runners do not need many privileges. You can separate the rights for your runners and the rights of the jobs launched by your runners for CI/CD. Usually, the jobs need really specific rights, whereas the runners mostly need to be able to run those jobs.
Risks Addressed:
- Mitigates the risk of unauthorized access to sensitive resources
- Restricting privileges helps prevent attackers from escalating their access to more sensitive parts of the system
I have a good node isolation strategy
Recommendation: Implement a robust node isolation strategy
To enhance security and resource management, adopt a solid node isolation strategy. Isolate CI/CD resources from application workloads by using separate node groups for your runners and jobs.
Risks Addressed:
- Inadequate isolation can result in easier attacker's lateral movements
- Sensitive information from the apps may be more difficult for an attacker to access in case of node separation
I use docker images fit for my needs
Recommendation: Limit the content of the images you use for your job to what you need.
CI/CD jobs are launched using a base Docker image. This image must be chosen wisely and contain only what is strictly needed for your pipelines to work. Depending on the job, you can use different images.
Risks Addressed:
- Using unverified or vulnerable images may introduce known security vulnerabilities
- Images with vulnerabilities can be exploited by attackers to gain access to your Kubernetes nodes.
I use Checkov or Kyverno to forbid the deployment of insecure resources
Recommendation: Implement tools that check misconfiguration for your CI/CD runners setup.
Ensure that your CI/CD runners setup is checked by tools such as Checkov or Kyverno to identify and prevent deploying insecure resources.
Chekov can check your Kubernetes yaml files and detect security issues, misconfigurations, or compliance violations before deployment.
Kyverno can prevent privileged runner or job pods from being deployed using its strong policy as a code engine.
This can help check that step 1 (having low privileges) is well applied.
Risks Addressed:
- Limit the risk of deploying high-privileged resources
- Prevent accidental misconfiguration of resources
Secure your runner configuration on your git provider
Recommendation: Use secure parameters when configuring your runners on your git provider.
This part may need a proper article. However, it is essential to make sure your runner configuration is correctly configured on your git provider to address strictly your needs.
Taking the example of gitlab CI :
- Runners can be associated with specific projects or groups or shared (i.e., used in a specific repository or from anywhere). The use of shared runners should be limited.
- Runners can be registered using an authentication token or registration token (the second is dangerous)
The configuration is significant to take care of and should not be set randomly.
Risks Addressed:
- Restrict the use of your runner to people and repositories that should not need it.
- Limit the exploitation of deprecated ci/cd configuration.
Conclusion
By following the best practices presented in this article, you can significantly reduce the risk of Kubernetes compromission or data cluster leakage and improve the overall security of your infrastructure when using self-hosted runners.