Kubernetes Policies as code using Kyverno

Kubernetes native policy management

Using Kubernetes has become quite common these days. ML Applications, Web Applications, Big Data applications, etc are now very compatible with Kubernetes that they are being used and deployed at scale. But how secure are these workloads? Assume that there’s an Internal application that was accidentally exposed as LoadBalancer to the Internet, also assume a situation where a public docker image was pulled from the Internet rather than the internal artifactory registry. There are many such situations that are potential harm to the Kubernetes Infrastructure. Is there a way that can control such misconfiguration? Is there a way that certain policies can be written that can restrict these misconfigurations? Kyverno to help. Kyverno is a policy engine designed for Kubernetes. Policies are managed as Kubernetes resources and no new language is required to write policies. Kyverno policies can validate, mutate, generate, and clean up Kubernetes resources plus ensure OCI image supply chain security. The Kyverno CLI can be used to test policies and validate resources as part of a CI/CD pipeline.

Kyverno

What is the entire story all about? (TLDR)

1. Understanding how Kyverno works.
2. Understanding various Kyverno policies ( Validate, Mutate, Clean up, and Delete ).

Story Resources

1. GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
2. GitHub Branch: kyverno

Prerequisites

1. A Kubernetes Cluster ( EKS, AKS, Kind Cluster, k3d, etc ).

Installing Kyverno

Kyverno can be installed using Helm or deployed from the YAML manifests directly. Kyverno can be installed in two modes. Production Mode ( That has 3 replicas ), Dev Mode ( Just has 1 replica ). Kyverno does not support two replicas. For a highly-available installation, the only supported count is three. Refer to the Installation method here. For the sake of the demo, we can consider installing Kyverno in Dev Mode.

$ helm install kyverno kyverno/kyverno -n kyverno --create-namespace --set replicaCount=1

NAME: kyverno
LAST DEPLOYED: Fri Apr  7 11:09:15 2023
NAMESPACE: kyverno
STATUS: deployed
REVISION: 1
NOTES:
Chart version: 2.7.2
Kyverno version: v1.9.2

Thank you for installing kyverno! Your release is named kyverno.
⚠️  WARNING: Setting replicas count below 3 means Kyverno is not running in high availability mode.

💡 Note: There is a trade-off when deciding which approach to take regarding Namespace exclusions. Please see the documentation at https://kyverno.io/docs/installation/#security-vs-operability to understand the risks.

Once the helm chart is installed, we should see the following pods installed.

kyverno namespace resources

Creating Kyverno Policies ( Mutating )

Assume that you have a multi-tenant Kubernetes cluster, where multiple teams use the same cluster. As a Kubernetes administrator, you’d want to segregate various teams on the basis of namespaces and node pools. Multitenancy architecture can be designed something like this
TEAM A -> Nodepool A

TEAM B -> Nodepool B

As an administrator, we would want to make sure that all the Pods / Deployment resources in namespace “a” are scheduled on “a” node pool and the pods in “b” namespaces are scheduled on “b” node pool. We will have to make sure that the node selectors to the pods are automatically added whenever a pod is scheduled in the corresponding namespace.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b kyverno

$ cd medium-manifests

$ kubectl create ns a 

$ kubectl apply -f add-node-selector.yaml

This policy will make sure that any pod that is being scheduled in the namespace “a” will get the nodeSelector spec ( defined in Line 27 ) and will automatically get appended.

$ kubectl create deploy a-test --image=nginx -n=a

$ kubectl get deploy a-test -n a -o yaml | grep -i "node"  
                      
      autogen-add-nodeselector.add-nodeselector.kyverno.io: added /spec/template/spec/nodeSelector
      nodeSelector:
        nodes: a

Creating Kyverno Policies ( Validating )

Assume that you’d want to have an airgap cluster ( An air gap, air wall, air gapping, or disconnected network is a network security measure to ensure that a secure computer network is physically isolated from unsecured networks, such as the public Internet. ), and you’d want to pull the docker images always from the internal artifactory registry but not from the internet. Let us see how Kyverno can help us here.

Using this policy, we are making sure that the registry of the images is always mycompany.docker but not any other registry. Let us try this out.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b kyverno

$ cd medium-manifests

$ kubectl create ns a 

$ kubectl apply -f allowed-image-repos.yaml

Let us now create a deployment that pulls images from a public repository.

$ kubectl create deploy a-test --image=dockerhub.io/nginx -n=a                                                               1 
 pavakuma@INM1D7G9NMD6M
error: failed to create deployment: admission webhook "validate.kyverno.svc-fail" denied the request: 

policy Deployment/a/a-test for resource violation: 

allowed-image-repos:
  autogen-good-repos: All images in this Pod must come from an authorized repository.

That did not work, as the images are being pulled from a public repository. Let us now change the image repository to “ mycompany.docker”.

$ kubectl create deploy a-test --image=mycompany.docker:nginx -n=a                                                               pavakuma@INM1D7G9NMD6M

deployment.apps/a-test created

Surprised????

Yes, that was pretty simple. This is what I liked about kyverno. Policies are managed as Kubernetes resources and no new language is required to write policies.

Closing Thoughts

So this is how kyverno helps Kubernetes administrators write policies without even learning a new language. There are a bunch of other policies that can be reused while creating any Kubernetes clusters. These policies can also be used to mutate/validate many other open-source tools like CertManager, Istio, Flux, ArgoCD, etc. Please feel free to share your experience while implementing this in your clusters.

Until next time…..

Recommended

Introduction to Helm Dashboard

How I reduced the size of my Docker Image by 95%

Terraforming the GitOps Way !!!

How to guess the right size for your Kubernetes Pods?

Level Up Coding

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the Level Up Coding publication
• 💰 Free coding interview course ⇒ View Course
• 🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job