Hashicorp Vault Secrets in Kubernetes with CSI Driver

Injecting secrets to Kubernetes Pods using Vault CSI Driver

You might have already guessed the context of this article. Well, however, to provide a use case on the CSI driver let us assume that there are certain secrets that are to be mounted to the Pod as an ephemeral volume. Usually, in this case, you may have to create a Kubernetes secret by getting the values from Vault and then Mounting it to the Pod. This is very complicated. How about a solution where we can easily create this using a CRD and then Mount it to the Pod? Imagine having the flexibility to control the Pod’s lifecycle till the secrets are retrieved? Yes, getting Vault Secrets through Container Storage Interface (CSI) Volume is possible. The CSI Secrets Store driver allows users to create SecretProviderClass objects. This object defines which secret provider to use and what secrets to retrieve. When pods requesting CSI volumes are created, the CSI Secrets Store driver will send the request to the Vault CSI Provider if the provider is vault. The Vault CSI Provider will then use Secret Provider Class specified and the pod's service account to retrieve the secrets from Vault, and mount them into the pod's CSI volume.

The secret is retrieved from Vault and populated to the CSI secrets store volume during the ContainerCreation phase. This means that pods will be blocked from starting until the secrets have been read from Vault and written to the volume.

Vault with Kubernetes CSI Driver. Image Credits: Google Images

What is the entire story all about? (TLDR)

1. Deploying CSI Provider in Kubernetes.
2. Integrate Vault with CSI Provider in Kubernetes.
3. Mount the secrets as an ephemeral volume to the Pod from the Vault.

Prerequisites

1. A Kubernetes Cluster ( EKS, AKS, Kind, etc ).
2. Integrate Vault with CSI Provider

Story Resources

1. GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
2. GitHub Branch: vault-csi

Installing Hashicorp vault using Helm Chart

We will install the official helm chart for the vault and unseal it manually. However, this is not the Ideal way for running a vault in Production. You may want to unseal the vault using a KMS Key ( If being installed in AWS ) or a Google KMS key ( If being installed in GCP ).

Enabling Kubernetes Auth Method

The Kubernetes auth method can be used to authenticate with the vault using the Kubernetes service account token. This will help the vault to inject the vault token into the Kubernetes Pod. For this let us enable the vault Kubernetes backend by exec into the pod, since the token_reviewer_jwt has to be passed from the vault pod.

We now have a Kubernetes auth role created. This role is bound to the service account names issuer and namespace named default and the policy named betttercallpavan_pki which was created earlier.

Install the secrets store CSI driver

The Secrets Store CSI driver secrets-store.csi.k8s.io allows Kubernetes to mount multiple secrets, keys, and certs stored in enterprise-grade external secrets stores into their pods as a volume. Once the Volume is attached, the data in it is mounted into the container’s file system.

Add the Secrets Store CSI driver Helm repository.

The vault CSI driver should now be installed on your Kubernetes cluster.

CSI Driver

Injecting the Secrets using Secret Provider Class

I am using the vault kv backend to store my secrets. I have already uploaded a couple of secrets in the vault kv backend earlier. We will create a secret class to fetch the secrets from the KV backend.

Secret Storage Class

a) provider: The name of the secret storage class to be used. Here we will be using vault as the secret storage class provider.

b) roleName: The name of the Vault KV role to be used for authentication.

c) vaultAddress: The address where the vault is exposed.

d) Objects: The KV paths from where secrets are to be extracted

Creating a Deployment

Let us now create a deployment that will use this Secret Provider class to mount the secrets as a volume to the deployment. Once you apply then you should find the secrets mounted as a volume to the Pod. Let us now examine the deployment.

Deployment Volumes

a) VolumeMounts: The set of volumes to be mounted to the Pod

b) driver: secrets-store.csi.k8s.io

c) volumeAttributes.secretProviderClass: The name of the secret provider class, created earlier

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b vault-csi
$ cd medium-manifests
$ kubectl apply -f vault-csi.yaml

That’s it. Boom !! You should now see that the secrets are mounted to “/mnt/secrets-store” location on the Pod. I have now exec-ed into the Pod and find the secrets already mounted.

Secrets mounted to the Pod

Hold on, we are not done yet. The Secrets Store CSI Driver also supports syncing the secrets from vault KV to Kubernetes secrets. Let us see that in action now.

Secret synced from Vault

I have also created a Pod, that would consume this secret as an env variable. The pod will wait for the secret to be created before starting, and the secret will be deleted when the pod stops.

$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b vault-csi
$ cd medium-manifests
$ kubectl apply -f vault-csi-sync.yaml

Alright, That is it for the Article. I hope this might help you to optimize your secret creation and retrieval process. Please feel free to comment down your queries and experiences while implementing this.

Here are some of my other articles that may interest you

Until next time…..

Recommended

Multi-Tenancy in Kubernetes using Loft’s Vcluster

PKI Certs Injection to K8s Pods with Vault Agent Injector

Analyze Terraform costs with Infracost ( The GitOps Way )

Terraforming the GitOps Way !!!