Multi-Tenancy in Kubernetes using Loft’s Vcluster
Multi-Tenancy in Kubernetes using Loft Vcluster
Kubernetes is almost everywhere. If you want to deploy a web application, you’d need Kubernetes. You’d need to train an ML algorithm ( Kubeflow ) you’d need Kubernetes. Run Data analytics you'd need Kubernetes. So ideally Kubernetes is being used in every possible way. But are you using it the right way? Are you saving costs? Are you making use of the entire compute resources? Are you also sharing it the right way? Ahh, this is the actual point of the article, Multi-Tenancy. Multitenancy is a reference to the mode of operation of software where multiple independent instances of one or multiple applications operate in a shared environment. The instances (tenants) are logically isolated, but physically integrated.“Tenants” is a term for a group of users or software applications that all share access to the hardware through the underlying software. Multiple tenants on a server all share the memory, which is dynamically allocated and cleaned up as needed. They also share access to system resources, such as the network controller.
What is the entire story all about? (TLDR)
- Multi-Tenancy in Kubernetes.
- Using loft.sh’s Vcluster solution.
- Exploring various scenarios in multitenancy using loft.sh vcluster.
Prerequisites
- A Kubernetes Cluster ( EKS, AKS, Kind, etc ).
- helm, loft.sh’s vcluster binary.
Story Resources
- GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
- GitHub Branch: multitenancy
Scenarios for Multi-Tenancy ( Problem Statement )
Earlier I have explained the definitions of multitenancy and the tools that are used to enable this. But first, let us understand the problem statement for this use case. Assume you are a company that provides Kubernetes clusters to customers to deploy their workloads. Let us suppose you have 100 customers. Provisioning one cluster per customer is a nightmare. Managing those clusters is not an easy task, as the number of customers grows the number of clusters also increases. Drawbacks of having such architecture?
- Increase in the overall cost.
- Redundancy of the components to be installed ( For example bootstrap components like istio, vault, consul, etc to be installed on all the clusters ).
- Management of the clusters will be a nightmare.
- Lots of duplicate work to be done.
- Heavy spike in the costs ( You end up paying money for both the control plane and worker nodes ).
- No Isolation.
- Each customer would end up using a lot more than allocated resources. We cannot control the number of resources that the customer can use.
What if? What if there is a solution to this. How would you react if you can create a Kubernetes cluster inside a Kubernetes cluster?
Yes, you have read that right. Here comes loft.sh’s vcluster ( virtual cluster ) into the picture. Virtual clusters are fully working Kubernetes clusters that run on top of other Kubernetes clusters. Compared to fully separate “real” clusters, virtual clusters do not have their own node pools. Instead, they are scheduling workloads inside the underlying cluster while having their own separate control plane. The virtual cluster itself only consists of the core Kubernetes components: API server, controller manager, and storage backend (such as etcd, SQLite, MySQL, etc.).
Advantages of using loft.sh vcluster
- Fine-grained Isolation per tenant.
- Reduction in the Overall Cost.
- No Redundancy of components.
- Easy management of tenants/clusters.
- Resource allocation per tenant can be controlled easily.
Well, I hope you now have a clear picture of multitenancy and Virtual clusters. Let us now get started with the Demo. :)
Installing Vcluster using helm
Make sure you have vcluster and helm binary installed
I am already having a Kubernetes cluster up and running. You can use any Kubernetes distribution for this demo. Let us now create two namespaces in our physical cluster. These namespaces are nothing but our customers. Let us name the customers as
- customer-1 ( Trial Customer )
- customer-2 ( Paid Customer )
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b multitenancy$ cd medium-manifests
Trial Customer
Alright, we have now created two namespaces for two new customers. So let us assume that customer-1 is a free trial customer and he should only be given very few compute resources. For example CPU: 5cores, Memory: 10Gi, Pods 3 etc.
Let us now design a Virtual Cluster ( Vcluster ) for the free tier customer.
$ kubectl create ns customer-1$ kubectl config set-context --current --namespace=customer-1$ helm upgrade --install customer-1 vcluster \--values customer1-values.yaml \--repo https://charts.loft.sh \--namespace customer-1
The vcluster should now be created. You can check the pod in the customer-1 namespace. Things would now get interesting. Open another terminal tab and execute the following commands to connect to your virtual cluster.
$ vcluster connect customer-1 --namespace customer-1$ export KUBECONFIG="./kubeconfig.yaml"
Let me create a namespace in the Virtual cluster.
$ kubectl create ns testThe namespace should now be visible in the vcluster but not in the actual host cluster. Let us now create a deployment and scale the replicas to 2 ( As the hard limit is 3, no more than 2 pods will be able to run on the virtual cluster. There is already one pod by a system component ). The same is also the case with the other Kubernetes components.
Paid Customer
Now let’s switch to the new customer ( Customer-2 ). The Paid customer should be having a huge quantity of resources including CPU, memory, services, etc.
$ kubectl create ns customer-2$ kubectl config set-context --current --namespace=customer-2$ helm upgrade --install customer-2 vcluster \--values customer2-values.yaml \--repo https://charts.loft.sh \--namespace customer-2
A virtual cluster is now created for customer-2 as well.
Open a new terminal tab.
$ vcluster connect customer-2 --namespace customer-2$ export KUBECONFIG="./kubeconfig.yaml"
You should now be able to create a huge number of computing resources including pods, volumes, etc.
This is how loft.sh’s Vcluster can be used to achieve multitenancy. The aforementioned example is one of the use-cases of multitenancy. In the Kubernetes Era, there are many more scenarios where multitenancy could be applied. Please feel free to share your experiences/thoughts/ideas to implement multitenancy in Kubernetes. Also, feel free to get in touch for any queries/consultation on Kubernetes here.
Cleanup
$ helm delete customer-1 -n customer-1$ helm delete customer-2 -n customer-2
Here are some of my other articles that may interest you
Until next time…..
