Vcluster — Architecture Overview and Installation
For container orchestration, Kubernetes has become the de facto standard. You can deploy your microservices-based applications, databases, machine learning algorithms, or anything else on Kubernetes. As we are doing so many things on Kubernetes, there must be a scenario where we want to share our Kubernetes cluster among teams because spinning up a new cluster for everything is not a cost-efficient idea and it will also get difficult for us to manage multiple clusters.
This is where multi-tenancy comes into the picture. You must be thinking, "What is multi-tenancy in Kubernetes?" Multi-tenancy in Kubernetes means that instead of using different Kubernetes clusters for different purposes, like sharing a cluster between multiple users or deploying applications for different environments, we will use a single Kubernetes cluster and deploy multiple applications for different environments in a logically isolated cluster.
In this blog post, we are going to talk about what vcluster is and how it will help us achieve multi-tenancy in Kubernetes.
What is vcluster?
If we have multiple tenants using the same Kubernetes cluster, we will end up having multiple namespaces. And as multiple tenants are using the same Kubernetes cluster, the administrator of the cluster must have enforced restrictions on the tenants. They might not have cluster-wide permissions to create CRDs or ValidatingWebhooks, or they might not have permissions to list namespaces, create privileged resources, or create a third-party resource, this will end up making tenants frustrated.
This is where vcluster comes into the picture. Vcluster is an open-source project of Loft Labs. It is used to spin up a Kubernetes cluster inside a namespace. With vCluster on top, you get an automated solution for multi-tenancy. Vcluster resolves the problem that multi-tenants are facing, by providing each tenant with their own cluster, but those clusters will be virtual clusters running inside the Kubernetes cluster. The tenant will feel that they own the Kubernetes cluster, and they will have almost all the admin privileges inside their vcluster, without changing the privileges in the underlying cluster or without compromising the underlying cluster. It secures the main cluster because it encapsulates a lot of things that we do inside the vCluster and does not touch the underlying cluster. Compared to the Kubernetes cluster, vcluster does not have its own nodes, it deploys its workload on the underlying cluster.
Features of vcluster are: -
- Vclusters have their own API Server which provides more control and better isolation as compared to namespaces. As all the namespaces in a Kubernetes cluster use the same API server whereas every vcluster has its own API server.
- Multi-tenant environment: In vcluster, the tenant will have admin privilege and in namespace-based isolation, the tenant will have very restricted access.
- In vcluster, we can enable “pause” which will help to reduce the Infrastructure cost as well and we can “resume” the cluster whenever we want.
For example, let us assume your team is running multiple workloads on the cluster. With the help of vcluster, you can pause your environment, which will put vcluster in sleep mode. It will scale all the replica controlled resources to 0 (zero), like deployment, demonset and statefulset. And the nodes of the underlying Kubernetes will come back to the desired capacity.
#To pause vcluster
vcluster pause vcluster_name -n vcluster_namespace
To resume your vcluster, you just need to run the resume command, and it will scale all the replicas back to their desired count, or you can run the “vcluster connect” command, and it will automatically resume the vcluster.
#To resume your vcluster
vcluster resume vcluster_name -n vcluster_namespace
#or
vcluster connect vcluster_name
- You can create multiple vcluster in a single namespace ( Host namespace ) as well and if one vcluster goes down it will not affect the other cluster as they have their own API server.
- You can create a nested vcluster which means that you can create a vlcuster inside an already running vcluster.
- The objects which you have created inside a vcluster like Demonset, Deploymentsets, statefulset, CRDs, RBAC etc will remain inside the vcluster, they can never access the components of the control plane of the underlying host.
- The tenant can launch the privilege pod in vcluster and that privilege pod will not have any privilege on the underlying host.
- You can run the clusters in high availability mode.
Vcluster Architecture
Vcluster runs a statefulset inside a namespace on your underlying host. This statefulset runs a pod with two containers running inside it. The component of these 2 containers are: -
- Control Plane.
- API Server
- Controller Manager
- Storage Backend
- Syncer
Control Plane: The default configuration of vcluster uses the API server and controller manager of k3s and SQLite as a data store. You can use different storage backends also like etcd, MySQL, SQLite, PostgreSQL etc.
Syncer: Vcluster uses the syncer in place of a scheduler to schedule the pod. Syncer copies the pods that are created in the vcluster on the underlying host. The scheduling of the vcluster’s pod is taken care of by the scheduler of the underlying host. Syncer keeps the vcluster pod and underlying host pod in synchronisation.
CoreDNS: Each cluster has its own CoreDNS pod running inside the vcluster. When you create a vcluster, it will also spin up a DNS pod.
There are a few things that the underlying hosts take care of in your clusters:
- Storage Class: By default, the vcluster user can use the storage class of the host without creating a storage class inside the vcluster. You can change this behaviour by enabling sync of “storageclasses” or “hoststorageclasses”.
- Pod-to-pod or pod-to-service, communication is done by the underlying host.
- Vcluster uses Container runtime and Networking of the underlying host.
- Network Isolation: To stop communication of pods deployed in one vcluster to the pods of another vcluster or with the pods of the underlying host. We need to implement a network policy on the namespace of the underlying host.
- Resource Quota & Limit Range: To ensure that vcluster doesn’t end up consuming all the resources of the underlying host we can implement the Resource Quota & Limit Range on the namespace of the underlying host where vcluster is running.
In the above image, you can see that the API server, data store, controller manager, and syncer are running inside the vcluster. Custom-resource, deployment, and pods are created inside the demo-app namespace of the vcluster. Syncer is controlling how vcluster pods are getting scheduled on the host cluster. With the help of the scheduler of the host cluster, the syncer is scheduling the pods on the host cluster.
The cons of not having a scheduler in vcluster are: -
- If you taint a node inside a vcluster and try to schedule a pod based on that taint, it will have no effect on the scheduling of the pod.
- If you label a node inside the vcluster and try to schedule your pod based on that labelling it will not have any effect while scheduling the pod.
- Draining a node inside a vcluster will not have any effect on the scheduling of the pods.
If you have a requirement to manage the scheduling of your workload and you want to use affinity and topology spreading methods, then vcluster provides you with two options.
- Vcluster supports running the scheduler inside the vcluster instead of using the underlying cluster scheduler. You can enable this while creating your Vcluster.
- Vcluster also allows you to use the underlying cluster scheduler and customize the scheduling up to a certain degree. You can achieve this by following methods: -
* Automatically applying tolerations to all pods synced by cluster.
* Limiting pod scheduling to selected nodes.
* Using priority scheduling.
These options can also be enabled while creating the cluster.
Use Case of vcluster
There are so many scenarios where you can use the vcluster, a few of them are listed below: -
- You can use vcluster to create different working environments like dev, staging etc.
- You can use vcluster for multi-tenant applications.
- You can use vcluster to give a demo.
- You can use vlcuster to do a POC.
- Vclusters can be used for Database shardings.
- You can use vcluster for training and workshops.
- When we provision vcluster we can specify the Kubernetes version to run on it, therefore we can use vcluster for testing Kubernetes resources across different versions.
- We can use vcluster in CI/CD pipelines where we have the requirement to spin up a cluster for testing.
There are “N” numbers of scenarios other than those mentioned above where you can use the vcluster.
Installation
To deploy the vcluster on our Kubernetes cluster, first, we need to install the vcluster CLI on the base machine. If you are using Linux (AMD), you can follow the steps mentioned in this blog, or if you are using any other OS or OS architecture, you can follow the steps mentioned in the vcluster documentation.
curl -L-o vcluster “https://github.com/loft-sh/vcluster/releases/latest/download/vcluster-linux-amd64" && sudo install -c -m 0755 vcluster /usr/local/bin && rm -f vcluster
To verify the installation run the below-mentioned command: -
vcluster --version
assuming you have the vcluster CLI installed on your machine now and are ready to create the vcluster in the Kubernetes cluster.
To deploy the vcluster, make sure you have access to a Kubernetes cluster on which you are going to deploy the vlcuster and that you are able to run the kubectl command against that Kubernetes cluster from your machine, on which the vlcuster CLI is installed.
To install vcluster run the following command: -
# Create a new vcluster in namespace sagar
vcluster create sagar
Check the official documentation of vcluster to explore more options for deploying vcluster.
If you use the above command, it will launch a vcluster named “sagar” in the namespace “sagar” on the host cluster. It will create the namespace sagar if the namespace is not already created.
You can deploy your vcluster in a pre-created namespace, for this, you need to pass the name of the namespace while creating the cluster, as shown in the below-mentioned command.
vcluster create NAME_OF_YOUR_VCLUSTER --namespace demo --expose
By default, the cluster uses port forwarding to connect with the vCluster. But there are a few ways using which you can expose your vcluster, like NodePort, LoadBalancer, Ingress etc.
Once the cluster is created you can use the following command to connect and disconnect with the vcluster: -
To connect with vcluster run the following command: -
vcluster connect NAME_OF_YOUR_VCLUSTER
To disconnect vcluster run the following command: -
vcluster disconnect
On the underlying cluster, vcluster is created inside the namespace, if you want to see what component vcluster has deployed inside that namespace run the following command: -
kubectl get all -n vcluster_namespace
You will see that a statefulset pod is running, which contains two containers, and that one DNS pod is running.
If you see the name of the DNS pod or DNS service it will look something like this: -
service/kube-dns-x-kube-system-x-sagar
In the above naming convention, you can see that kube-dns is the name of the service, then “X” is used as a separator, then “kube-system” which is the name of the namespace in which the kube-dns pod is deployed, then again “X” for separation, and then the name of the namespace in which the vcluster is deployed, in our case it is “sagar”‘. Vcluster uses the same naming convention for all the pods and services that are created inside the vcluster.
If you check the IP of the pods running in the underlying cluster and the pods of the vlcuster both are using the same network, which means that the vcluster is using the CNI of the underlying cluster.
Using vcluster
As we mentioned earlier, Vcluster is the same as our normal Kubernetes cluster. We can do almost everything that we can do on our normal Kubernetes cluster, and in vCluster, the user has administrator privileges, so he can run almost every command.
For this blog, we will deploy a demo application inside the vcluster, and then we will try to access the vcluster.
The manifest that we have used to deploy the cluster is given below, you can use this manifest or you can use your manifest.
---
#service
apiVersion: v1
kind: Service
metadata:
labels:
app: demo-sagar
name: demo-sagar
spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: demo-sagar
type: LoadBalancer
---
#deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-sagar
spec:
replicas: 1
selector:
matchLabels:
app: demo-sagar
template:
metadata:
labels:
app: demo-sagar
spec:
containers:
- name: spring
image: sagar27/petclinic-demo
ports:
- containerPort: 8080
To verify the status of the deployment you can use the below-mentioned command: -
kubectl get all
And you will get an output like this
You can see in the above screenshot that our deployment is deployed and the service “LoadBalancer” is created inside the vcluster.
On the base cluster, if you check the namespace where vcluster is deployed. It has deployed a pod named “sagar-0” and service type LoadBalancer.
And you can use this loadbalancer IP/URL to access your application
You can use any service type to expose your application.
Clean Up

To remove your vcluster deployment, you can just run the delete command, and it will delete your vcluster along with all the resources that you have deployed in the vcluster.
vcluster delete NAME_OF_YOUR_VCLUSTER
I hope you found this post informative and engaging. I would love to hear your thoughts on this post, so do start a conversation on Twitter or LinkedIn :)
Here are some of my other articles that you may find interesting.
Until Next time…..