Deep Dive into Cortex Metrics Part II
Deep Dive into Cortex Metrics.io Part II
In Part I of this article we have understood the various components of Cortex and its use cases. In this Part II, we will configure Cortex with AWS S3 as its backend bucket storage and understand how metrics can be retained for longer periods using Cortex. We will also configure Grafana to use the Cortex Query Frontend to visualize graphs from various Prometheus clusters.
What is the entire story all about? (TLDR)
- Install Cortex using the Official helm chart.
- Configure Cortex to use AWS S3 as its Object Store.
Prerequisites
- A Kubernetes Cluster ( Can be either On-Prem, AKS, EKS, GKE, Kind ).
- Helm, kubectl installed.
What is not covered in this article
- How to set up a Production-based Cortex cluster.
- Compare the various monitoring solutions in the market ( Thanos vs Cortex )
Story Resources
- GitHub Link: https://github.com/pavan-kumar-99/medium-manifests
- GitHub Branch: cortex
Installing Prometheus Operator.
Well, if you use a package manager like helm, Cortex installation is made easier. Cortex has an official helm chart that can be used to Install a Cortex cluster on a Kubernetes cluster. You can find the helm chart here.
Before we install the Cortex cluster, let us Install the Prometheus Operator using the helm chart here. Later the Prometheus Instance will then be configured with a remote Write configuration to push it to the Cortex Ingesters. But for now, let us install the Prometheus Operator as per your customization. The values for this chart can be configured here. I have however installed consul in a non HA mode. The helm values can be found here.
$ helm repo add prometheus-community \ https://prometheuscommunity.github.io/helm-charts$ helm repo update$ helm upgrade -i prom prometheus-community/kube-prometheus-stack
Installing Cortex in the Kubernetes cluster
The Prometheus Operator is now installed. Let us now install Cortex using the official helm chart. But before that cortex requires the consul to be installed, I have used the official helm chart from the Harshicorp repository. You can follow the steps mentioned in the document here in the cortex namespace.
As mentioned earlier, for this cortex installation we would be using the AWS S3 bucket as the backend object storage. I have created an AWS secret key and access key from the IAM console and added them to my cortex-values.yaml ( Line 25, 26 ) file and also added the consul URL ( Line 18 ), S3 bucket name ( Line 24 ), and I have also enabled Prometheus Service Monitors with custom labels.
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex$ helm repo add cortex-helm \
https://cortexproject.github.io/cortex-helm-chart$ helm repo update $ helm install cortex --namespace cortex cortex-helm/cortex \
-f cortex-values.yaml
Cortex and Consul Components in the cortex namespace
Let us now see, the services present in the cortex namespace. For now, we should consider paying more attention to the following services.
a) cortex-nginx ( http://192.168.29.168 )
b) cortex-query-frontend ( http://192.168.29.171 )
Feel free to pause here and understand the components in Cortex from the Part-I of this series.
Installing Prometheus Instance using Prometheus Operator ( With RemoteWrite Configuration ).
You may use the Prometheus Custom Resource Definition to create a Prometheus Instance. But the new change that we bring over here is the addition of these 2 lines.
If you notice the URL clearly, you will realize that the URL is the cortex-nginx IP. Now the understanding should be easier. We are sending all the metrics from Prometheus to cortex-nginx which in turn sends to the rest of the cortex components.
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex$ kubectl apply -f prometheus-rbac.yaml$ kubectl apply -f prometheus-monitoring.yaml
Now, we should see all the metrics being written to cortex nginx pods, and the metrics are now being processed by the rest of the cortex components. Multiple Prometheus Instances from different namespaces can also push to this cortex instance.
Cortex Front End Querier UI?
But how do I get all of these metrics at a single source? Unfortunately, Cortex doesn't have a Query Frontend UI like Thanos. So we will have to deploy another Prometheus Instance with Remote Read Configuration. This configuration will read from the Cortex Query Frontend EndPoint. If you now see line 23, you should realize the fact that the IP is the IP of cortex-query-frontend.
$ git clone https://github.com/pavan-kumar-99/medium-manifests.git \
-b cortex$ kubectl apply -f prometheus-remote-read.yaml
You should now see the metrics from all the various Prometheus clusters being visible here. Isn’t that cool? Yes, it is !!
And now the time comes to verify the main thing. Yeah, the retention. I have been running cortex metrics in my home cluster for quite a long time. I now wanted to get the compute resources utilization for the past 30 days. Can I do it? Yes, obviously because I have already configured cortex to use AWS S3 bucket as its backend block storage. Ahh, here is the Data.
Cortex Performance Dashboard:
There’s a beautiful dashboard here, to visualize the performance of Cortex in Grafana. I have tuned it by changing a couple of panels and queries and making it look more attractive. I have added that to my repo, feel free to suggest some modifications to the Dashboard.
Conclusion
Thus, with the help of Cortex and Prometheus, one can set up a HA metrics aggregation solution. While there are many other solutions available in the market, the cortex is one of the solutions which is used by many companies in their Production environment. Initially, I had a hard time understanding every component of Cortex. But exploring each of the components individually has given me greater insights into Cortex Metrics. Feel free to share your experiences with Cortex in the comments section below.
Until next time…..
