How to deploy Prometheus on GKE cluster

Reading Time: 4 minutes

Hello readers, I’ll be covering about the details of How to deploy Prometheus on GKE cluster.

Scenario

A GKE cluster where an application is running and that application need to be monitored.

Prometheus

An open-source system monitoring and alerting toolset called Prometheus was created at SoundCloud.
Prometheus has been used by several businesses and organisations since its launch in 2012, and the project has a thriving developer and user community.
It is now an autonomous open source project that is maintained by no particular business.
Prometheus, the second hosted project after Kubernetes, joined the Cloud Native Computing Foundation in 2016 to underline this and make the project’s governance structure clear.

Prometheus collects and stores its metrics as time series data, which means that the information about the metrics is kept together with the timestamp at which it was captured and optional key-value pairs known as labels.

prerequisite

create a K8s cluster

1. After creating your cluster, you need to get authentication credentials to interact with the cluster
2. An application deployed on k8s engine which is already exposing prometheus metrics.

gcloud container clusters get-credentials <clustername>

Deploy the application on k8s cluster

Let’s create a manifest file named app.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: prom-example
  labels:
    app: prom-example
spec:
  selector:
    matchLabels:
      app: prom-example
  replicas: 3
  template:
    metadata:
      labels:
        app: prom-example
    spec:
      nodeSelector:
        kubernetes.io/os: linux
        kubernetes.io/arch: amd64
      containers:
      - image: nilebox/prometheus-example-app@sha256:dab60d038c5d6915af5bcbe5f0279a22b95a8c8be254153e22d7cd81b21b84c5
        name: prom-example
        ports:
        - name: metrics
          containerPort: 1234
        command:
        - "/main"
        - "--process-metrics"
        - "--go-metrics"

Apply the yml file using command

kubectl apply -f apps.yaml

check the deployment using command

kubectl get deployment

deploy helm chart(prometheus-community/kube-prometheus-stack) in a different namespace

Get Helm Repository Info

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo update

now install the helm chart of prometheus operator.(prometheus-operator is the name of the namespace)

helm install prometheus prometheus-community/kube-prometheus-stack -n=prometheus-operator --create-namespace

check the status of the namespace.

kubectl get namespace

You can then use kubectl describe crd to get a description of the CRD. And of course kubectl get crd -o yaml to get the complete definition of the CRD.

now , check the metrics of the applicaton by port forwarding to 5000 .

 kubectl port-forward deployment/prom-example 5000:1234

open a new tab on your browser and type

http://localhost:5000/metrics

Creating service file for application

name the manifest file applicationservice.yaml

apiVersion: v1
kind: Service
metadata:
  name: applicationservice
  labels:
    name: applicationservice
    
spec:
  selector:
    app: prom-example
  ports:
  - port: 5000
    targetPort: 1234
    name: port-name

apply the manifest file using command

kubectl apply -f applicationservice.yaml

Creating cluster role, role binding, and service account

The Prometheus server requires access to the Kubernetes API in order to scrape targets and reach the Alertmanager clusters. As a result, in order to allow access to those resources, a ServiceAccount must be created and linked to a ClusterRole:

name the manifest file rbac.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
  namespace: default
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

apply this manifest file using command

kubectl apply -f rbac.yaml

• Check that the role was created and bound to the ServiceAccount.

kubectl describe clusterrolebinding prometheus

Creating service monitor resource

Create a file service-monitor.yaml with the following content to add a ServiceMonitor so that the Prometheus server scrapes only its own metrics endpoints.

name the manifest file service-monitor.yaml

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: prometheus
  labels:
    name: prometheus
spec:
  selector:
    matchLabels: 
      name: applicationservice  
  endpoints:
    - port: port-name

    

apply this manifest file using command

kubectl apply -f service-monitor.yaml

Creating prometheus custom resource

After creating the Prometheus ServiceAccount and giving it access to the Kubernetes API, we can deploy the Prometheus instance.

Create a mainfest file prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  
  serviceMonitorSelector: 
    matchLabels:
      name: prometheus
  resources:
    requests:
      memory: 400Mi

apply this manifest file using command

kubectl apply -f prometheus.yaml

Grafana Dashboard

Forward the port to access the grafana dashboard using command

kubectl port-forward svc/prometheus-grafana 9000:80 --namespace=prometheus-operator

• Go to your browser and type 127.0.0.1:9000
• for username and password type these commands on your terminal

kubectl get secrets -n prometheus-operator
kubectl get secrets prometheus-grafana --namespace=prometheus-operator -o yaml

decode the username and password using command

echo cHJvbS1vcGVyYXRvcg== | base64 --decode

finally open your dashboard of grafana and follow the given steps :

1. Go to the settings
2. Click on “Data Sources
3. Click on “Add data source”
4. name : prometheus
5. URL : Set the appropriate Prometheus server URL (ex : http://prometheus-operated.default.svc.cluster.local:9090 )
6. Adjust other data source settings as desired (for example, choosing the right Access method). Click “Save & Test” to save the new data source.

Reference:

https://blog.knoldus.com/getting-started-with-prometheus/

https://www.tutorialspoint.com/kubernetes/kubernetes_monitoring.htm

In Conclusion:

In conclusion ,The blog explained how to deploy prometheus on GKE cluster.I hope you enjoyed this practical instruction. Motivate yourself to deploy more monitoring tools, link them, and utilize them while looking up more examples on Google.