Dataflow is a fully managed GCP product where developers can build pipelines to process a batch or stream of data from a source. It minimizes latency, processing time, and cost through autoscaling and batch processing. It uses compute engine instances for processing & analytics.
For monitoring & observability, you might be using Grafana as a single source for monitoring all services cloud or on-prem. Creating dashboards, alerts, and annotations is time taking process so the preferred quick way is to import JSON dashboard files & add variables to them.
- Helm v3
- GCP Project ID
- Dataflow Running Job
- Service Account & Key in JSON format ( If needs to add GCP stackdriver Datasource in Grafana)
Ways to create a dashboard on Grafana
- Import prebuilt dashboard JSON into grafana
- Adding & Creating panels from scratch
- Import JSON from Terraform scripts
In this blog we will create a dashboard by using a rebuild JSON file. And then add variables accordingly to customize the dashboard.
Steps to create Dataflow Grafana dashboard
1. Download Dashboard JSON
Download the JSON dashboard file by running the below command.
Note: If you already have a running Grafana instance. You can directly import this JSON file into grafana instance & create the dashboard. Otherwise, if you also want to deploy a Grafana instance follow below steps.
2. Clone Grafana Helm chart from repo
Clone the Grafana dashboard repo from this GitHub repo. Add the content of the above JSON file in this grafana/dashboards/dataflow-dashboard.json file at this path in the cloned repo.
git clone https://github.com/knoldus/grafana-dataflow-dashboard cd grafana-dataflow-dashboard
3. Deploy helm chart & import the dashboard JSON
Create a custom-values.yaml file to customize the input values in a helm chart. Add the below content in this file.
dashboardProviders: dashboardproviders.yaml: apiVersion: 1 providers: - name: 'default' orgId: 1 folder: '' type: file disableDeletion: false editable: true options: path: /var/lib/grafana/dashboards/default adminUser: admin adminPassword: admin dashboards: default: custom-dashboard: file: dashboards/dataflow-dashboard.json
Run Helm Upgrade command
Run this command to update or deploy a fresh Grafana instance to your Kubernetes.
helm upgrade --install grafana ./ -f ./custom-values.yaml -n grafana --create-namespace
Note: If you are deploying a fresh instance then you need to add Google Cloud Stackdriver Datasource in the Grafana.
Grafana Dataflow Dashboard
After setting up your new dataflow dashboard & Datasource it should look like this. You can either port-forward your Grafana port or use an ingress load balancer to access Grafana UI.
With this simple way, you can create a Grafana dashboard with the help of Helm Chart & prebuilt JSON file. You can either import JSON directly into existing Grafana or you can upgrade with the helm chart to create the dashboard.