How to create Grafana Dataflow Dashboard

Table of contents

Reading Time: 3 minutes

Dataflow is a fully managed GCP product where developers can build pipelines to process a batch or stream of data from a source. It minimizes latency, processing time, and cost through autoscaling and batch processing. It uses compute engine instances for processing & analytics.

For monitoring & observability, you might be using Grafana as a single source for monitoring all services cloud or on-prem. Creating dashboards, alerts, and annotations is time taking process so the preferred quick way is to import JSON dashboard files & add variables to them.

Prerequisites

Helm v3
GCP Project ID
Dataflow Running Job
Service Account & Key in JSON format ( If needs to add GCP stackdriver Datasource in Grafana)

Ways to create a dashboard on Grafana

Import prebuilt dashboard JSON into grafana
Adding & Creating panels from scratch
Import JSON from Terraform scripts

In this blog we will create a dashboard by using a rebuild JSON file. And then add variables accordingly to customize the dashboard.

Steps to create Dataflow Grafana dashboard

1. Download Dashboard JSON

Download the JSON dashboard file by running the below command.

wget https://raw.githubusercontent.com/knoldus/grafana-dataflow-dashboard/master/dataflow.json

Note: If you already have a running Grafana instance. You can directly import this JSON file into grafana instance & create the dashboard. Otherwise, if you also want to deploy a Grafana instance follow below steps.

2. Clone Grafana Helm chart from repo

Clone the Grafana dashboard repo from this GitHub repo. Add the content of the above JSON file in this grafana/dashboards/dataflow-dashboard.json file at this path in the cloned repo.

git clone https://github.com/knoldus/grafana-dataflow-dashboard
cd grafana-dataflow-dashboard

3. Deploy helm chart & import the dashboard JSON

Create a custom-values.yaml file to customize the input values in a helm chart. Add the below content in this file.

custom-values.yaml

dashboardProviders:
 dashboardproviders.yaml:
   apiVersion: 1
   providers:
   - name: 'default'
     orgId: 1
     folder: ''
     type: file
     disableDeletion: false
     editable: true
     options:
       path: /var/lib/grafana/dashboards/default

adminUser: admin
adminPassword: admin

dashboards:
  default:
    custom-dashboard:
      file: dashboards/dataflow-dashboard.json

Run Helm Upgrade command

Run this command to update or deploy a fresh Grafana instance to your Kubernetes.

helm upgrade --install grafana ./ -f ./custom-values.yaml -n grafana --create-namespace

Note: If you are deploying a fresh instance then you need to add Google Cloud Stackdriver Datasource in the Grafana.

Grafana Dataflow Dashboard

After setting up your new dataflow dashboard & Datasource it should look like this. You can either port-forward your Grafana port or use an ingress load balancer to access Grafana UI.

Conclusion

With this simple way, you can create a Grafana dashboard with the help of Helm Chart & prebuilt JSON file. You can either import JSON directly into existing Grafana or you can upgrade with the helm chart to create the dashboard.