Transport Cinnamon Matrices From Lagom To Prometheus

Monitoring is a pain when it comes to distributed applications, and even more when you have shared or non-shared variables to monitor in your application.

Here in this blog, I’ll explain two tools which can ease the monitoring efforts, one for generating metrics called Cinnamon and other to visualize them, called Prometheus.

Let’s have a quick brief intro about these two –

Prometheus – An open-source monitoring system with a dimensional data model, flexible query language, efficient time series database and modern alerting approach.

Cinnamon Metrics – (a.k.a Lightbend Telemetry) provides insight into applications built with Lightbend technologies. It does so by instrumenting frameworks and toolkits such as Akka. The instrumentation is done by a Java agent that runs when your application is starting up. It collects information, at runtime, about your application based on a configuration that you must provide. One can use Lightbend Telemetry for free during development, but you must have a valid license to use it in production.

In this blog, we will see how we can export our cinnamon metrics into the monitoring tool Prometheus. For details like generating the Cinnamon Metrics, please visit the below blogs –

Monitor Lagom using LightBend Telemetry a.k.a Cinnamon Metrics (coming soon)

Cinnamon Way For Monitoring Metrics For Akka Actor System

So let’s start the configuration –

CONFIGURATION AT LAGOM SIDE- 

1. To ship the metrics into Prometheus one need an exporter. A simple example of shipping Metrics into ElasticSearch (ES) one needs an ES exporter, for example –

/* Configuration for Cinnamon to ElasticSearch */
cinnamon {
  chmetrics {
    reporters += "elasticsearch-reporter"
    elasticsearch-reporter {
      hostString = "http://127.0.0.1:9200"
      index-date-format = "yyyy-MM-dd"
      basic-auth {
        username = "YourESUsername"
        password = "Password"
      }
    }
  }
}

Similarly, you will require and Prometheus Exporter –

/* Configuration for Cinnamon to Prometheus */
cinnamon {
    prometheus {
    exporters += http-server
    http-server {
      host = "localhost"
      port = 9001
    }
  }
}

This configuration will start exporting your cinnamon metrics defined in the Lagom code at your defined host and port. For more details on how to Generate Cinnamon Metrics in Lagom please visit this blog –  Monitor Lagom using LightBend Telemetry a.k.a Cinnamon Metrics. (coming soon)

Now let’s do the configuration at Prometheus level –

On Prometheus, these metrics can also be checked –
http://hostname:9090/graph

where hostname is the IP address where Prometheus is installed.

CONFIGURATION AT PROMETHEUS SIDE- 

We are using the property “file_sd_config” in Prometheus configuration which fetches the properties defined in the file as mentioned and ships them into the server.

- job_name: lagom
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  file_sd_configs:
  - files:
    - /prometheus/config/lagom.yml
    refresh_interval: 5m

This points to a config file named – lagom.yml which contains the path where the lagom is exporting the cinnamon metrics.

This is the best way to define the TARGET for the Prometheus where you use a file and in that file defined your URL’s or other configurations.

Other ways are to hardcode the path e.g –

- job_name: kafka_exporter
  scrape_interval: 15s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost1:port1
    - localhost2:port2

In our case, Lagom is running on DCOS on top of conductR so the address will be something like  –

cat /opt/prometheus/config/lagom.yml
- targets: [ "lagom-impl-snapshot-8d8q1-s3.conductr-2.1.14.slave.mesos:9001" ]
  labels:
    environment: test4

It can vary from where and how you are deploying your App.

That’s all for today, See you guys in the next blog, where we will see how one can dynamically fetch the metrics from Lagom app running on DCOS.