Monitoring Akka based applications with Cinnamon and DataDog

Table of contents

Reading Time: 5 minutes

Application monitoring is an important part of software development life cycle and aims to check if the application is performing as expected or not. With applications becoming Reactive, having distributed architecture and industry moving towards Fastdata, there is a great urge in the market for tools and integrations that could provide cutting edge monitoring facility for the application deployed in the production.

In this blog we will explore how Lightbend Telemetry could be used to efficiently monitor the performance of Akka based applications running on a single node. Telemetry makes it possible to gather metric, event and trace information from Akka, Scala, and Lagom based applications, I have choosed Akka based application for this blog as Akka provide an efficient and convenient way to implement Reactive and Fastdata application.
Lightbend Monitoring has two essential parts :-

1). Telemetry (Cinnamon) ->

By definition, telemetry means the process of recording and transmitting the readings of an instrument/system. Lightbend Telemetry (a.k.a. Cinnamon) collects information(metric, event and trace), in runtime, about application based on provided configuration. Cinnamon is a Java agent that runs in the same JVM as the application. Using Lightbend Telemetry is free during development, but one must have a valid license to use it in production.

2). Backend plugins and Visualization tools ->

Cinnamon collects metrics from the Akka-based Reactive application and reports them to the integrated backend. Various options among backend plugins are OpsClarity, DataDog, CodaHale Matrics, OverOps etc. Visualization means graphically depicting the collected metrics and could be achieved via OpsClarity, DataDog, grafana etc.

For this blog we would be using DataDog as our Backend / Visualization plugin; it provides a predefined dashboard to jump-start with monitoring of Reactive applications. Let us now get into action, following steps could be followed to start monitoring the application :-

Step 1 “Generate Bintray Credentials” ->

Bintray credentials are required to get access to the required libraries for Cinnamon.

1). Login to your lightbend account and generate the Bintray Credentials from here.

2). On your system create a directory named .lightbend and in it create a file commercial.credentials with bintray credentials you just generated. You may choose any directory structure for your credential file; ~/.lightbend/commercial.credentials is the Lightbend convention. The content of the file should be as follows.


realm = Bintray
host = dl.bintray.com
user = bintray_user_name
password = bintray_password

Step 2 “Setting up the project to use the Cinnamon Agent” ->

I have cloned this simple Akka-Scala application from github to demonstrate functioning of telemetry. In your Akka based application :-

1). Update the sbt version to 0.13.7 or higher in build.properties file and update Scala version to 2.11.8 or higher(I have not tried with Scala version lower than this) in build.sbt.

2). Create (if not existing) plugins.sbt file in the project directory of your application and add following code into it :-


addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.2") // Cinnamon plugin

credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials") // Specify the path of the file created in step 1 above which contains bintray credentials

resolvers += Resolver.url("lightbend-commercial",
 url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)

3). Lightbend Telemetry currently supports Akka 2.3.x/2.4.x/2.5.x; set the Akka version of your project accordingly.

4). Add following library dependency and configurations for Cinnamon in your build.sbt


libraryDependencies += Cinnamon.library.cinnamonAkka //Akka based application monitoring via Cinnamon

lazy val app = project in file(".") enablePlugins Cinnamon
cinnamon in run := true
cinnamon in test := true
cinnamonLogLevel := "INFO"

Step 3 “Configuring Cinnamon Core and selecting actors for telemetry” ->

Since your application could create thousands or even tens of thousands of actors, the default setting is configured with actor telemetry turned off. You need to select which actors to observe in your application.conf file (present in src/main/resources) which is the same file used to set up your Akka system. Some metrics support configured thresholds. If the threshold is exceeded then an event is recorded.
With respect to my project, following Actor configuration is defined in application.conf file


cinnamon.akka {
  actors {
   "com.example.*" { //All the actors present under com.example package would be monitored
     report-by = class
     thresholds { // Thresholds for metrics are defined here
       mailbox-size = 1000
       stash-size = 50
       mailbox-time = 3s
       processing-time = 500ms
     }
   }
 }
}

Refer the lightbend documentation here for possible ways for Actor selection and settings.

Step 4 “Backend Integration” ->

I am using DataDog as my backend and visualization tool because its hot in the market and provides a great predefined dashboard for awesome visualization of the collected matrices. Following steps enables Backend integration :-

1). Create an account on DataDog. Since DataDog is a paid privilege, I have taken their 14 day trail for this blog.

2). Install a DataDog agent on the host machine where your application would be running. The host machine could either be a cloud or your local machine, you just need to be connected with internet. The DataDog agent could be installed from here based on your operating system.

3). Under Integrations menu in DataDog website, search for lightbend reactive platform and install the integration.

4). Add following library dependency for DataDog


libraryDependencies += Cinnamon.library.cinnamonDataDog //For using DataDog as backend agent

5). Add following configuration to application.conf file of your project.


cinnamon.datadog {
  statsd {
    host = "127.0.0.1"
    port = 8125
    frequency = 60s
  }
  report {
    histogram = ["min", "max", "p98", "p99", "p999"]
  }
}

Here, statsd is the matric collector(DogStatsD for DataDog) and since DataDog agent is running on the same machine on which the Akka application is running, statsd host is 127.0.0.1 (localhost). 8125 is the default listening port of DogStatsD. Frequency is the time interval after which the DataDog agent sends the matrices to your DataDog account. We may change defaults by altering the /etc/dd-agent/datadog.conf file followed by DataDog agent restart.

All set to test monitoring

Run your Akka application with some load,if possible,to test how actors are performing. Now log into your DataDog account and Under Dashboard menu select Lightbend Akka Overview dashboard from Integration Dashboards in the right pane of the page.

My dashboard with 2 actors looks as follows

Conclusion ->

In this blog we discussed how application monitoring is important specially for reactive applications. We then saw how Cinnamon could be used to monitor Akka-based applications using DataDog as Backend as well as visualization tool.

You may view a sample application with latest sbt, Scala and Akka versions.

Hope this compilation helps you. Share your thoughts in the comments below.

Happy Monitoring