Cinnamon: Way For Monitoring & Metrics Generation for Akka ActorSystem

Reading Time: 2 minutes

We are developing huge applications and deployed on multiple virtual machines and clusters. For monitoring applications, we need to enable logs into our application and analysis that logs with help of some tools like elastic stack.
But !! what if we need to check health of application on virtual machines and clusters? For that, we are using several Metrics for system health check like Gauges, Histograms and more.

Lightbend Telemetry gives us one of the way for metrics generation and monitoring systems(application) by using Cinnamon plugins. Today’s we look into Cinnamon for monitoring akka ActorSytem. Configuring cinnamon is not a rocket science, there are simple steps, which, we are going to define here.

The first step for using cinnamon is, we need to create and account on lightbend, from where we can download credentials and paste into our home directory(for linux users). All instructions are define in this link.

Note: For today’s example, we are using sbt project, but we can easily integrate with maven and gradle as well.

>>> We need to add cinnamon plugin in our plugin.sbt file as below:

credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")
resolvers += Resolver.url("lightbend-commercial",
url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.2")

view raw
plugins.sbt
hosted with ❤ by GitHub

>>> Now, require to add some dependencies to add in our build.sbt:

val akkaVersion = "2.4.17"
lazy val `akka-cinammon` = project in file(".") enablePlugins (Cinnamon)
// Add the Cinnamon Agent for run and test
cinnamon in run := true
cinnamon in test := true
// Set the Cinnamon Agent log level
cinnamonLogLevel := "INFO"
libraryDependencies ++= Seq(
Cinnamon.library.cinnamonCHMetrics,
Cinnamon.library.cinnamonAkka,
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-testkit" % akkaVersion,
"org.scalatest" %% "scalatest" % "3.0.1"
)

view raw
build.sbt
hosted with ❤ by GitHub

>>> Add cinnamon configuration on application.conf as below:

cinnamon.akka {
actors {
"/user/*" {
report-by = instance
}
}
}
cinnamon.chmetrics {
reporters += "console-reporter"
}

view raw
application.conf
hosted with ❤ by GitHub

There are lots of option for configure, monitoring to actor system, for more information please click on this link.

Example:

My example is a simple hello world actor application, but when we run that example, in console we are seeing cinnamon metrics. The example as below:
package com.harmeetsingh13.actors
import akka.actor.{Actor, ActorLogging, ActorSystem, Props}
/**
* Created by Harmeet Singh(Taara) on 1/7/17.
*/
class HelloWorldActor extends Actor with ActorLogging {
override def receive: Receive = {
case any =>
Thread.sleep(1000)
log.info(s"Hello $any")
}
}
object HelloWorldActor extends App {
val system = ActorSystem("HelloWorldSystem")
val ref1 = system.actorOf(Props[HelloWorldSelectionActor])
for (i < 1 to 10000) ref1 ! "James"
}

For running the application we need to execute sbt run command as below:
harmeet@harmeet-Vostro-3558:~/akka-samples/akka-cinammon$ sbt run
[info] Loading global plugins from /home/harmeet/.sbt/0.13/plugins
[info] Loading project definition from /home/harmeet/akka-samples/akka-cinammon/project
[info] Set current project to akka-cinammon (in build file:/home/harmeet/akka-samples/akka-cinammon/)
[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list
Multiple main classes detected, select one to run:
[1] com.harmeetsingh13.actors.HelloWorldActor
[2] com.harmeetsingh13.actors.HelloWorldPoolActor
[3] com.harmeetsingh13.actors.HelloWorldSelectionActor
Enter number: 1
[info] Running com.harmeetsingh13.actors.HelloWorldActor
[info] [INFO] [07/01/2017 15:38:02.766] [CoreAgent] Cinnamon Agent version 2.4.2
[error] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
[error] SLF4J: Defaulting to no-operation (NOP) logger implementation
[error] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
[info] [INFO] [07/01/2017 15:38:05.029] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
[info] [INFO] [07/01/2017 15:38:06.029] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
[info] [INFO] [07/01/2017 15:38:07.030] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
[info] [INFO] [07/01/2017 15:38:08.030] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
[info] 7/1/17 3:38:08 PM ==============================================================
[info]
[info] — Gauges ———————————————————————-
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.active-threads
[info] value = 1
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.parallelism
[info] value = 12
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.pool-size
[info] value = 3
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.queued-tasks
[info] value = 0
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.running-threads
[info] value = 0
[info]
[info] — Counters ——————————————————————–
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.running-actors
[info] count = 1
[info]
[info] — Histograms ——————————————————————
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-size
[info] count = 10005
[info] min = 15
[info] max = 9993
[info] mean = 5109.87
[info] stddev = 2904.06
[info] median = 5251.00
[info] 75% <= 7591.00
[info] 95% <= 9532.00
[info] 98% <= 9841.00
[info] 99% <= 9941.00
[info] 99.9% <= 9991.00
[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-time
[info] count = 5
[info] min = 416839
[info] max = 4003915716
——————————– more ————————————————
[info] — Meters ———————————————————————-
[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.actor-failure
[info] count = 0
[info] mean rate = 0.00 events/second
[info] 1-minute rate = 0.00 events/second
[info] 5-minute rate = 0.00 events/second
[info] 15-minute rate = 0.00 events/second
[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.dead-letter
[info] count = 0
[info] mean rate = 0.00 events/second
[info] 1-minute rate = 0.00 events/second
[info] 5-minute rate = 0.00 events/second
[info] 15-minute rate = 0.00 events/second
[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.log-error
[info] count = 0
[info] mean rate = 0.00 events/second
[info] 1-minute rate = 0.00 events/second
[info] 5-minute rate = 0.00 events/second
[info] 15-minute rate = 0.00 events/second
[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.log-warning
[info] count = 0
[info] mean rate = 0.00 events/second
[info] 1-minute rate = 0.00 events/second
[info] 5-minute rate = 0.00 events/second
[info] 15-minute rate = 0.00 events/second
[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-size-limit
[info] count = 0
[info] mean rate = 0.00 events/second
[info] 1-minute rate = 0.00 events/second
[info] 5-minute rate = 0.00 events/second
[info] 15-minute rate = 0.00 events/second

view raw
console
hosted with ❤ by GitHub

As in logs, we have seeing, 4 types of metrics are there. Every metrics have its own befits and analysis. These metrics gives us a report on akka actor systems like actors count, threads count, mailbox capacity and more.

For more examples, you can check github repo.

NOTE: Cinnamon can easily integrated with DATADOG. For more please click on this link.

Written by 

Harmeet Singh is a lead consultant, with experience of more than 5 years. He has expertise in Scala, Java, JVM, and functional programming. On a personal front; he is a food lover.

2 thoughts on “Cinnamon: Way For Monitoring & Metrics Generation for Akka ActorSystem2 min read

Comments are closed.

%d bloggers like this: