Cinnamon: Way For Monitoring & Metrics Generation for Akka ActorSystem

Table of contents

Reading Time: 2 minutes

We are developing huge applications and deployed on multiple virtual machines and clusters. For monitoring applications, we need to enable logs into our application and analysis that logs with help of some tools like elastic stack.
But !! what if we need to check health of application on virtual machines and clusters? For that, we are using several Metrics for system health check like Gauges, Histograms and more.

Lightbend Telemetry gives us one of the way for metrics generation and monitoring systems(application) by using Cinnamon plugins. Today’s we look into Cinnamon for monitoring akka ActorSytem. Configuring cinnamon is not a rocket science, there are simple steps, which, we are going to define here.

The first step for using cinnamon is, we need to create and account on lightbend, from where we can download credentials and paste into our home directory(for linux users). All instructions are define in this link.

Note: For today’s example, we are using sbt project, but we can easily integrate with maven and gradle as well.

>>> We need to add cinnamon plugin in our plugin.sbt file as below:

This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters

Show hidden characters

	credentials += Credentials(Path.userHome / ".lightbend" / "commercial.credentials")

	resolvers += Resolver.url("lightbend-commercial",
	url("https://repo.lightbend.com/commercial-releases"))(Resolver.ivyStylePatterns)

	addSbtPlugin("com.lightbend.cinnamon" % "sbt-cinnamon" % "2.4.2")

view raw

plugins.sbt

hosted with ❤ by GitHub

>>> Now, require to add some dependencies to add in our build.sbt:

Show hidden characters

	val akkaVersion = "2.4.17"

	lazy val `akka-cinammon` = project in file(".") enablePlugins (Cinnamon)

	// Add the Cinnamon Agent for run and test
	cinnamon in run := true
	cinnamon in test := true

	// Set the Cinnamon Agent log level
	cinnamonLogLevel := "INFO"

	libraryDependencies ++= Seq(
	Cinnamon.library.cinnamonCHMetrics,
	Cinnamon.library.cinnamonAkka,
	"com.typesafe.akka" %% "akka-actor" % akkaVersion,
	"com.typesafe.akka" %% "akka-testkit" % akkaVersion,
	"org.scalatest" %% "scalatest" % "3.0.1"
	)

view raw

build.sbt

hosted with ❤ by GitHub

>>> Add cinnamon configuration on application.conf as below:

Show hidden characters

	cinnamon.akka {
	actors {
	"/user/*" {
	report-by = instance
	}
	}
	}
	cinnamon.chmetrics {
	reporters += "console-reporter"
	}

view raw

application.conf

hosted with ❤ by GitHub

There are lots of option for configure, monitoring to actor system, for more information please click on this link.

Example:

My example is a simple hello world actor application, but when we run that example, in console we are seeing cinnamon metrics. The example as below:

Show hidden characters

	package com.harmeetsingh13.actors

	import akka.actor.{Actor, ActorLogging, ActorSystem, Props}

	/**
	* Created by Harmeet Singh(Taara) on 1/7/17.
	*/
	class HelloWorldActor extends Actor with ActorLogging {

	override def receive: Receive = {
	case any =>
	Thread.sleep(1000)
	log.info(s"Hello $any")
	}
	}

	object HelloWorldActor extends App {

	val system = ActorSystem("HelloWorldSystem")
	val ref1 = system.actorOf(Props[HelloWorldSelectionActor])

	for (i <- 1 to 10000) ref1 ! "James"
	}

view raw

HelloWorldActor.scala

hosted with ❤ by GitHub

For running the application we need to execute sbt run command as below:

Show hidden characters

	harmeet@harmeet-Vostro-3558:~/akka-samples/akka-cinammon$ sbt run
	[info] Loading global plugins from /home/harmeet/.sbt/0.13/plugins
	[info] Loading project definition from /home/harmeet/akka-samples/akka-cinammon/project
	[info] Set current project to akka-cinammon (in build file:/home/harmeet/akka-samples/akka-cinammon/)
	[warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list

	Multiple main classes detected, select one to run:

	[1] com.harmeetsingh13.actors.HelloWorldActor
	[2] com.harmeetsingh13.actors.HelloWorldPoolActor
	[3] com.harmeetsingh13.actors.HelloWorldSelectionActor

	Enter number: 1

	[info] Running com.harmeetsingh13.actors.HelloWorldActor
	[info] [INFO] [07/01/2017 15:38:02.766] [CoreAgent] Cinnamon Agent version 2.4.2
	[error] SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
	[error] SLF4J: Defaulting to no-operation (NOP) logger implementation
	[error] SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
	[info] [INFO] [07/01/2017 15:38:05.029] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
	[info] [INFO] [07/01/2017 15:38:06.029] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
	[info] [INFO] [07/01/2017 15:38:07.030] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
	[info] [INFO] [07/01/2017 15:38:08.030] [HelloWorldSystem-akka.actor.default-dispatcher-4] [akka://HelloWorldSystem/user/$a] Hello James
	[info] 7/1/17 3:38:08 PM ==============================================================
	[info]
	[info] — Gauges ———————————————————————-
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.active-threads
	[info] value = 1
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.parallelism
	[info] value = 12
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.pool-size
	[info] value = 3
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.queued-tasks
	[info] value = 0
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.running-threads
	[info] value = 0
	[info]
	[info] — Counters ——————————————————————–
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.running-actors
	[info] count = 1
	[info]
	[info] — Histograms ——————————————————————
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-size
	[info] count = 10005
	[info] min = 15
	[info] max = 9993
	[info] mean = 5109.87
	[info] stddev = 2904.06
	[info] median = 5251.00
	[info] 75% <= 7591.00
	[info] 95% <= 9532.00
	[info] 98% <= 9841.00
	[info] 99% <= 9941.00
	[info] 99.9% <= 9991.00
	[info] metrics.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-time
	[info] count = 5
	[info] min = 416839
	[info] max = 4003915716
	——————————– more ————————————————

	[info] — Meters ———————————————————————-
	[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.actor-failure
	[info] count = 0
	[info] mean rate = 0.00 events/second
	[info] 1-minute rate = 0.00 events/second
	[info] 5-minute rate = 0.00 events/second
	[info] 15-minute rate = 0.00 events/second
	[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.dead-letter
	[info] count = 0
	[info] mean rate = 0.00 events/second
	[info] 1-minute rate = 0.00 events/second
	[info] 5-minute rate = 0.00 events/second
	[info] 15-minute rate = 0.00 events/second
	[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.log-error
	[info] count = 0
	[info] mean rate = 0.00 events/second
	[info] 1-minute rate = 0.00 events/second
	[info] 5-minute rate = 0.00 events/second
	[info] 15-minute rate = 0.00 events/second
	[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.log-warning
	[info] count = 0
	[info] mean rate = 0.00 events/second
	[info] 1-minute rate = 0.00 events/second
	[info] 5-minute rate = 0.00 events/second
	[info] 15-minute rate = 0.00 events/second
	[info] events.akka.systems.HelloWorldSystem.dispatchers.akka_actor_default-dispatcher.actors._user_$a.mailbox-size-limit
	[info] count = 0
	[info] mean rate = 0.00 events/second
	[info] 1-minute rate = 0.00 events/second
	[info] 5-minute rate = 0.00 events/second
	[info] 15-minute rate = 0.00 events/second

view raw

console

hosted with ❤ by GitHub

As in logs, we have seeing, 4 types of metrics are there. Every metrics have its own befits and analysis. These metrics gives us a report on akka actor systems like actors count, threads count, mailbox capacity and more.

For more examples, you can check github repo.

NOTE: Cinnamon can easily integrated with DATADOG. For more please click on this link.