Monitor a Kafka stream application with Graphite-Grafana using JMX metrics

Table of contents

Reading Time: 5 minutes

A few days back, we got the requirement that we need to monitor a Kafka stream application using JMX metrics. We looked for the solution and reached to the conclusion which we will discuss in this blog. I will try to explain each and every component of the solution along with the setup and the integration part of the whole system.

Proposed solution:

Service (application) exposes the JMX metrics at some port which will be captured by Jolokia java agent. Then Jolokia exposes those metrics at some port which is easily accessible through a rest endpoint (we call it Jolokia URL). Then we have JMX2Graphte which polls the metrics from Jolokia URL and push it to Graphite. Then Grafana reads the Graphite metrics and creates a beautiful dashboard for us along with the alerts.

So this is the working of the proposed monitoring solution. Now let’s discuss the components of the monitoring solution.

As you can see in the above diagram, we have the following components for the monitoring solution:

JMX Metrics
Jolokia agent
JMX2Graphite
Graphite
Grafana

Now we will see each component one by one, in brief, so let’s start:

JMX Metrics:

Java Management Extensions (JMX) is a standard component of the Java Standard Edition, which provides a simple, standard way of managing resources such as applications, devices, and services at runtime. We can also use JMX technology to monitor and manage the Java Virtual Machine (Java VM).

JMX uses MBeans to expose the metrics. An MBean is a managed Java object, similar to a JavaBeans component, that represents a device, an application, or any resource that needs to be managed

A JMX agent consists of an MBean server, in which MBeans are registered, and a set of services for handling the MBeans. In this way, JMX agents directly control resources and make them available to remote management applications.

JMX architecture:

The above diagram shows the JMX architecture.

There are 3 different levels: Probe, Agent, and the Remote management level.

It has MBeans at the Probe level, MBeans server at Agent level and there are a lot of connectors available to see the JMX metric which is at remote management level.

Enable JMX:

To enable the JMX metrics, run the MyJar as follows:

java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=8008
-Dcom.sun.management.jmxremote.authenticate=false MyJar.jar

There are so many options which we can provide to the java command. We are not going into much more detail to this. You can take a look here.

JConsole:

The JConsole graphical user interface is a monitoring tool that complies to the Java Management Extensions (JMX) specification. JConsole uses the extensive instrumentation of the Java Virtual Machine (Java VM) to provide information about the performance and resource consumption of applications running on the Java platform.

The JConsole executable is in JDK_HOME/bin, where JDK_HOME is the directory where the JDK is installed. If this directory is on your system path, you can start the tool by simply typing jconsole in a command (shell) prompt. Otherwise, you have to type the full path to the executable file.

You will get the below screen after typing jconsole on terminal:

Here, you can connect to local as well remote process and can see the JMX metrics. You can get more details here.

Jolokia Agent:

Jolokia is an agent-based approach to JMX, which requires that clients install an extra piece of software, the so-called agent. This software either needs to be deployed on the target server which should be accessed via remote JMX or it can be installed on a dedicated proxy server. It comes with different kind of flavors of agents which you can get here.

Here we are going to discuss the JVM flavor of the agent.

Attach as java agent:

We start the agent by providing a simple startup option -javaagent to the java process as follows:

java -javaagent:/jolokia-jvm-1.6.0-agent.jar=host=localhost;port=8778

Now you can access the Jolokia at http://localhost:8778/jolokia/ and get the JMX metrics.

If you are running your service inside docker then just pass the above option in exec command as follows:

exec java -cp “$APP_LIB_DIR/*” -javaagent:/jolokia-jvm-1.6.0-agent.jar=host=0.0.0.0 com.knoldus.monitoring.StreamsExecutor

JMX2Graphite:

JMX2Graphite is a tool which polls the metrics from the Jolokia URL and pushes them into Graphite. We need to provide Graphite Host and Jolokia URL while running the JMX2Graphite.

docker run -i -t -d –name jmx2graphite –network=host \
-e “JOLOKIA_URL=http://127.0.0.1:8778/jolokia/” \
-e “SERVICE_NAME=Demo_app” \
-e “GRAPHITE_HOST=127.0.0.1” \
-e “GRAPHITE_PROTOCOL=pickled” \
–rm=true \
logzio/jmx2graphite

Note: You will notice, all the docker images in this blog are running as host network. You can run that as a bridge network. In that case, you need to provide Docker container IP instead of 127.0.0.1.

As Jolokia requires Graphite host so before running jmx2graphite, Graphite should be up.

You can get more details here.

Graphite:

It is a data source which stores numeric time series data and renders graphs of this data on demand. It is used for the monitoring purpose.

Graphite works on push-based approach i.e. it does not collect the data instead, we need to use a tool to ingest data in Graphite. Here, we are using jmx2graphite for the same purpose.

docker run -d\
–name graphite\
–restart=always\
–network=host \
graphiteapp/graphite-statsd

Here, we are running Graphite docker images as host network so no need to bind any port with the container, but if you want to run it as a bridge network, then you need to bind the ports as below:

docker run -d\
–name graphite\
–restart=always\
-p 80:80\
-p 2003-2004:2003-2004\
-p 2023-2024:2023-2024\
-p 8125:8125/udp\
-p 8126:8126\
graphiteapp/graphite-statsd

Graphite UI runs at port 80 by default so you can check the Graphite UI at http://localhost/

For more details, check out here.

Grafana:

Grafana is an open source platform for time series analytics. We can create beautiful dashboards on the time series data over Graphite time series data. We can set up an alert notification as well.

We will see the creation of a dashboard and set up the alerts in upcoming blogs.