Monitoring application using Prometheus and grafana

Hello readers, In this blog I will discuss about application and system monitoring using Prometheus and Grafana. we need something to monitor! Part of that is going to be our Ubuntu host and the application will be a simple web page host program called sample-App that uses the Express web framework to do most of the hard work for us. we are going to download the application from the repository and build a docker image of the application and run it.

Why Prometheus and Grafana ?

Promethues

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database built using a HTTP pull model, with flexible queries and real-time alerting.

Garfana

Grafana is a multi-platform open source analytics and interactive visualization software. It provides charts, graphs, and alerts for the web when connected to supported data sources. 

Sample Application

Lets first download our sample application you can choose your own application to monitor. My sample application is scala based project expose an endpoint at http://localhost:8000/name/devops

$ git clone https://github.com/MukeshAtGit/devops.git

$ cd devops/

~/devops$ ls

Now we have to update our project with metrics fetcher so that it exposes an endpoint for Prometheus which pulls the metrics for the endpoint.
Update project/plugin.sbt and project/Dependencies.scala file with given below content for collecting data from forethought such that Prometheus will able to pull data from the application :
$ vim project/Dependencies.scala

// versions
lazy val kamonPrometheusVersion: String = "2.1.0"
lazy val kamonBundleVersion: String ="2.0.0"

// libraries
  lazy val kamonBundle: ModuleID ="io.kamon" %% "kamon-bundle" % kamonBundleVersion
  lazy val kamonPrometheus: ModuleID = "io.kamon" %% "kamon-prometheus" % kamonPrometheusVersion

  // project
lazy val devopsDependencies: Seq[ModuleID] =
    Seq(akkaHttp, akkaHttpTestKit, akkaActor, akkaStream, akkaActorTestKit, akkaStreamTestKit, scalaTest, kamonBundle, kamonPrometheus)
}

$ vim project/plugin.sbt

// add plugin
addSbtPlugin("io.kamon" % "sbt-kanela-runner" % "2.0.6")

Let Configure our application for getting some specific metrices:

$ vi project/application.conf

kamon.instrumentation.akka.filters {
  actors.track {
    includes = [ "DevOps/user/*" ]
    excludes = [ "Controller/user/*" ]
  }
}

~/devops$ docker build -t sample-app .

Now as we have downloaded our application let just run it.

$ docker run -d -p 8000:8000 -p 9095:9095 --name sample-app sample-app
aece13da3d949d6a4c0e7e72346bbc42bb403d7fa3acb985a4e9f788efd598a3
$ docker  ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                                            NAMES
9f788efd598a3        sample-app   "/bin/sh -c 'java -j…"   About an hour ago   Up About an hour    0.0.0.0:8000->8000/tcp, 0.0.0.0:9095->9095/tcp   sample-app

Go to http://localhost:8000/name/devops and http://localhost:9095. Verify that you are getting a response from application or not.

It look like our application is working fine lets add monitoring tools.

Adding Prometheus

Now let us get started with adding Prometheus to our system for collection of matrices so that we can monitor the application.

$  sudo useradd --no-create-home --shell /bin/false prometheus

[sudo] password for Desktop: ********

$  sudo mkdir /etc/prometheus

$  sudo mkdir /var/lib/prometheus

$  sudo chown prometheus:prometheus /var/lib/prometheus

$  cd /tmp/

$ wget https://github.com/prometheus/prometheus/releases/download/v2.7.1/prometheus-2.7.1.linux-amd64.tar.gz

Now just unzip the prometheus application :

$ tar -xvf prometheus-2.7.1.linux-amd64.tar.gz

Installing and configuring the prometheus application such that it can collect metrics the application:

$  cd prometheus-2.7.1.linux-amd64

$  sudo mv console* /etc/prometheus

$  sudo mv prometheus.yml /etc/prometheus

$  sudo chown -R prometheus:prometheus /etc/prometheus

$  sudo mv prometheus /usr/local/bin/

$  sudo mv promtool /usr/local/bin/

$  sudo chown prometheus:prometheus /usr/local/bin/prometheus

$  sudo chown prometheus:prometheus /usr/local/bin/promtool

Add daemon file for Prometheus :
$ sudo vim /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

As our prometheus application is configure now just reload the changes and start the prometheus :

$  sudo systemctl daemon-reload

$ sudo systemctl start prometheus

$  sudo systemctl enable prometheus
Created symlink /etc/systemd/system/multi-user.target.wants/prometheus.service → /etc/systemd/system/prometheus.service.

$ sudo systemctl status prometheus

Update prometheus.yml under static_configs to receive data from the application so that Prometheus can listen to the data of sampleApp :
sudo vim /etc/prometheus/prometheus.yml

   - job_name: 'sampleApp'
     static_configs:
     - targets: ['localhost:9095']

After update just restart the prometheus:

$ sudo systemctl restart prometheus
and navigate to http://localhost:9090/targets you will be seeing like these.

Now go to http://localhost:8080 and check for metrics you will be getting metrics of our application :

Adding Grafana

Add grafana for visualization and alerting, However prometheus can also be used for alerts but grafana has good visualization power and also that makes setting-up alerts easy and helpful.

$ sudo apt-get install libfontconfig

$ wget https://dl.grafana.com/oss/release/grafana_5.4.3_amd64.deb

Go to localhost:3000 to access the grafana default login user & password is admin & admin.

Adding Datasources in grafana navigate to menu->setting->datatsources then select prometheus as data sources.
Add http://localhost:8080 in Url and click on save and test datasources.

Adding Dashboards in grafana is so simple just click on add option(+) amd select add dashboards. Rename the dashboards as Forethought.

Add a panel in grafana in dashboards and select the visualization you want to choose as i have selected histrogram.

Add a query you want to display in graph you will be seeing this kind of graph.

Conclusion

Monitoring is the regular observation and recording of activities taking place in a project or program. Reporting enables the gathered information to be used in making decisions for improving performance. Monitoring is very important in project planning and implementation