Simple Things You Can Learn From Cassandra Nodetool (Monitor/Manage) For DC/OS


As we know, DC/OS gave us minimal packages of services for running on apache mesos like cassandra. Cassandra services only give us the functionality for storing data , not other things like native tools as `nodetool`, `cqlsh`, `cassandra-stress` and more. Today, we are looking into the way for using `nodetool`, for monitoring DC/OS cassandra services nodes or cluster.

Brief:

Nodetool is a native command line interface for managing and monitoring cassandra cluster. While our cassandra services are running on DC/OS,  we can manage and monitor cluster using nodetool.  We need to execute nodetool on DC/OS using cassandra docker image on any of the DC/OS cassandra nodes by performing below steps:

  1. First, need to ssh any of  DC/OS cassandra node: ssh @
  2. Second, execute the docker command for download cassandra docker image and pass nodetool command for execution as below:
$ sudo docker run -t --net=host cassandra:3.0 nodetool tablestats -H .

Note: Sometime nodetool commands may not work if Cassandra version and Nodetool version is incompatible e.g: Cassandra Version 3.0 and Nodetool version 3.10, nodetool will not be able to run all commands.

Below, we are looking into some of the important nodetool commands for monitoring perspective of cassandra cluster. But still, nodetool have lots of commands, which we are not discussing now. For reference please click on this link.

1. CASSANDRA CLUSTER STATUS

For looking into cassandra cluster health or status using node tool, we need to execute below command:

sudo docker run -t --net=host cassandra:3.0 nodetool -p 7199 status

OUTPUT:

Address Load Tokens Owns (effective) Host ID Rack
UN 23.26 GB 256 21.7% rac1
UN 20.48 GB 256 19.7% rac1
UN 21.62 GB 256 19.7% rac1
  • U indicate whether the node is UP.
  • N indicates whether the node is NORMAL.
  • DC/OS cassandra node IP address.
  • Load: the amount of filesystem data under the cassandra data directory and updated every 90 seconds.
  • Tokens:  number of tokens set for the node.

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsStatus.html

2. MONITOR CASSANDRA CLUSTER SINGLE NODE INFO

If we need to require cluster node stats including:

  •  Load
  • Uptime
  • Heap Memory
  • Exceptions Count
  • Key Cache Hit Rate

and more, for complete we need to execute below command. But this command gave us the stats of current node in which we are making ssh.

sudo docker run -t --net=host cassandra:3.0 nodetool info

OUTPUT

ID
Gossip active true
Thrift active false
Native Transport active true
Load 22.57 GB
Generation No 1506714058
Uptime (seconds) 1089322
Heap Memory (MB) 1594.28 / 4086.00
Off-Heap Memory (MB) 3.42
Data Center dc1
Rack rac1
Exceptions 0
Key Cache entries 158173, size 56.76 MB, capacity 100 MB, 9092924 hits, 9352093 requests, 0.972 recent hit rate, 14400 save period in seconds
Row Cache entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token (invoke with -T/–tokens to see all 256 tokens)

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsInfo.html

3. MONITOR CLUSTER STATISTICS OF KEYSPACES AND TABLES

If required to monitor statistics of all keyspaces and tables like:

  • Read Latency
  • Write Latency
  • Total Disk Used

and more. For complete we need to execute below command.

sudo docker run -t --net=host cassandra:3.0 nodetool tablestats // Show stats for all keyspaces and tables

sudo docker run -t --net=host cassandra:3.0 nodetool tablestats . … . // show stats for specific keyspace and tables

OUTPUT

Keyspace: 
Read Count: 75738
Read Latency: 1.0500195410494073 ms.
Write Count: 34312
Write Latency: 0.12817600256470038 ms.
Pending Flushes: 0
Table: 
SSTable count: 6
Space used (live): 3390578917
Space used (total): 3390578917
Space used by snapshots (total): 0
Off heap memory used (total): 423687
SSTable Compression Ratio: 1.003568418469647
Number of keys (estimate): 3967
Memtable cell count: 94
Memtable data size: 11470945
Memtable off heap memory used: 0
Memtable switch count: 75
Local read count: 75738
Local read latency: 1.032 ms
Local write count: 34312
Local write latency: 6.078 ms
Pending flushes: 0
and more...

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablestats.html

4. MONITOR CLUSTER DISK SPACE RECLAIM

If we require checking, how many garbage value we have and need to monitor how much disk is available for re-claim, we need to execute below command.

sudo docker run -t --net=host cassandra:3.0 nodetool gcstats

OUTPUT:

Interval (ms) Max GC Elapsed (ms) Total GC Elapsed (ms) Stdev GC Elapsed (ms) GC Reclaimed (MB) Collections Direct Memory Bytes
1091695368 530 1353160 12 4603764734720 56632 -1

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsGcstats.html

5. MONITOR CLUSTER THREADS AND PENDING PROCESS

Cassandra is based on a Staged Event Driven Architecture (SEDA). Different tasks are separated into stages that are connected by a messaging service. Stages have a queue and thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The queues can back up if executing at the next stage is too busy and cause performance bottlenecks.

So, if we require monitoring cluster threads, active task, pending task, completed blocked and more. It’s updated when SSTables change through compaction or flushing. we need to execute below command:

sudo docker run -t --net=host cassandra:3.0 nodetool tpstats

6. MONITOR CLUSTER THREADS AND PENDING PROCESS

If required statistics about the table like:

  • Read/Write Latency
  • Partition Size
  • Column Count
  • Number of SSTable

and more… This command is kind of tablestats command, but its output gave us histogram matrices for stats. Below is the command: 

EXAMPLE:

sudo docker run -t --net=host cassandra:3.0 nodetool tablehistograms  

OUTPUT

Percentile SSTables Write Latency (micros) Read Latency (micros) Partition Size Cell Count
50% 0.00 182.79 0.00 1331 4
75% 1.00 182.79 0.00 1331 4
95% 1.00 219.34 0.00 1597 4
98% 1.00 379.02 0.00 1597 4
99% 1.00 379.02 0.00 1597 4
Min 0.00 51.01 0.00 447 4
Max 1.00 1629.72 0.00 1916 4

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablehisto.html

As we mention above there are lots of commands are available for monitoring and manage cluster by using nodetool. So, as per our requirements, we will use nodetool for manage or monitor.

References:

https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsNodetool.html


 

knoldus-advt-sticker

Advertisements
This entry was posted in Best Practices, big data, Cassandra, cluster, NoSql and tagged , , , , , , , , , , , , , . Bookmark the permalink.

2 Responses to Simple Things You Can Learn From Cassandra Nodetool (Monitor/Manage) For DC/OS

  1. Hari Sekhon says:

    Thanks for the post!

    You might be interested in my nagios plugins which check and integrate most of these nodetool outputs in to any nagios compatible monitoring system for monitoring and alerting purposes.

    You can find the Cassandra nagios monitoring plugins here:

    https://github.com/harisekhon/nagios-plugins

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s