Simple Things You Can Learn From Cassandra Nodetool (Monitor/Manage) For DC/OS

Table of contents

Reading Time: 4 minutes

As we know, DC/OS gave us minimal packages of services for running on apache mesos like cassandra. Cassandra services only give us the functionality for storing data , not other things like native tools as `nodetool`, `cqlsh`, `cassandra-stress` and more. Today, we are looking into the way for using `nodetool`, for monitoring DC/OS cassandra services nodes or cluster.

Brief:

Nodetool is a native command line interface for managing and monitoring cassandra cluster. While our cassandra services are running on DC/OS, we can manage and monitor cluster using nodetool. We need to execute nodetool on DC/OS using cassandra docker image on any of the DC/OS cassandra nodes by performing below steps:

First, need to ssh any of DC/OS cassandra node: ssh @
Second, execute the docker command for download cassandra docker image and pass nodetool command for execution as below:

$ sudo docker run -t --net=host cassandra:3.0 nodetool tablestats -H .

Note: Sometime nodetool commands may not work if Cassandra version and Nodetool version is incompatible e.g: Cassandra Version 3.0 and Nodetool version 3.10, nodetool will not be able to run all commands.

Below, we are looking into some of the important nodetool commands for monitoring perspective of cassandra cluster. But still, nodetool have lots of commands, which we are not discussing now. For reference please click on this link.

1. CASSANDRA CLUSTER STATUS

For looking into cassandra cluster health or status using node tool, we need to execute below command:

sudo docker run -t --net=host cassandra:3.0 nodetool -p 7199 status

OUTPUT:

—	Address	Load	Tokens	Owns (effective)	Host ID	Rack
UN		23.26 GB	256	21.7%		rac1
UN		20.48 GB	256	19.7%		rac1
UN		21.62 GB	256	19.7%		rac1

U indicate whether the node is UP.
N indicates whether the node is NORMAL.
DC/OS cassandra node IP address.
Load: the amount of filesystem data under the cassandra data directory and updated every 90 seconds.
Tokens: number of tokens set for the node.

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsStatus.html

2. MONITOR CASSANDRA CLUSTER SINGLE NODE INFO

If we need to require cluster node stats including:

Load
Uptime
Heap Memory
Exceptions Count
Key Cache Hit Rate

and more, for complete we need to execute below command. But this command gave us the stats of current node in which we are making ssh.

sudo docker run -t --net=host cassandra:3.0 nodetool info

OUTPUT

ID
Gossip active	true
Thrift active	false
Native Transport active	true
Load	22.57 GB
Generation No	1506714058
Uptime (seconds)	1089322
Heap Memory (MB)	1594.28 / 4086.00
Off-Heap Memory (MB)	3.42
Data Center	dc1
Rack	rac1
Exceptions	0
Key Cache	entries 158173, size 56.76 MB, capacity 100 MB, 9092924 hits, 9352093 requests, 0.972 recent hit rate, 14400 save period in seconds
Row Cache	entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache	entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Token	(invoke with -T/–tokens to see all 256 tokens)

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsInfo.html

3. MONITOR CLUSTER STATISTICS OF KEYSPACES AND TABLES

If required to monitor statistics of all keyspaces and tables like:

Read Latency
Write Latency
Total Disk Used

and more. For complete we need to execute below command.

sudo docker run -t --net=host cassandra:3.0 nodetool tablestats // Show stats for all keyspaces and tables

sudo docker run -t --net=host cassandra:3.0 nodetool tablestats . … . // show stats for specific keyspace and tables

OUTPUT

Keyspace: 
Read Count: 75738
Read Latency: 1.0500195410494073 ms.
Write Count: 34312
Write Latency: 0.12817600256470038 ms.
Pending Flushes: 0
Table: 
SSTable count: 6
Space used (live): 3390578917
Space used (total): 3390578917
Space used by snapshots (total): 0
Off heap memory used (total): 423687
SSTable Compression Ratio: 1.003568418469647
Number of keys (estimate): 3967
Memtable cell count: 94
Memtable data size: 11470945
Memtable off heap memory used: 0
Memtable switch count: 75
Local read count: 75738
Local read latency: 1.032 ms
Local write count: 34312
Local write latency: 6.078 ms
Pending flushes: 0
and more...

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablestats.html

4. MONITOR CLUSTER DISK SPACE RECLAIM

If we require checking, how many garbage value we have and need to monitor how much disk is available for re-claim, we need to execute below command.

sudo docker run -t --net=host cassandra:3.0 nodetool gcstats

OUTPUT:

Interval (ms)	Max GC Elapsed (ms)	Total GC Elapsed (ms)	Stdev GC Elapsed (ms)	GC Reclaimed (MB)	Collections	Direct Memory Bytes
1091695368	530	1353160	12	4603764734720	56632	-1

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsGcstats.html

5. MONITOR CLUSTER THREADS AND PENDING PROCESS

Cassandra is based on a Staged Event Driven Architecture (SEDA). Different tasks are separated into stages that are connected by a messaging service. Stages have a queue and thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The queues can back up if executing at the next stage is too busy and cause performance bottlenecks.

So, if we require monitoring cluster threads, active task, pending task, completed blocked and more. It’s updated when SSTables change through compaction or flushing. we need to execute below command:

sudo docker run -t --net=host cassandra:3.0 nodetool tpstats

6. MONITOR CLUSTER THREADS AND PENDING PROCESS

If required statistics about the table like:

Read/Write Latency
Partition Size
Column Count
Number of SSTable

and more… This command is kind of tablestats command, but its output gave us histogram matrices for stats. Below is the command:

EXAMPLE:

sudo docker run -t --net=host cassandra:3.0 nodetool tablehistograms

OUTPUT

Percentile	SSTables	Write Latency (micros)	Read Latency (micros)	Partition Size	Cell Count
50%	0.00	182.79	0.00	1331	4
75%	1.00	182.79	0.00	1331	4
95%	1.00	219.34	0.00	1597	4
98%	1.00	379.02	0.00	1597	4
99%	1.00	379.02	0.00	1597	4
Min	0.00	51.01	0.00	447	4
Max	1.00	1629.72	0.00	1916	4

For more Info: https://docs.datastax.com/en/cassandra/3.0/cassandra/tools/toolsTablehisto.html

As we mention above there are lots of commands are available for monitoring and manage cluster by using nodetool. So, as per our requirements, we will use nodetool for manage or monitor.