As we know, DC/OS gave us minimal packages of services for running on apache mesos like cassandra. Cassandra services only give us the functionality for storing data , not other things like native tools as `nodetool`, `cqlsh`, `cassandra-stress` and more. Today, we are looking into the way for using `nodetool`, for monitoring DC/OS cassandra services nodes or cluster.
Nodetool is a native command line interface for managing and monitoring cassandra cluster. While our cassandra services are running on DC/OS, we can manage and monitor cluster using nodetool. We need to execute nodetool on DC/OS using cassandra docker image on any of the DC/OS cassandra nodes by performing below steps:
- First, need to ssh any of DC/OS cassandra node: ssh @
- Second, execute the docker command for download cassandra docker image and pass nodetool command for execution as below:
$ sudo docker run -t --net=host cassandra:3.0 nodetool tablestats -H .
Note: Sometime nodetool commands may not work if Cassandra version and Nodetool version is incompatible e.g: Cassandra Version 3.0 and Nodetool version 3.10, nodetool will not be able to run all commands.
Below, we are looking into some of the important nodetool commands for monitoring perspective of cassandra cluster. But still, nodetool have lots of commands, which we are not discussing now. For reference please click on this link.
1. CASSANDRA CLUSTER STATUS
For looking into cassandra cluster health or status using node tool, we need to execute below command:
sudo docker run -t --net=host cassandra:3.0 nodetool -p 7199 status
|—||Address||Load||Tokens||Owns (effective)||Host ID||Rack|
- U indicate whether the node is UP.
- N indicates whether the node is NORMAL.
- DC/OS cassandra node IP address.
- Load: the amount of filesystem data under the cassandra data directory and updated every 90 seconds.
- Tokens: number of tokens set for the node.
2. MONITOR CASSANDRA CLUSTER SINGLE NODE INFO
If we need to require cluster node stats including:
- Heap Memory
- Exceptions Count
- Key Cache Hit Rate
and more, for complete we need to execute below command. But this command gave us the stats of current node in which we are making ssh.
sudo docker run -t --net=host cassandra:3.0 nodetool info
|Native Transport active||true|
|Heap Memory (MB)||1594.28 / 4086.00|
|Off-Heap Memory (MB)||3.42|
|Key Cache||entries 158173, size 56.76 MB, capacity 100 MB, 9092924 hits, 9352093 requests, 0.972 recent hit rate, 14400 save period in seconds|
|Row Cache||entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds|
|Counter Cache||entries 0, size 0 bytes, capacity 50 MB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds|
|Token||(invoke with -T/–tokens to see all 256 tokens)|
3. MONITOR CLUSTER STATISTICS OF KEYSPACES AND TABLES
If required to monitor statistics of all keyspaces and tables like:
- Read Latency
- Write Latency
- Total Disk Used
and more. For complete we need to execute below command.
sudo docker run -t --net=host cassandra:3.0 nodetool tablestats // Show stats for all keyspaces and tables sudo docker run -t --net=host cassandra:3.0 nodetool tablestats . … . // show stats for specific keyspace and tables
Keyspace: Read Count: 75738 Read Latency: 1.0500195410494073 ms. Write Count: 34312 Write Latency: 0.12817600256470038 ms. Pending Flushes: 0 Table: SSTable count: 6 Space used (live): 3390578917 Space used (total): 3390578917 Space used by snapshots (total): 0 Off heap memory used (total): 423687 SSTable Compression Ratio: 1.003568418469647 Number of keys (estimate): 3967 Memtable cell count: 94 Memtable data size: 11470945 Memtable off heap memory used: 0 Memtable switch count: 75 Local read count: 75738 Local read latency: 1.032 ms Local write count: 34312 Local write latency: 6.078 ms Pending flushes: 0 and more...
4. MONITOR CLUSTER DISK SPACE RECLAIM
If we require checking, how many garbage value we have and need to monitor how much disk is available for re-claim, we need to execute below command.
sudo docker run -t --net=host cassandra:3.0 nodetool gcstats
|Interval (ms)||Max GC Elapsed (ms)||Total GC Elapsed (ms)||Stdev GC Elapsed (ms)||GC Reclaimed (MB)||Collections||Direct Memory Bytes|
5. MONITOR CLUSTER THREADS AND PENDING PROCESS
Cassandra is based on a Staged Event Driven Architecture (SEDA). Different tasks are separated into stages that are connected by a messaging service. Stages have a queue and thread pool. Some stages skip the messaging service and queue tasks immediately on a different stage when it exists on the same node. The queues can back up if executing at the next stage is too busy and cause performance bottlenecks.
So, if we require monitoring cluster threads, active task, pending task, completed blocked and more. It’s updated when SSTables change through compaction or flushing. we need to execute below command:
sudo docker run -t --net=host cassandra:3.0 nodetool tpstats
6. MONITOR CLUSTER THREADS AND PENDING PROCESS
If required statistics about the table like:
- Read/Write Latency
- Partition Size
- Column Count
- Number of SSTable
and more… This command is kind of tablestats command, but its output gave us histogram matrices for stats. Below is the command:
sudo docker run -t --net=host cassandra:3.0 nodetool tablehistograms
|Percentile||SSTables||Write Latency (micros)||Read Latency (micros)||Partition Size||Cell Count|
As we mention above there are lots of commands are available for monitoring and manage cluster by using nodetool. So, as per our requirements, we will use nodetool for manage or monitor.