Stern: A Tool to Read Logs Across Multiple Pods.

Table of contents

Reading Time: 3 minutes

Kubernetes is a scalable container orchestrator. On the one end, it can support complex enterprise systems with thousands of apps and hosts, on the other end, it is quite light-weighted to support IoT appliances. In the absence of centralized logging and aggregation service in Micro Kubernetes clusters, kubectl logs is there for survival. It is good for a pod but if you want to debug your distributed application which spans across multiple pods then kubectl might not be a better fit. Here you can read K8S logs using Stern. In this blog post, I will walk you through an awesome utility called Stern.

Create pods to generate sample log data.

$ kubectl create deployment web  --image=busybox --replicas=2  -- /bin/sh -c 'while :; do  echo "Hello from WEB"; sleep 10; done'

$ kubectl create deployment app  --image=busybox --replicas=2  -- /bin/sh -c 'while :; do  echo "Hello from APP"; sleep 10; done'

Here I have created 2 deployments named web and app. Each of the deployment has 2 replicas.

Read Logs Using Kubectl

$  kubectl logs -f --timestamps app-dcbc59d67-2bs2q
----------------
2021-01-26T10:05:22.343379745Z Hello from APP
2021-01-26T10:05:32.344856289Z Hello from APP
2021-01-26T10:05:42.346489985Z Hello from APP
----------------

kubectl allows reading logs of a particular pod. It is not possible to get logs from all the pods of a deployment. However, you can use label-selector to get logs from multiple pods. Let’s add a label (env=dev) in all the pods and give it a try.

$ kubectl label pod app-dcbc59d67- env=dev
app-dcbc59d67-j7wzw  app-dcbc59d67-krrzp  
$ kubectl label pod app-dcbc59d67-j7wzw env=dev
pod/app-dcbc59d67-j7wzw labeled 
$ kubectl label pod web-6dbfc4dd77-l6n44 env=dev
pod/web-6dbfc4dd77-l6n44 labeled 
$ kubectl label pod web-6dbfc4dd77-vczs5 env=dev
pod/web-6dbfc4dd77-vczs5 labeled

$ kubectl logs --timestamps --selector=env=dev
2021-01-26T11:11:06.822321598Z Hello from WEB
2021-01-26T11:11:16.824553849Z Hello from WEB
2021-01-26T11:11:26.826307112Z Hello from WEB
2021-01-26T11:09:55.182803064Z Hello from APP
2021-01-26T11:10:05.184847241Z Hello from APP
2021-01-26T11:10:15.186545450Z Hello from APP

Limitations with KubeCtl

You can see that kubectl can read logs from multiple pods using label selector but there is a limitation of this solution. Firstly, logs from different pods are mixed, which prohibits you from knowing which log line came from which pod. Secondly, it doesn’t work in tail mode (using –follow (-f)). This is because –follow streams the logs from the API server. You open a connection to the API server per pod, which will open a connection to the corresponding kubelet itself to stream the logs continuously. This does not scale well and translates to a lot of inbound and outbound connections to the API server, therefore, it became a design decision to limit multiple connections. So you can either stream the logs of one pod, or select a bunch of pods at the same time without streaming.

What is Stern?

Stern is a feature-rich open-source tool to read logs produced by pods in Kubernetes. It allows you to tail multiple pods on your cluster and multiple containers on each pod. It enable you to debug and trace distributed applications. It shows logs from the application span across multiple pods in real-time.

Read Logs Using stern

Stern opens a connection per pod with the API server and kubelet to fetch log streaming. Yes, it creates some network load on Kubernetes but it is usually fine with small or medium-size clusters. It is a light weighted tool that doesn’t need any persistence. It uses color-coding to distinguish lines of different pods. It works best with the live logs and of-course it supports –follow (-f) streaming. You can use the –timestamp(-t) flag to print the timestamp.

Get logs by label selector

$ stern -t --selector=env=dev

Get logs by pod query

stern -t --selector=env=dev

In the above use case there are multiple pods but each pod contains only a container.

More Examples

Show logs using pod query: It will match any pod containing the term web and all containers within it.

stern web

If there are multiple containers per pod, Stern can fetch logs of a particular container using the flag –container (-c). The following command will Show logs of a particular container. It will match any pod with contains the term web and only the nginx container within it.

stern web -c nginx

Show logs from 15m ago with timestamp

stern web -t --since 15m

Conclusion

Stern is an awesome utility to get logs from multiple pods and containers. It shows logs from multiple applications in real-time so It can be a useful tool to debug and trace distributed applications in real-time. It is a light weighted tool without any persistence. The limitation with Stern is that when a node goes down, its logs are not available anymore, since they are only in that node. This is where centralized logging plays an important role.