Istio logs with fluentbit on kubernetes

Reading Time: 3 minutes

Fluentbit is used for collecting the logs from server, linux machines, Kubernetes nodes etc. You might want to collect these Istio logs and parse them & route them to a separate index to read in kibana & elastic search.

Lets find out how we can achieve this functionality

Prerequisites

Kubernetes cluster
Helm Installed in your machine.
Istio setup & proxy injection enabled
Some pods running to generate logs
Optionally Elasticsearch & Kibana to visualize logs index

Fluent-bit Data pipeline configuration File

The files contain sections to define where to pick data, how to pick, what to pick & where to route data. Lets see each section.

SERVICE
INPUT
FILTER
OUTPUT
PARSER

[SERVICE] Section

The global properties are defined in the SERVICE section of the configuration file.

[INPUT] Section

The INPUT section defines the source & the input plugin. This section decides how, where & what data is picked from sources. To read about input plugins follow this page.

[FILTER] Section

Set filters on incoming records or data. It supports many Filter plugins to filter & transform collected records. Read more in detail about the filter plugin.

[OUTPUT] Section

This section defines the route & destination of the matched records. It uses many output plugins for example in this case we are using elasticsearch. Read more about Output plugins.

[PARSER] Section

Parsers are basically used to structurize the collected logs. Logs collected may or may not be in the right structure that we want. It makes log processing easier. Read more about Parser plugins. Regex & JSON are mostly used.

    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On


    [INPUT]
        Name tail
        Tag_Regex  (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Tag  kube.<container_name>.<namespace_name>
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Mem_Buf_Limit 20MB
        Skip_Long_Lines On

    [FILTER]
        Name                parser
        Match               kube.istio-proxy.*
        Key_Name            log
        Reserve_Data        On
        Parser              envoy

    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-master
        Logstash_Format On
        Logstash_Prefix istio
        Retry_Limit False

    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

    [PARSER]
        Name    envoy
        Format  regex
        Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<code>[^ ]*) (?<response_flags>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)"
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On
        Time_Key start_time

    [PARSER]
        Name    istio-envoy-proxy
        Format  regex
        Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<response_code>[^ ]*) (?<response_flags>[^ ]*) (?<response_code_details>[^ ]*) (?<connection_termination_details>[^ ]*) (?<upstream_transport_failure_reason>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<x_request_id>[^\"]*)" (?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)" (?<upstream_cluster>[^ ]*) (?<upstream_local_address>[^ ]*) (?<downstream_local_address>[^ ]*) (?<downstream_remote_address>[^ ]*) (?<requested_server_name>[^ ]*) (?<route_name>[^  ]*)
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On
        Time_Key start_time

Helm Custom Values file for Istio logs index

Create a custom-values.yaml file for helm values. This file will be used in fluent-bit deployment. Carefully place the pipeline configuration under .Values. config: dictionary as shown below.

custom-values.yaml

annotations:
  sidecar.istio.io/inject: "false"
podAnnotations:
  sidecar.istio.io/inject: "false"

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File parsers.conf
        Parsers_File custom_parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On

  ## https://docs.fluentbit.io/manual/pipeline/inputs
  inputs: |
    [INPUT]
        Name tail
        Tag_Regex  (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
        Tag  kube.<container_name>.<namespace_name>
        Path /var/log/containers/*.log
        multiline.parser docker, cri
        Mem_Buf_Limit 20MB
        Skip_Long_Lines On

  ## https://docs.fluentbit.io/manual/pipeline/filters
  filters: |
    [FILTER]
        Name                parser
        Match               kube.istio-proxy.*
        Key_Name            log
        Reserve_Data        On
        Parser              envoy

  ## https://docs.fluentbit.io/manual/pipeline/outputs
  outputs: |
    [OUTPUT]
        Name es
        Match kube.*
        Host elasticsearch-master
        Logstash_Format On
        Logstash_Prefix istio
        Retry_Limit False

  ## https://docs.fluentbit.io/manual/pipeline/parsers
  customParsers: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

    [PARSER]
        Name    envoy
        Format  regex
        Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<code>[^ ]*) (?<response_flags>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)"
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On
        Time_Key start_time

    [PARSER]
        Name    istio-envoy-proxy
        Format  regex
        Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<response_code>[^ ]*) (?<response_flags>[^ ]*) (?<response_code_details>[^ ]*) (?<connection_termination_details>[^ ]*) (?<upstream_transport_failure_reason>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<x_request_id>[^\"]*)" (?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)" (?<upstream_cluster>[^ ]*) (?<upstream_local_address>[^ ]*) (?<downstream_local_address>[^ ]*) (?<downstream_remote_address>[^ ]*) (?<requested_server_name>[^ ]*) (?<route_name>[^  ]*)
        Time_Format %Y-%m-%dT%H:%M:%S.%L%z
        Time_Keep   On
        Time_Key start_time

    [PARSER]
        Name        k8s-nginx-ingress
        Format      regex
        Regex       ^(?<host>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (\[(?<proxy_alternative_upstream_name>[^ ]*)\] )?(?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<reg_id>[^ ]*).*$
        Time_Key    time
        Time_Format %d/%b/%Y:%H:%M:%S %z

    [PARSER]
        Name    kube-custom
        Format  regex
        Regex   (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$

Clone this fluent-bit repo with helm chart

Clone the github repo which already have a configured fluentbit data pipeline configurations. Run the below command in your terminal.

git clone https://github.com/knoldus/istio-fluent-bit-logs.git
cd istio-fluent-bit-logs

Deploy Fluent-bit with Helm

Now, you just need to run a helm command to deploy the fluentbit on Kubernetes in the logging namespace.

helm upgrade --install fluent-bit ./ -f ./custom-values.yaml --namespace logging --create-namespace

This will take our custom data pipeline configuration & enable Istio logs collection in a separate index.

Conclusion – Istio logs

These istio logs can be used in troubleshooting the service mesh. These logs also be used in auditing the access logs & much more. Elasticsearch & Kibana is required to visualize the logs entries coming from fluentbit.