Fluentbit is used for collecting the logs from server, linux machines, Kubernetes nodes etc. You might want to collect these Istio logs and parse them & route them to a separate index to read in kibana & elastic search.
Lets find out how we can achieve this functionality
Prerequisites
- Kubernetes cluster
- Helm Installed in your machine.
- Istio setup & proxy injection enabled
- Some pods running to generate logs
- Optionally Elasticsearch & Kibana to visualize logs index
Fluent-bit Data pipeline configuration File
The files contain sections to define where to pick data, how to pick, what to pick & where to route data. Lets see each section.
- SERVICE
- INPUT
- FILTER
- OUTPUT
- PARSER
[SERVICE] Section
The global properties are defined in the SERVICE section of the configuration file.
[INPUT] Section
The INPUT section defines the source & the input plugin. This section decides how, where & what data is picked from sources. To read about input plugins follow this page.
[FILTER] Section
Set filters on incoming records or data. It supports many Filter plugins to filter & transform collected records. Read more in detail about the filter plugin.
[OUTPUT] Section
This section defines the route & destination of the matched records. It uses many output plugins for example in this case we are using elasticsearch. Read more about Output plugins.
[PARSER] Section
Parsers are basically used to structurize the collected logs. Logs collected may or may not be in the right structure that we want. It makes log processing easier. Read more about Parser plugins. Regex & JSON are mostly used.
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
[INPUT]
Name tail
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Tag kube.<container_name>.<namespace_name>
Path /var/log/containers/*.log
multiline.parser docker, cri
Mem_Buf_Limit 20MB
Skip_Long_Lines On
[FILTER]
Name parser
Match kube.istio-proxy.*
Key_Name log
Reserve_Data On
Parser envoy
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master
Logstash_Format On
Logstash_Prefix istio
Retry_Limit False
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name envoy
Format regex
Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<code>[^ ]*) (?<response_flags>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)"
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Key start_time
[PARSER]
Name istio-envoy-proxy
Format regex
Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<response_code>[^ ]*) (?<response_flags>[^ ]*) (?<response_code_details>[^ ]*) (?<connection_termination_details>[^ ]*) (?<upstream_transport_failure_reason>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<x_request_id>[^\"]*)" (?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)" (?<upstream_cluster>[^ ]*) (?<upstream_local_address>[^ ]*) (?<downstream_local_address>[^ ]*) (?<downstream_remote_address>[^ ]*) (?<requested_server_name>[^ ]*) (?<route_name>[^ ]*)
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Key start_time
Helm Custom Values file for Istio logs index
Create a custom-values.yaml file for helm values. This file will be used in fluent-bit deployment. Carefully place the pipeline configuration under .Values. config: dictionary as shown below.
custom-values.yaml
annotations:
sidecar.istio.io/inject: "false"
podAnnotations:
sidecar.istio.io/inject: "false"
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Tag_Regex (?<pod_name>[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-
Tag kube.<container_name>.<namespace_name>
Path /var/log/containers/*.log
multiline.parser docker, cri
Mem_Buf_Limit 20MB
Skip_Long_Lines On
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name parser
Match kube.istio-proxy.*
Key_Name log
Reserve_Data On
Parser envoy
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host elasticsearch-master
Logstash_Format On
Logstash_Prefix istio
Retry_Limit False
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name envoy
Format regex
Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<code>[^ ]*) (?<response_flags>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<request_id>[^\"]*)" "(?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)"
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Key start_time
[PARSER]
Name istio-envoy-proxy
Format regex
Regex ^\[(?<start_time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)? (?<protocol>\S+)" (?<response_code>[^ ]*) (?<response_flags>[^ ]*) (?<response_code_details>[^ ]*) (?<connection_termination_details>[^ ]*) (?<upstream_transport_failure_reason>[^ ]*) (?<bytes_received>[^ ]*) (?<bytes_sent>[^ ]*) (?<duration>[^ ]*) (?<x_envoy_upstream_service_time>[^ ]*) "(?<x_forwarded_for>[^ ]*)" "(?<user_agent>[^\"]*)" "(?<x_request_id>[^\"]*)" (?<authority>[^ ]*)" "(?<upstream_host>[^ ]*)" (?<upstream_cluster>[^ ]*) (?<upstream_local_address>[^ ]*) (?<downstream_local_address>[^ ]*) (?<downstream_remote_address>[^ ]*) (?<requested_server_name>[^ ]*) (?<route_name>[^ ]*)
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
Time_Key start_time
[PARSER]
Name k8s-nginx-ingress
Format regex
Regex ^(?<host>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (\[(?<proxy_alternative_upstream_name>[^ ]*)\] )?(?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<reg_id>[^ ]*).*$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name kube-custom
Format regex
Regex (?<tag>[^.]+)?\.?(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$
Clone this fluent-bit repo with helm chart
Clone the github repo which already have a configured fluentbit data pipeline configurations. Run the below command in your terminal.
git clone https://github.com/knoldus/istio-fluent-bit-logs.git
cd istio-fluent-bit-logs
Deploy Fluent-bit with Helm
Now, you just need to run a helm command to deploy the fluentbit on Kubernetes in the logging namespace.
helm upgrade --install fluent-bit ./ -f ./custom-values.yaml --namespace logging --create-namespace
This will take our custom data pipeline configuration & enable Istio logs collection in a separate index.
Conclusion – Istio logs
These istio logs can be used in troubleshooting the service mesh. These logs also be used in auditing the access logs & much more. Elasticsearch & Kibana is required to visualize the logs entries coming from fluentbit.