How to add add Kubernetes metadata labels to Loki using FluentBit

So, after playing around with ElasticSearch for a while on my 3 node cluster for a few months I thought it was time for a lighter logging solution. Elastic was using 1GB per replica and most of the CPU, for a small cluster like mine this was simply too high – I needed Loki.

Loki, unlike Elastic, can be described as grep on steroids. You still have the normal inverted index à la Elastic but only the labels are indexed, the actual log message isn’t. Labels can be things like node, pod_name etc, this helps you to narrow down your search to avoid parsing hundreds of GBs of logs.

My cluster uses fluent-bit to read, filter and ship the logs to Loki. In fact, I use the CRDs provided by the Fluent Operator. The problem I was having was that the Kubernetes labels, such as namespace_name weren’t appearing as labels. Only the manually defined job=fluentbit label was being sent, this made it quite challenging to debug individual pods. I turned on autoKubernetesLabel and set-up my Kubernetes filter, like so:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: k8s-app-loki
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  matchRegex: (?:kube|service)\.(.*)
  loki:
    host: loki-gateway.loki.svc.cluster.local
    port: 80
    labels:
      - job=fluentbit
    autoKubernetesLabels: "on"

But, nothing?

My filter was correct as well:

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterFilter
metadata:
  name: kubernetes
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  match: kube.*
  filters:
  - kubernetes:
      kubeURL: https://kubernetes.default.svc:443
      kubeCAFile: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      kubeTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      labels: false
      annotations: false
      mergeLog: true
      mergeLogKey: "log_processed"
      keepLog: false
      k8sLoggingParser: true
      k8sLoggingExclude: false

Well, it seems the field autoKubernetesLabel doesn’t “automatically” add labels. You still need to

• have the Kubernetes ClusterFilter defined and
• also add the labels you want with say: $kubernetes[pod_name] in the Loki output section.

So, to finish the post off I used something like this (feel free to add additional labels):

apiVersion: fluentbit.fluent.io/v1alpha2
kind: ClusterOutput
metadata:
  name: k8s-app-loki
  labels:
    fluentbit.fluent.io/enabled: "true"
    fluentbit.fluent.io/mode: "k8s"
spec:
  matchRegex: (?:kube|service)\.(.*)
  loki:
    host: loki-gateway.loki.svc.cluster.local
    port: 80
    labels:
      - job=fluentbit
      - $kubernetes['pod_name']
      - $kubernetes['namespace_name']
      - $kubernetes['container_name']
    autoKubernetesLabels: "on"