hey does someone manage to do correctly the multil...
# fluent-bit
h
hey does someone manage to do correctly the multiline parsing with the kubernetes console logs? I tried to follow these documentation pages https://docs.fluentbit.io/manual/3.2/pipeline/inputs/tail https://docs.fluentbit.io/manual/3.2/pipeline/filters/kubernetes#workflow-of-tail--kubernetes-filter https://docs.fluentbit.io/manual/3.2/administration/configuring-fluent-bit/multiline-parsing But did not manage to get correctly the multiline logs 😞
configuration
Copy code
parsers:
      - format: regex
        name: kube-log-tag-parser
        regex: (?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>[^_]+)_(?<container_id>[a-z0-9]+)$
        skip_empty_values: true
        time_format: '%Y-%m-%dT%H:%M:%S.%L'
        time_keep: true
        time_key: time
        time_strict: true
    pipeline:
      inputs:
        - Tag: kube.var.logs.<pod_name>_<namespace_name>_<container_name>_<container_id>
          Tag_Regex: .+/(?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]+)\.log$
          buffer_chunk_size: 128KB
          buffer_max_size: 128KB
          multiline.parser: docker, cri, java
          name: tail
          path: /var/log/containers/health-seo-0_seo_cluster-controller-a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967.log
          processors:
            logs:
              - Buffer_Size: 2M
                Cache_Use_Docker_Id: "On"
                Kube_Meta_Cache_TTL: "30"
                Kube_Tag_Prefix: kube.var.logs.
                Merge_Log: "On"
                Namespace_Annotations: "On"
                Namespace_Labels: "On"
                Regex_Parser: kube-log-tag-parser
                name: kubernetes
                Merge_Log_Key: "log_merged"
                Keep_Log: "Off"
      filters: []
      outputs:
        - name: stdout
          match: '*'
sample of the output with 1 text line per record instead of 1 record with multiple lines
Copy code
[785] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284084088-04:00 stdout F     at java.base@17.0.5/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218)
"}]
[786] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284086331-04:00 stdout F     at java.base@17.0.5/java.nio.file.Files.newByteChannel(Files.java:380)
"}]
[787] kube.var.logs.health-seo-0_seo_cluster-controller_a043c85e36791270b7e4edbc6822d4a9a103b7fc500f6300226fb86a3048e967: [[1755084391.291220757, {}], {"log"=>"2025-08-13T07:26:31.284088054-04:00 stdout F     at java.base@17.0.5/java.nio.file.Files.newByteChannel(Files.java:432)
"}]
p
Yup, had it working fine
You need to use the cri multiline parser as that's the actual log file format
h
I used that multiline.parser: docker, cri, java
p
Ah I see what you want, to then combine with a multiline filter afterwards
No, that only applies the first parser that matches
You need to parse the kubelet log format then run the multiline filter with java or whatever as a processor after that I think is what you want
Usually I just make a local container test case with a sample log and iterate with that to work on it easily
h
Oh so if I am using the config from the example
Copy code
[INPUT]
    Name    tail
    Tag     kube.*
    Path    /var/log/containers/*.log
    multiline.parser              docker, cri
I should add a parser filter afterwards?
@Pat do you know if there is a possibility to tell to FluentBit not to parse the
log
field as json ? to keep it as text (even it is in json format)?
I added the multiline filter
Copy code
inputs:
        - multiline.parser: cri, docker
          name: tail
          path: /var/log/containers/bar-query-*.log
          ...
          processors:
            logs:
              - ...
                Merge_Log: "On"
                Namespace_Annotations: "On"
                Namespace_Labels: "On"
                name: kubernetes
              - name: multiline
                multiline.key_content: log
                multiline.parser: java, go, python
but would like to keep the
log
field as text for json lines
p
The merge log option in the K8S filter controls parsing
🙏 1
Check the docs for that filter
👍 1
h
thanks @Pat.. it works now for the parsing 🙂 but the bad new is that we deployed the new config on one of the kube cluster (test cluster) and can see that the filter multiline drops records 😞 how can we know what is it dropping and why? there are no logs in the fluentbit pod
p
What do you mean by drops records?
h
according to this metric
sum(rate(fluentbit_filter_drop_records_total{job="$job",namespace="$namespace", pod=~"$pod"}[$__rate_interval])) by (pod, name)
image.png
the multiline filter metrics show that there are drop_records
p
are you seeing missing output?
h
Copy code
- Skip_Empty_Lines: "On"
          Tag: kube.var.logs.<pod_name>_<namespace_name>_<container_name>_<container_id>
          Tag_Regex: .+/(?<pod_name>[^_]+)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]+)\.log$
          buffer_chunk_size: 128KB
          buffer_max_size: 128KB
          multiline.parser: cri, docker
          name: tail
          path: /var/log/containers/*.log
          processors:
            logs:
              - buffer: "On"
                multiline.key_content: log
                multiline.parser: java, go, python
                name: multiline
              - action: insert
                key: kubernetes_cluster
                name: content_modifier
                value: ${KUBENERTES_CLUSTER}
              - action: insert
                key: kubernetes_host
                name: content_modifier
                value: ${KUBERNETES_NODE_NAME}
              - Buffer_Size: 2M
                Cache_Use_Docker_Id: "On"
                Kube_Meta_Cache_TTL: "30"
                Kube_Tag_Prefix: kube.var.logs.
                Merge_Log: "Off"
                Namespace_Annotations: "On"
                Namespace_Labels: "On"
                Regex_Parser: kube-log-tag-parser
                name: kubernetes
p
I'm not sure exactly what that metric records for multiline filter - do you know @lecaros?
h
humm.. there are millions of logs (from multiple apps/pods) and I am not the "final" user so I cannot tell what is missing yet 😞 but the metrics show the drop so I would like to know what's dropping before the users 🙂
l
oh, yeah. Every single record that is part of a ml record is marked as dropped
p
yeah so it's dropped == combined right?
l
correct
p
I figured it was something like that, really should be a new metric for it lol
h
oh .. it's misleading 😰
so it should be ok for the ml filter if I understand well
p
depends on your terminology really I guess but ultimately yeah from your perspective
l
say you have 2 records: one is 4 lines multiline, the other is a single line record. You'll have 4 dropped, and 2 delivered (1 ml, 1 single line)
👍 1
p
the records are dropped because they become part of another record is the understanding
➕ 1
@Hung maybe send a docs update specifically for the multiline filter page to cover this
h
yes .. I will try to send something soon
thanks @lecaros for the explanation
metal 1
p
@lecaros is doing some talks on this I believe right?
l
yeah, I forgot to mention the drop metric thingy 😕
p
Follow up talk then? Catch up at the start of the next? 😆
l
haha, it's possible 🙂
n
Does it work if the kubernetes logs are printed in json format instead?
p
Format shouldn't matter, that's down to the parser. You may need to configure some things differently but fundamentally the process is the same
Also, which format is configured to be JSON - the format from the app and/or the kubelet log format? The kubelet logs need to be parsed first regardless in whatever format you have configured them to use then multiline added.