Hi, is there a way to keep keys after the multilin...
# fluent-bit
j
Hi, is there a way to keep keys after the multiline filter? I scrape Kubernetes logs with multiline app logs and when there is a multiline match, the filename key "disappears". Config:
Copy code
[INPUT]
  Name tail
  Path /var/log/containers/*.log
  multiline.parser docker, cri
  Tag kube.*
  Mem_Buf_Limit 5MB
  Skip_Long_Lines On
  Path_Key filename
  DB /run/fluent-bit/containers.db

[FILTER]
  Name kubernetes
  Match kube.*
  Owner_References On
  Cache_Use_Docker_Id On
  Use_Kubelet Off
  Buffer_Size 1024k

[FILTER]
  Name multiline
  Match kube.*
  Multiline.Key_Content log
  Multiline.Parser go,python,java
p
I think there is an option but you could also just copy it to another key beforehand - also processors might work better
I don't think your multiline config touches the actual filename key though
j
Hmm, I see that for log lines merged by the multiline step the filename is lost. For all other lines the filename is kept
p
Ah right, maybe because it constructs a new record out of multiple existing ones. You could potentially add it back in afterwards from the tag using a LUA filter or similar.
j
This seems to occur with a specific pod (other pods with e.g. go multiline matches work). Additionally I suspect that the match timeout is hit, as the multiline output is incomplete.