tomas_z
06/07/2025, 2:37 PM[PARSER]
Name pw_tmstp
Format regex
Regex (?pw_time\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{9})\s+(?log.*)
Time_Key pw_time
Time_Format %Y-%m-%d %H:%M:%S.%L
Time_Keep Off
Preserve_Key On
Reserve_Data On
And then applied it to the INPUT plugin like this:
[INPUT]
Name tail
Path C:\var\log\containers\*.log
multiline.parser docker, cri
DB C:\fluent-bit\buffer-storage\fluent-buffer.db
Tag kube.*
Mem_Buf_Limit 100MB
Skip_Long_Lines On
Refresh_Interval 10
Parser pw_tmstp
And finally, i instructed fluent-bit to populate "@timestamp" field in Elasticsearch with that custom timestamp value:
[OUTPUT]
Name es
Match *
Host ${FLUENT_ELASTICSEARCH_HOST}
Port 9200
HTTP_User ${FLUENT_ELASTICSEARCH_USER}
HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
Logstash_Format On
Logstash_Prefix ${FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX}
Time_Key @timestamp
Trace_Error On
Trace_Output On
tls On
tls.verify Off
Suppress_Type_Name On
Pipeline ${FLUENT_ELASTICSEARCH_INGEST_PIPELINE}
Retry_Limit False
Buffer_Size 400MB
Replace_Dots On
storage.total_limit_size 500MB
However, when i check the ES index, i still see that the "*@timestamp*" field is being populated with fluent-bit's timestamp, instead of the custom timestamp value that i extracted with "pw_tmstp" parser.
What is wrong with my config then?tomas_z
06/07/2025, 2:42 PMPat
06/08/2025, 7:42 AMPat
06/08/2025, 7:44 AMPat
06/08/2025, 7:57 AMparser filter after the tail input that then targets the specific log message format you're after.
You could do it all in one but you potentially miss some of the edge cases for the kubelet logs (e.g. when lines are split). Keeping the normal tail input using the kubelet format also means you'll handle all other logs as well. The parser filter will only apply if it matches and is ignored if it does not.
There is also a stdout filter you can use to see what the data looks like in between stages, e.g. after tail before the next filter if you need to debug anythingtomas_z
06/10/2025, 8:27 AMParser docker
So i removed it when testing in my local environment and it started to work. This "docker" parser appears to be some kind of built-in thing because i could not find it explicitly defined anywhere in the config files. My understanding is that "docker" parser expects input in JSON format, but if it detects input as plain unstructured text (as my app emits it) then fluent bit ignores whatever custom Time_Key is set up and puts its own timestamp value (i.e. the time fluent-bit happened to read the input entry).
So once i got it working with "docker" parser removed, i did not bother with that "multiline.parser docker, cri" line, somehow thinking that it wouldnt cause fluent-bit ignore my custom Time_Key. But now i know it does mess up things.
So lesson learned - disable all built-in "docker" things if your app emits plain text logs, haha