Morning, I want to create a custom timestamp fiel...
# fluent-bit
t
Morning, I want to create a custom timestamp field in fluent-bit. As it is now, fluent-bit populates the timestamp value which corresponds to the time it read the log entry from the app. As a result, there is constantly a mismatch between the actual app timestamp and the timestamp that is set by fluent-bit. So i defined the custom timestamp like this:
Copy code
[PARSER]
  Name pw_tmstp
  Format regex
  Regex (?pw_time\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}.\d{9})\s+(?log.*)
  Time_Key pw_time
  Time_Format %Y-%m-%d %H:%M:%S.%L
  Time_Keep Off
  Preserve_Key On
  Reserve_Data On
And then applied it to the INPUT plugin like this:
Copy code
[INPUT]
  Name tail
  Path C:\var\log\containers\*.log
  multiline.parser docker, cri
  DB C:\fluent-bit\buffer-storage\fluent-buffer.db
  Tag kube.*
  Mem_Buf_Limit 100MB
  Skip_Long_Lines On
  Refresh_Interval  10
  Parser pw_tmstp
And finally, i instructed fluent-bit to populate "@timestamp" field in Elasticsearch with that custom timestamp value:
Copy code
[OUTPUT]
  Name es
  Match *
  Host ${FLUENT_ELASTICSEARCH_HOST}
  Port 9200
  HTTP_User ${FLUENT_ELASTICSEARCH_USER}
  HTTP_Passwd ${FLUENT_ELASTICSEARCH_PASSWORD}
  Logstash_Format On
  Logstash_Prefix ${FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX}
  Time_Key @timestamp
  Trace_Error On
  Trace_Output On
  tls On
  tls.verify Off
  Suppress_Type_Name On
  Pipeline ${FLUENT_ELASTICSEARCH_INGEST_PIPELINE}
  Retry_Limit False
  Buffer_Size 400MB
  Replace_Dots On
  storage.total_limit_size 500MB
However, when i check the ES index, i still see that the "*@timestamp*" field is being populated with fluent-bit's timestamp, instead of the custom timestamp value that i extracted with "pw_tmstp" parser. What is wrong with my config then?
btw, a sample log entry looks like this: 2025-06-04T084422.0769588Z stdout F 2025-06-02 231105.272000000 +0000 log.myapp.logs: {"message":"2025-06-02 231105,272 WARN [0x000008c8] myapp.security - New master file access token key generated. Please update any other servers in this cluster with the shared key value."} My fluent-bit config targets the 2nd timestamp (in this case 2025-06-02 231105.272000000 )
p
You're missing up config for tail, you have some other multiline parsers defined as well so my guess is that's being used instead
Also, I always encourage folks to not debug via some downstream stack which may do it's own things but instead use the stdout output to see what fluent bit actually is storing for the data first. You can easily iterate locally with some sample data using a container as well to simplify things nicely.
I think what you want to do here is first parse the kubelet log format using the normal multiline parsers and then you want a second stage
parser
filter after the tail input that then targets the specific log message format you're after. You could do it all in one but you potentially miss some of the edge cases for the kubelet logs (e.g. when lines are split). Keeping the normal tail input using the kubelet format also means you'll handle all other logs as well. The parser filter will only apply if it matches and is ignored if it does not. There is also a stdout filter you can use to see what the data looks like in between stages, e.g. after tail before the next filter if you need to debug anything
t
Thank you! I commented out the "multiline.parser docker, cri" line and now it works perfectly. Initially, there was also some built-in parser called "docker" enabled in INPUT plugin:
Copy code
Parser docker
So i removed it when testing in my local environment and it started to work. This "docker" parser appears to be some kind of built-in thing because i could not find it explicitly defined anywhere in the config files. My understanding is that "docker" parser expects input in JSON format, but if it detects input as plain unstructured text (as my app emits it) then fluent bit ignores whatever custom Time_Key is set up and puts its own timestamp value (i.e. the time fluent-bit happened to read the input entry). So once i got it working with "docker" parser removed, i did not bother with that "multiline.parser docker, cri" line, somehow thinking that it wouldnt cause fluent-bit ignore my custom Time_Key. But now i know it does mess up things. So lesson learned - disable all built-in "docker" things if your app emits plain text logs, haha