Hello, I’m having hard time understanding why my b...
# fluent-bit
j
Hello, I’m having hard time understanding why my basic filter grep doesn’t work 😕 Pipeline: INPUT forward (fluentd) -> fluentbit fluentbit config:
Copy code
custom_parsers.conf: |
    [PARSER]
        Name docker_no_time
        Format json
        Time_Keep Off
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
  fluent-bit.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    debug
        Config_Watch On
        HTTP_Server  On
        HTTP_Listen  0.0.0.0
        HTTP_Port    2020
        Parsers_File /opt/bitnami/fluent-bit/conf/parsers.conf
        Parsers_File /opt/bitnami/fluent-bit/conf/custom_parsers.conf

    [INPUT]
        Name              forward
        Listen            0.0.0.0
        Port              24224
        Buffer_Chunk_Size 1M
        Buffer_Max_Size   6M


    [FILTER]
        Name   modify
        Match  *
        Condition Key_exists message
        Add    has_message true

    [OUTPUT]
        Name   stdout
        Match  *
        Format json_lines
fluetnbit logs:
Copy code
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* <https://fluentbit.io>

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 


[2025/09/05 09:30:38] [ info] Configuration:
[2025/09/05 09:30:38] [ info]  flush time     | 1.000000 seconds
[2025/09/05 09:30:38] [ info]  grace          | 5 seconds
[2025/09/05 09:30:38] [ info]  daemon         | 0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  inputs:
[2025/09/05 09:30:38] [ info]      forward
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  filters:
[2025/09/05 09:30:38] [ info]      modify.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  outputs:
[2025/09/05 09:30:38] [ info]      stdout.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info]  collectors:
[2025/09/05 09:30:38] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/09/05 09:30:38] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/09/05 09:30:38] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/09/05 09:30:38] [ info] [simd    ] disabled
[2025/09/05 09:30:38] [ info] [cmetrics] version=1.0.4
[2025/09/05 09:30:38] [ info] [ctraces ] version=0.6.6
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] initializing
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2025/09/05 09:30:38] [debug] [forward:forward.0] created event channels: read=25 write=26
[2025/09/05 09:30:38] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2025/09/05 09:30:38] [debug] [downstream] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [debug] [filter:modify:modify.0] Initialized modify filter with 1 conditions and 1 rules
[2025/09/05 09:30:38] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/09/05 09:30:38] [ info] [output:stdout:stdout.0] worker #0 started
[2025/09/05 09:30:38] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/09/05 09:30:38] [ info] [sp] stream processor started
[2025/09/05 09:30:38] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [task] created task=0x7f002d4296e0 id=0 OK
[2025/09/05 09:30:57] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
{"date":1757064652.918883,"time":"2025-09-05T09:30:52.918883142+00:00","stream":"stdout","log":"HTTP server listening on :8080","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"M2M3OGY1ZDYtY2E2MC00ZjBjLWE3NTEtOThiMDQxYzQ3OWJj"}
{"date":1757064652.919008,"time":"2025-09-05T09:30:52.919007627+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.918899636Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"org.springframework.boot.devtools.env.DevToolsPropertyDefaultsPostProcessor","message":"Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZDcxZjJjMDQtYmUxMC00Y2ZhLThkZTMtY2UxNDNmY2JmMmQ3"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","log":"Finished printing sample Spring Boot log lines.","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZjgyYTEwZDMtMzZmNC00MWQyLThkMGQtY2ZmNGIyZTRiNDU2"}
[2025/09/05 09:30:57] [debug] [out flush] cb_destroy coro_id=0
[2025/09/05 09:30:57] [debug] [task] destroy task=0x7f002d4296e0 (task_id=0)
--- For some reason FILTER says that key “*message*” does not exist but if you check the ouput you can clearly see that key “*message*” is there:
Copy code
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
...
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
I’m definitely missing something crucial. I have checked docs, github issues and also tried three different AI tools but all of them saying that FILTER should work 😄…. I would greatly appreciate if someone could point me in the direction how I could debug it! Thanks.
p
That's not grep but modify 🙂 I think it would be easier to simplify it down to using a dummy input rather than forward so you can iterate locally: https://www.fluent.do/blog/debugging-tips
👀 1
I didn't check it all but it looks like some of your logs do not have a
message
key
however it's super hard to debug it with a random dump so I'd simplify with dummy input to control things a bit more and help figure it out
j
Yes, I have used modify to show that condition is not met (I have replace it with grep result would be the same, but no logs would be visible)
some logs don’t have “message” key, but most do • issue is that filter ignores all records
p
Again, make a simple reproducible example with dummy input rather than relying on forward and it'll probably either show the actual issue easily or make it easier for folks to help you.
✅ 1
I bet it'll be some simple misunderstanding or deviation in the input data you're not expecting - it usually is and simplifying the problem helps me every time realise it's my fault 🙂
✅ 1
We're working on a simple sandbox service to help debug these kinds of issues but it's not ready yet unfortunately: https://fluent.do
✅ 1
j
I think so too! I’m having hard time to view the exact data format from INPUT. I’ll look into local reproducer.
p
just make a dummy with a message field to start with then extend it
I think the default is that in fact...
you'll speed up the whole process being able to run things all locally, I guarantee it - that's why I use containers for it: https://www.fluent.do/blog/debugging-tips
j
Dummy message of course works as expected 🙂. The issues is 100% in data format from forwarder. Do you have any tips how to debug that locally?
p
Get it to dump the message before the filter using the stdout filter (not the stdout output).
✅ 1
Then check the format, I bet the message field is not where you expect it to be
take some sample log files that cause the problem and use locally - you can use tail then to ingest them
j
I have found “solution” • run every single record through custom function that will convert everything to string ◦ otherwise fluentbit FILTER doesn’t see the keys I still don’t know why because FILTER stdout prints valid json with key in exactly same structure but fail on expect check…. ---
Copy code
-- Recursively convert all values in a record to strings
local function stringify(val)
  local t = type(val)
  if t == "table" then
    local out = {}
    for k,v in pairs(val) do
      -- also convert keys to string just to be safe
      out[tostring(k)] = stringify(v)
    end
    return out
  else
    return tostring(val)
  end
end

function stringify_record(tag, ts, record)
  local converted = stringify(record)

  -- Debug output (optional)
  print(string.format("[normalize] tag=%s ts=%s", tostring(tag), tostring(ts)))

  -- Return the fully stringified record
  return 1, ts, converted
end
p
Ah, yeah if it's not a string then some of the filters will not work with it
There is an approach to convert to strings with existing processors though I believe
So grep may not work unless it's a string but if you're using LUA remember you can always drop the record with that - you don't need to convert it then call another filter. Just return the right value to drop or allow the record: https://docs.fluentbit.io/manual/data-pipeline/filters/lua#return-values That's probably a lot better than converting everything to strings, just check the fields you need and LUA will work with any type.
j
thnaks for the docs!
I still don’t understand why this happens as INPUT is “forward” from fluetnd (thus I would expect that string will be received).
I will try to identify the record with lua.
p
Plus the lack of support for anything other than strings in grep is known, e.g. https://github.com/fluent/fluent-bit/issues/7271#issuecomment-1523070663 However it would be good to properly document it as I think I say in that comment 🙂 @hhatake I think it is still true it has to be a string for grep right?
🙌 1
my guess is you have an integer in the JSON data rather than a string - you should be able to identify it in the stdout output as it won't have quotes
j
@Pat that is definitely possible! I have nested json inside the record, other than that everything is a string (according to FILTER stdout or Lua check)…
p
I would just use a LUA filter/processor to select/reject as you'll have a lot more control
✅ 1
plus you can print debug for failures, etc. 🙂
✅ 1