jl
09/05/2025, 9:43 AMcustom_parsers.conf: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
Config_Watch On
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Parsers_File /opt/bitnami/fluent-bit/conf/parsers.conf
Parsers_File /opt/bitnami/fluent-bit/conf/custom_parsers.conf
[INPUT]
Name forward
Listen 0.0.0.0
Port 24224
Buffer_Chunk_Size 1M
Buffer_Max_Size 6M
[FILTER]
Name modify
Match *
Condition Key_exists message
Add has_message true
[OUTPUT]
Name stdout
Match *
Format json_lines
fluetnbit logs:
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* <https://fluentbit.io>
______ _ _ ______ _ _ ___ _____
| ___| | | | | ___ (_) | / || _ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
[2025/09/05 09:30:38] [ info] Configuration:
[2025/09/05 09:30:38] [ info] flush time | 1.000000 seconds
[2025/09/05 09:30:38] [ info] grace | 5 seconds
[2025/09/05 09:30:38] [ info] daemon | 0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info] inputs:
[2025/09/05 09:30:38] [ info] forward
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info] filters:
[2025/09/05 09:30:38] [ info] modify.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info] outputs:
[2025/09/05 09:30:38] [ info] stdout.0
[2025/09/05 09:30:38] [ info] ___________
[2025/09/05 09:30:38] [ info] collectors:
[2025/09/05 09:30:38] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/09/05 09:30:38] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/09/05 09:30:38] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/09/05 09:30:38] [ info] [simd ] disabled
[2025/09/05 09:30:38] [ info] [cmetrics] version=1.0.4
[2025/09/05 09:30:38] [ info] [ctraces ] version=0.6.6
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] initializing
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] storage_strategy='memory' (memory only)
[2025/09/05 09:30:38] [debug] [forward:forward.0] created event channels: read=25 write=26
[2025/09/05 09:30:38] [debug] [in_fw] Listen='0.0.0.0' TCP_Port=24224
[2025/09/05 09:30:38] [debug] [downstream] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [ info] [input:forward:forward.0] listening on 0.0.0.0:24224
[2025/09/05 09:30:38] [debug] [filter:modify:modify.0] Initialized modify filter with 1 conditions and 1 rules
[2025/09/05 09:30:38] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/09/05 09:30:38] [ info] [output:stdout:stdout.0] worker #0 started
[2025/09/05 09:30:38] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2025/09/05 09:30:38] [ info] [sp] stream processor started
[2025/09/05 09:30:38] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Conditions not met, not touching record
[2025/09/05 09:30:57] [debug] [task] created task=0x7f002d4296e0 id=0 OK
[2025/09/05 09:30:57] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
{"date":1757064652.918883,"time":"2025-09-05T09:30:52.918883142+00:00","stream":"stdout","log":"HTTP server listening on :8080","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"M2M3OGY1ZDYtY2E2MC00ZjBjLWE3NTEtOThiMDQxYzQ3OWJj"}
{"date":1757064652.919008,"time":"2025-09-05T09:30:52.919007627+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.918899636Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"org.springframework.boot.devtools.env.DevToolsPropertyDefaultsPostProcessor","message":"Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZDcxZjJjMDQtYmUxMC00Y2ZhLThkZTMtY2UxNDNmY2JmMmQ3"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","log":"Finished printing sample Spring Boot log lines.","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"ZjgyYTEwZDMtMzZmNC00MWQyLThkMGQtY2ZmNGIyZTRiNDU2"}
[2025/09/05 09:30:57] [debug] [out flush] cb_destroy coro_id=0
[2025/09/05 09:30:57] [debug] [task] destroy task=0x7f002d4296e0 (task_id=0)
---
For some reason FILTER says that key “*message*” does not exist but if you check the ouput you can clearly see that key “*message*” is there:
[2025/09/05 09:30:57] [debug] [filter:modify:modify.0] Condition not met : Key_exists message
...
{"date":1757064652.919029,"time":"2025-09-05T09:30:52.919028978+00:00","stream":"stdout","kubernetes":{"container_name":"log-generator","namespace_name":"some-namespace-cz","pod_name":"some-pod-68c68cbc94-bttf2","container_image":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator:latest","container_image_id":"image-registry.openshift-image-registry.svc:5000/some-namespace-cz/log-generator@sha256:cf810fb1770a73f04c90d25569d05e1d0833c5b1b8e36d9d46ea2238f7b5a8b1","pod_ip":"172.32.98.190","host":"some-host","labels":{"pod-template-hash":"68c68cbc94","app_kubernetes_io/instance":"log-generator","app_kubernetes_io/name":"log-generatorlog-generator"}},"@timestamp":"2025-09-05T09:30:52.919006474Z","log_level":"INFO","application":"some-namespace","thread":"restartedMain","logger":"redacted","message":"xxx","es_index_name":"ocp.app.some-namespace-cz.dev","_hash":"MzUwYTJjYTUtNjE2ZS00NWVkLWI5ZDgtNGJmZWJiODljZjIx"}
I’m definitely missing something crucial. I have checked docs, github issues and also tried three different AI tools but all of them saying that FILTER should work 😄….
I would greatly appreciate if someone could point me in the direction how I could debug it!
Thanks.Pat
09/05/2025, 9:49 AMPat
09/05/2025, 9:50 AMmessage keyPat
09/05/2025, 9:51 AMjl
09/05/2025, 9:52 AMjl
09/05/2025, 9:52 AMPat
09/05/2025, 9:52 AMPat
09/05/2025, 9:55 AMPat
09/05/2025, 9:56 AMjl
09/05/2025, 9:57 AMPat
09/05/2025, 9:58 AMPat
09/05/2025, 9:58 AMPat
09/05/2025, 9:58 AMPat
09/05/2025, 9:59 AMjl
09/05/2025, 10:03 AMPat
09/05/2025, 10:04 AMPat
09/05/2025, 10:04 AMPat
09/05/2025, 10:04 AMjl
09/12/2025, 10:23 AM-- Recursively convert all values in a record to strings
local function stringify(val)
local t = type(val)
if t == "table" then
local out = {}
for k,v in pairs(val) do
-- also convert keys to string just to be safe
out[tostring(k)] = stringify(v)
end
return out
else
return tostring(val)
end
end
function stringify_record(tag, ts, record)
local converted = stringify(record)
-- Debug output (optional)
print(string.format("[normalize] tag=%s ts=%s", tostring(tag), tostring(ts)))
-- Return the fully stringified record
return 1, ts, converted
endPat
09/12/2025, 10:31 AMPat
09/12/2025, 10:31 AMPat
09/12/2025, 10:32 AMPat
09/12/2025, 10:33 AMjl
09/12/2025, 10:34 AMjl
09/12/2025, 10:35 AMjl
09/12/2025, 10:36 AMPat
09/12/2025, 10:36 AMPat
09/12/2025, 10:37 AMjl
09/12/2025, 10:44 AMPat
09/12/2025, 10:45 AMPat
09/12/2025, 10:45 AM