Sven
07/01/2025, 11:01 AMMax
07/01/2025, 4:42 PMEric D. Schabell
07/01/2025, 5:36 PMhenrik Rexed
07/02/2025, 9:13 AMEric D. Schabell
07/02/2025, 5:01 PMAnusha Hegde
07/03/2025, 5:07 AMNizar
07/03/2025, 9:08 AMoutputs: |
[OUTPUT]
Name loki
Match kube.*
host lokiurl
port 3100
uri /loki/api/v1/push
labels collector="fluentbit"
auto_kubernetes_labels true
So logically in my logs i should have a new label called collector with the value fluentbit right ?Eric D. Schabell
07/03/2025, 9:15 AMEric D. Schabell
07/03/2025, 9:32 AMAnshuman Goyal
07/03/2025, 10:10 AMbasic_auth:
username: loki
password_file: /etc/secrets/password-file
Eric D. Schabell
07/03/2025, 10:10 AMEric D. Schabell
07/03/2025, 1:16 PMSupriya Mishra
07/03/2025, 5:32 PMAshish
07/04/2025, 7:59 AMAnton
07/04/2025, 12:26 PMSaral
07/04/2025, 12:26 PM- lua:
call: normalize_product_logs
script:
name: fluent-bit-modify-k8s-fields
key: custom.lua
The custom.lua
script uses environment variables like drop_error_product
, drop_debug_product
, etc., to control which logs should be dropped dynamically. For example:
if log_level == "ERROR" and drop_error_product == "true" then
return -1 -- drop log
end
These variables are passed via environment variables in the Fluent Bit container or set via config maps.
Problem
When I update the values of these variables (e.g., setting drop_error_product
from false
to true
) and confirm that the ConfigMap
is updated and pods are restarted manually, the updated logic does not take effect.
However, the moment I:
• remove the entire Lua block from the filter config and deploy
• then add the same Lua block back and deploy again
...the changes start reflecting and logs get dropped correctly.
Is Fluent Bit or the Lua plugin caching the script or its environment somewhere internally?
Is there a supported or recommended way to force Fluent Bit to reload updated Lua script logic and/or environment config?
Thanks a lot — any insights or best practices would be appreciated!Eric D. Schabell
07/04/2025, 2:02 PMEric D. Schabell
07/04/2025, 3:47 PMEric D. Schabell
07/07/2025, 1:05 PMEric D. Schabell
07/07/2025, 4:07 PMEric D. Schabell
07/07/2025, 4:13 PMWilliam
07/07/2025, 5:51 PMrewrite_tag
filter as an input processor in their YAML config.
I currently use the tail
plugin to read Kubernetes container logs files. On this input, I define 3 rewrite_tag
"filters as processors" in order to categorize the records based on some internal grouping logic. The idea is to ensure that logical batch of records are grouped together in the final chunks, and consumed by different outputs. Each filter re-emit a record if a condition matches (based on the presence of a kubernetes pod annotation enriched by the kubernetes
filter).
The thing is, when I check the metrics, especially the v2 metric fluentbit_filter_records_total
, I noticed that the nth emitter defined in the config always shows N+X records, where N is the numbers of records processed by the kubernetes
filter itself, before in the processors list, and X is the amount of records that matched the condition of the previous rewrite_tag
filter.
To give an example, here is the config of the 3 processors:
- name: rewrite_tag
match: kube.*
alias: rewrite_syslog
rule: "$kubernetes_annotation_syslog/enabled ^(true)$ syslog true"
emitter_name: re_emit_syslog
emitter_storage.type: filesystem
- name: rewrite_tag
match: kube.*
alias: rewrite_kafka
rule: "$kubernetes_annotation_kafka-topic ^([a-zA-Z0-9]+[-._]?)+[a-zA-Z0-9]+$ kafka true"
emitter_name: re_emit_kafka
emitter_storage.type: filesystem
emitter_mem_buf_limit: 100M
- name: rewrite_tag
match: kube.*
alias: rewrite_graylog
rule: "$kubernetes_annotation_logs/sink-addr ^(host1|host2)$ graylog.$0 true"
emitter_name: re_emit_graylog
emitter_storage.type: filesystem
The v2 metrics are as follows:
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="kubernetes"} = 242
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_syslog"} = 242
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_kafka"} = 414
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_graylog"} = 426
If I match this with the metrics of the processed records per output (that match the new tags emitted by the rewrite_tag
processors):
2025-07-07T17:30:50.282031744Z fluentbit_output_proc_records_total{name="kafka"} = 12
2025-07-07T17:30:54.274884272Z fluentbit_output_proc_records_total{name="graylog"} = 300
2025-07-07T17:30:54.275732061Z fluentbit_output_proc_records_total{name="syslog"} = 172
We can see that:
• kafka - syslog = 172 (syslog output processed records count)
• graylog - kafka = 12 (kafka output processed records count)
Per my understanding, the new records emitted by the rewrite_tag
filter are re-evaluated from the beginning of the pipeline (and this shows that this is the case since the filters are clearly re-ingesting the new records emitted by previous filters), but I don't understand why that applies to the filters as processors of the tail
input plugin.
I have tried to add a match: kube.*
field to all rewrite_tag
processors, but this has no effect, other that suppressing a warning at startup about a potential infinite loop (which never happens anyway).
I know that I could move the filters in the main thread, but I actually do not want to go that route for performance reasons, and because I use threaded inputs.Eric D. Schabell
07/08/2025, 9:46 AMEric D. Schabell
07/08/2025, 11:49 AMF
07/08/2025, 12:49 PMPawel Gieniec
07/08/2025, 8:17 PMPriyanka
07/09/2025, 9:47 AMEric D. Schabell
07/09/2025, 6:05 PMDennyF
07/10/2025, 9:19 AM$message =~ s/cardNo>\d\d\d\d\d\d\d\d\d\d\d\d/cardNo>XXXXXXXXXXXX/g;
$message =~ s/cvv>\d\d\d/cvv>XXX/g;
$message =~ s/cvv=\d\d\d/cvv=XXX/g;
$message =~ s/Verification%3E\d\d\d/Verification%3EXXX/g;
$message =~ s/R19%3E\d\d\d/R19%3EXXX/g;
$message =~ s/R16%3E\d\d\d\d\d\d\d\d\d\d\d\d/R16%3EXXXXXXXXXXXX/g;
$message =~ s/ccn=\d\d\d\d\d\d\d\d\d\d\d\d/ccn=XXXXXXXXXXXX/g;
$message =~ s/card_account=\d\d\d\d\d\d\d\d\d\d\d\d/card_account=XXXXXXXXXXXX/g;
$message =~ s/cvc_code=\d\d\d/cvc_code=XXX/g;
$message =~ s/password=/Password=XXX\&/g;
to replace sensitive creditcard data. How we can do the same in fluent-bit ?Eric D. Schabell
07/10/2025, 4:58 PM