William
07/07/2025, 5:51 PMrewrite_tag filter as an input processor in their YAML config.
I currently use the tail plugin to read Kubernetes container logs files. On this input, I define 3 rewrite_tag "filters as processors" in order to categorize the records based on some internal grouping logic. The idea is to ensure that logical batch of records are grouped together in the final chunks, and consumed by different outputs. Each filter re-emit a record if a condition matches (based on the presence of a kubernetes pod annotation enriched by the kubernetes filter).
The thing is, when I check the metrics, especially the v2 metric fluentbit_filter_records_total , I noticed that the nth emitter defined in the config always shows N+X records, where N is the numbers of records processed by the kubernetes filter itself, before in the processors list, and X is the amount of records that matched the condition of the previous rewrite_tag filter.
To give an example, here is the config of the 3 processors:
- name: rewrite_tag
match: kube.*
alias: rewrite_syslog
rule: "$kubernetes_annotation_syslog/enabled ^(true)$ syslog true"
emitter_name: re_emit_syslog
emitter_storage.type: filesystem
- name: rewrite_tag
match: kube.*
alias: rewrite_kafka
rule: "$kubernetes_annotation_kafka-topic ^([a-zA-Z0-9]+[-._]?)+[a-zA-Z0-9]+$ kafka true"
emitter_name: re_emit_kafka
emitter_storage.type: filesystem
emitter_mem_buf_limit: 100M
- name: rewrite_tag
match: kube.*
alias: rewrite_graylog
rule: "$kubernetes_annotation_logs/sink-addr ^(host1|host2)$ graylog.$0 true"
emitter_name: re_emit_graylog
emitter_storage.type: filesystem
The v2 metrics are as follows:
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="kubernetes"} = 242
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_syslog"} = 242
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_kafka"} = 414
2025-07-07T17:30:55.121280325Z fluentbit_filter_records_total{name="rewrite_graylog"} = 426
If I match this with the metrics of the processed records per output (that match the new tags emitted by the rewrite_tag processors):
2025-07-07T17:30:50.282031744Z fluentbit_output_proc_records_total{name="kafka"} = 12
2025-07-07T17:30:54.274884272Z fluentbit_output_proc_records_total{name="graylog"} = 300
2025-07-07T17:30:54.275732061Z fluentbit_output_proc_records_total{name="syslog"} = 172
We can see that:
• kafka - syslog = 172 (syslog output processed records count)
• graylog - kafka = 12 (kafka output processed records count)
Per my understanding, the new records emitted by the rewrite_tag filter are re-evaluated from the beginning of the pipeline (and this shows that this is the case since the filters are clearly re-ingesting the new records emitted by previous filters), but I don't understand why that applies to the filters as processors of the tail input plugin.
I have tried to add a match: kube.* field to all rewrite_tag processors, but this has no effect, other that suppressing a warning at startup about a potential infinite loop (which never happens anyway).
I know that I could move the filters in the main thread, but I actually do not want to go that route for performance reasons, and because I use threaded inputs.Celalettin
07/08/2025, 2:14 PMmatch parameter only controls which records the filter processes, but it doesn't prevent re-emitted records from being processed by subsequent filters. As the multiline filter documentation notes: "Logs will be re-emitted by the multiline filter to the head of the pipeline- the filter will ignore its own re-emitted records, but other filters won't." multiline AFAIK this same principle applies to rewrite_tag.
I don't think of an alternative other than accepting this behaviourWilliam
07/08/2025, 2:21 PMmatch property on the rewrite_tag filters defined as processors on my tail input, it shouldn't match the re-emitted records since they have a different tag.
I presume that the metric fluentbit_filter_records_total does count records before the rule is evaluated to decide if the records should be handled by the filter/processor, because apart from the "metrics weirdness", I haven't noticed any other issues/duplicate records.