Hello,I use fluent-bit input plugin "tail" to read...
# fluent-bit
y
Hello,I use fluent-bit input plugin "tail" to read log(json_in_lines) ,then sent logs to Kafka. In my fluent-bit config at Kafka output section,use Topics ${TAG} ,but doesn't work. fluent-bit logs show kafka topic still configs to fluent-bit. Appreciate your help.
p
I'm pretty sure there's an open issue on this. Ultimately record accessor syntax is not always available for every config option
y
well.. first,thank you for replying. I also try to use modify filter add a new field "router" and assign value from ${TAG} as doc said. Then using Kafka output plugin settings "topics" and "topic_key" to point Kafka topic,but still failure . https://docs.fluentbit.io/manual/pipeline/outputs/kafka It seem accessor syntax not work in modify filter ?
p
It works fine, I suspect you're using topic key incorrectly. What's your full config when you use it?
You can use the stdout output as well to print out the record after the modify filter to check it's what you expect. Almost always there's an issue with some assumptions around the data so seeing the actual record is usually helpful
Ah no, curly braces is the syntax for variables not record accessor so it's trying to substitute a tag variable I suspect
https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/record-accessor is what you want, note it's just
$
rather than
${}
. Also I can't remember if TAG is part of that syntax or just available sometimes
y
@Pat thank you~~ here is my configuration
[INPUT] name tail path /NAS/*.log parser json tag test-json storage.type filesystem Threaded true db /opt/pos_db/test-json.db [Filter] Name modify Match * Set router ${TAG} [OUTPUT] Name kafka Match * topics test-json topic_key router dynamic_topic On
p
As I say, the syntax you're using is for variables not record accessor. Use stdout output to see your actual data as well.
Also, you should not mix topics and topic key: https://docs.fluentbit.io/manual/pipeline/outputs/kafka Looking at those docs you only have one topic set so it's using that and not the key (which is probably not set anyway because of the previous issue)
The lua filter may be better, there's an example of setting the tag: https://docs.fluentbit.io/manual/pipeline/filters/lua#inline-configuration
So: 1. Switch to the lua filter to set the key 2. Only use topic_key 3. Add a stdout output to see the actual record