Looks like the arbitrary 10 limit is still in plac...
# fluent-bit
p
Looks like the arbitrary 10 limit is still in place in latest version as well, I'm guessing it was just a number picked initially. @hhatake can you remember if it was specifically chosen?
github.com/fluent/fluent-bit/blob/…/flb_cf_yaml.c#… is the current code where we limit it to 10 processors I think
h
Hi, this is limitation of the current limitation. This is because filter_plugins run on the same thread of input (maybe output?) plugin so we need to limit the operation. By using processors, this limitation should not be effective for the each of processor plugins. Nowadays, we recommend to use processor_content_modifier in this case.
p
OK, probably need to update the issue and docs then I think for it as well @Eric D. Schabell
👍 1
l
it seems wider than just processors though right? it affects routing, outputs, filters etc?
I can work around outputs/filters by including a new file when I have more, but the routing doesn't merge
h
Internally, we've been suffered by the threading design of filter plugins so we decided to provide the new type of plugins. It could be handled by conditional routings but there's limitation to convert all of the use cases of filter_content_modifier, I suppose.
l
ok, thanks for the context, it helps me pick
4. Split out into separate files and go back to tag based routing for now
since my understanding of this discussion is:
It's not desired but not a bug. It's quite core so any changes if/when will need to be done by someone who understands it and with some planning amongst yourselves
👍 1