@Pat, could you please help me with this? I would really appreciate your guidance on the available alternatives and any recommendations you may have.

A quick google found this which @Anurag Gupta raised a few years ago so he may have some more/better suggestions? github.com/fluent/fluent-bit/discussions/5211

Ah it looks like there is no straightforward separator for messages, instead it is a binary protocol with a "size" value you have to parse for each message: github.com/elastic/logstash-forwarder/blob/…/PROTOCOL.md The TCP input plugin will not be able to handle it therefore currently it seems.

Further googling shows another article from @Anurag Gupta: chronosphere.io/learn/fluent-bit-vs-beats This has a worked example of sending data from Beats to Fluent Bit using the

in_elasticsearch

plugin so should be doable it seems. I would have a look at that and give it a go - you can then just use a

file

output plugin to write the data out. Not sure why you need to use another plugin to do it as it's all in the Elasticsearch one but I guess there's a specific reason, however I cannot find a way you can do that directly.

You can probably write a custom plugin based off the TCP one that handles the Lumberjack protocol directly (or update the TCP one itself to support binary protocols with a size field in them). The ES plugin will be based on the HTTP one but you could also potentially adjust this to support Lumberjack as well. If it was me I'd probably tweak the TCP one as support for binary protocols with a size field would be useful for everyone so could be merged back in. @hhatake may know of another supported way to do it already or some suggestions on the TCP plugin approach.

@Pat I'm using the Elastic input plugin receiving logs from multiple Filebeat senders. ​In in_syslog, I can easily capture the sender IP using Source_Address_Key. However, with Filebeat/Elastic inputs, the sender IP resides in an array (e.g., host.ip[0]), or I need to explicitly map the connection socket IP to a custom top-level field like siem-src-device-ip. Since array indexing like host.ip[0] isn't supported out-of-the-box by standard filters like modify, what is the best production-safe alternative to add socket remote addresses into a top-level JSON key? ​Is filter_nest or filter_rewrite_tag capable of this, or is Lua the only recommended path for array field extraction? ​Thanks!