Hey Team, I have a quick question. I’m planning to enable log compression in the fluentbit config and will this switch increase or decrease the memory footprint? I’m factoring in the backpressure as well here. Do let me know. I wanted to hear your thoughts/suggestions.

It depends is always going to be the answer 😂 Do you mean compression for the output? Almost all of them use the standard compression libraries to do the work so there's pretty good evidence for those already. I don't think there's any other type of compression you can enable, we do have some support for reading compressed logs though but that's not in OSS yet I don't think. Pretty simple though to test and verify locally even just with a simple container deployment though, that's probably the best option anyway as you get real world data.

Hey @Pat , Yes, it’s about enabling log compression and then shipping it to the destination. With the compression enabled, I know the cpu footprint goes up since the fluentbit has to do a compute intensive operation of compressing the logs but I wanted to know the impact on memory footprint. In a high log throughput environment, if I enable log compression, will it increase the existing backpressure? Since fluentbit might hold logs for some time in buffer to zip/compress it, eventually increasing the memory footprint? or since the log chunks get compressed, they might be shipped faster to the destination, which flushes out the memory buffer that reduces backpressure, eventually reducing the overall memory footprint?

What log compression do you mean though? Some outputs support it