https://linen.dev logo
Join Slack
Powered by
I wasn't quite sure whether to raise a GitHub issu...
# fluent-bit
g
I wasn't quite sure whether to raise a GitHub issue for this just yet since I don't believe it's a fluentbit problem per-se. Under high traffic volume with 
storage.type=filesystem
we see a rapid leak of memory in-use by the fluent-bit pod in k8s. Growing to as much as 16GB after 1d or so without a pod restart. We see that fluent-bit itself is not consuming much memory, maybe a few hundred MB, but rather that kernel slab associated to the container cgroup is accounting for all of the excess memory. 
slabtop
claims that VFS dentry cache is accounting for all of those leaked kernel objects. The behavior we're seeing is that since we are running buffering a large # of chunks/sec, we are creating easily hundreds of chunk files per second which leaks dentry entries rather quickly. Even upon file deletion the kernel will keep negative dentries which cache the non-existence of a file, and they aren't purged from kernel cache all that easily unless the system is under memory pressure. More context on this topic: https://lwn.net/Articles/894098/ Is this dentry cache bloat a well-known problem in the fluent-bit community? Are there good solutions / workarounds? Some workarounds we've considered, but are looking for guidance from maintainers & community: 1. Raise VFS cache pressure on the nodes. I'm not 100% sure on how much this changes VFS cache behavior here, and not sure what perf consequences this can have on the rest of workloads on the Node. It's worth experimenting with. 2. Periodically reboot fluent-bit pods. This resets its memory accounting, however doesn't actually clean up the bloat in the dentry cache since it's a system wide cache. If our system gets into memory pressure, the sheer volume of dentry entries could lock-up the system. Feels like sweeping a bigger problem under the rug. 3. Periodically migrate fluent-bit storage directory to another directory and delete the old directory. Supposedly when a directory is deleted, a negative dentry is kept for it, but nested entries are pruned since they are now made redundant. I think this is the most plausible option since we can add a script wrapper around fluent-bit to gracefully shut it down, reconfigure, and re-start, no code changes are required in fluent-bit itself. How do we handle periods of backpressure when there is an existing backlog of chunks? One idea to improve things within fluent-bit itself would be to re-use chunk file names so those cached dentries can be re-used. Either that, or use larger pre-allocated files with block-arena like memory management to store FS chunks. This may be more efficient? You can always add more files or extend the block arena if the FS storage buffer needs to grow. CC @Pandu Aji
p
@Pat @eduardo @Anurag Gupta, im wondering if you have any workaround for us. Thanks in advance
This is our forwarder config if you need it: 
[SERVICE]
Flush 1 Log_Level info Parsers_File /fluent-bit/etc/parsers.conf Parsers_File /forwarder/etc/parsers_custom.conf Plugins_File /fluent-bit/etc/plugins.conf HTTP_Server On storage.path /var/log/flb-storage/ storage.max_chunks_up 256 storage.backlog.mem_limit 512M storage.metrics on Health_Check On HC_Errors_Count 5 HC_Retry_Failure_Count 5 HC_Period 30 [INPUT] Name tail Alias tail.container Tag kube.* Path /var/log/containers/*_istio-*,/var/log/containers/*_kube-*,/var/log/containers/*_ns1-*,/var/log/containers/*_ns2-*,/var/log/containers/*_ns3-* Exclude_Path /var/log/containers/*_fluentbit-* Parser cri-o multiline.parser cri DB /var/log/flb-tail-kube.db DB.sync normal Refresh_Interval 15 Read_from_Head On Buffer_Chunk_Size 1M Buffer_Max_Size 4M Skip_Long_Lines On Mem_Buf_Limit 64M storage.type filesystem Threaded On Ignore_Older 1d [INPUT] Name tail Alias tail.container_big_ns Tag kube.* Path /var/log/containers/*_big-ns* Parser cri-o multiline.parser cri DB /var/log/flb-tail-big-ns.db DB.sync normal Refresh_Interval 15 Read_from_Head On Buffer_Chunk_Size 4M Buffer_Max_Size 16M Skip_Long_Lines On Mem_Buf_Limit 128M storage.type filesystem Threaded On Ignore_Older 1d [INPUT] Name tail Alias tail.syslog_mariner Tag syslog.info Path /var/log/messages Parser syslog-rfc3164-mariner-time-format DB /var/log/flb-tail-syslog.db DB.sync normal Read_from_Head On Buffer_Chunk_Size 128K Buffer_Max_Size 1M Skip_Long_Lines On Mem_Buf_Limit 32M storage.type filesystem Threaded On Ignore_Older 1d [INPUT] Name fluentbit_metrics Alias metrics.fluentbit Tag metrics.fluentbit scrape_interval 15 [FILTER] Name kubernetes Alias filter.kubernetes Match kube.var.log.containers.* Kube_Tag_Prefix kube.var.log.containers. Annotations Off K8S-Logging.Exclude On Kube_Meta_Cache_TTL 300 Buffer_Size 512K [FILTER] Name grep Alias grep.filter_syslog_audit Match syslog.info Exclude ident audit [FILTER] Name rewrite_tag Alias rewrite_tag.istio_log_gateway Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] ^istio-ingress$ istio.log.gateway false Emitter_Name re_emitted.istio_log_gateway Emitter_Mem_Buf_Limit 16M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.istio_proxy Match kube.var.log.containers.* Rule $kubernetes['container_name'] ^istio-proxy$ istio.log.proxy false Emitter_Name re_emitted.istio_proxy Emitter_Mem_Buf_Limit 16M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.istio_init Match kube.var.log.containers.* Rule $kubernetes['container_name'] ^istio-init$ container.log false Emitter_Name re_emitted.istio_init Emitter_Mem_Buf_Limit 2M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.large_ns_log Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] ^large-ns$ large.container.log false Emitter_Name re_emitted.large_ns_log Emitter_Mem_Buf_Limit 64M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.ns1_log Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] ^ns1$ ns1.log false Emitter_Name re_emitted.ns1_log Emitter_Mem_Buf_Limit 16M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.ns2_log Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] ^ns2$ ns2.log false Emitter_Name re_emitted.ns2_log Emitter_Mem_Buf_Limit 24M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.ns3_log Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] ^ns3$ ns3.log false Emitter_Name re_emitted.ns3_log Emitter_Mem_Buf_Limit 16M Emitter_Storage.type filesystem [FILTER] Name rewrite_tag Alias rewrite_tag.kube_ns Match kube.var.log.containers.* Rule $kubernetes['namespace_name'] (.*) kube.$1 false Emitter_Name re_emitted.kube_ns Emitter_Mem_Buf_Limit 4M Emitter_Storage.type filesystem [OUTPUT] Name forward Alias forward.hyena Match_Regex ^kube\.(istio-system|kube)(-\w+)*|syslog\.info$ Host genevabit-aggregator Port 24224 Retry_Limit 10 Require_ack_response True storage.total_limit_size 16G net.connect_timeout 30 net.keepalive on net.keepalive_idle_timeout 10 net.keepalive_max_recycle 600 [OUTPUT] Name forward Alias forward.common Match_Regex ^istio\.log\.(gateway|proxy)|ns1\.log|ns2\.log|ns3\.log$ Host common-aggregator.common-telemetry Port 24224 Retry_Limit 10 Require_ack_response True storage.total_limit_size 16G net.connect_timeout 30 net.keepalive on net.keepalive_idle_timeout 10 net.keepalive_max_recycle 600 [OUTPUT] Name forward Alias forward.openai Match_Regex ^large.container.log$ Host large-aggregator.large-telemetry Port 24224 Retry_Limit 10 Require_ack_response True storage.total_limit_size 32G net.connect_timeout 30 net.keepalive on net.keepalive_idle_timeout 10 net.keepalive_max_recycle 600
@Pat @eduardo @Anurag Gupta, gentle ping. Could you help us?
e
@Pandu Aji most important question: which version ?
Open in Slack
PreviousNext
Powered by