A k8s cluster has recently started increasing logging to Splunk, and with that, started having fluentbit workers balooning and reaching OOM after filling up 32 GB of memory. We do see spikes of data transmitting (100 MB/sec on one fluentbit pod), and then seeing memory start rising. What parameters should we focus on adjusting to ease this issue? Of the parameters i've gathered so far, we have these defined:

spec:
  tail:
    bufferChunkSize: 2M
    bufferMaxSize: 8M
    db: /fluent-bit/tail/pos.db
    dbSync: Normal
    excludePath: /var/log/containers/*_argocd_*.log
    memBufLimit: 50MB
    multilineParser: cri
    path: /var/log/containers/*.log
    pathKey: myCompanySplunkEventSource
    refreshIntervalSeconds: 10
    skipLongLines: true
    storageType: filesystem
    tag: kube.*

On the splunk side, we see data transmission up to about 600 GB per day from this cluster.

hey @ELT1 - disclaimer: I am a novice user not an expert of Flb 🙂 here you have set

storageType: filesystem

but I don't see

storage.path

so maybe Fluent Bit may still be leaning heavily on memory for the "active" chunks. if you have such kind of "back pressure" in memory (or maybe there were a leak somewhere else in Flb ??) the output must be the bottleneck to be optimised?

The multiline parsers don't respect buffer limits which can lead to massive memory usage when large logs are output. This configuration value was added to limit memory consumption for multiline parsers https://docs.fluentbit.io/manual/data-pipeline/parsers/multiline-parsing#additional-configuration-for-buffer-limit-of-multiline