Greg
09/29/2025, 2:58 PMMultiline: Buffer truncation limit, metrics and diagnostics .
We have a particular problem where we see high memory usages when we receive large log lines, which appears to be linked to the multiline processor.
Ideally we would fix this in the log source, and just never have to handle large log lines, but unfortunately this is out of our control.
So if this "buffer truncation" feature works anything like I hope, it would be great for us.
How do I configure this to be enabled?
cc @hhatakeGreg
09/29/2025, 2:59 PMGreg
09/29/2025, 3:00 PMservice:
multiline_buffer_limit: 200K
But it didn't seem to do anythinghhatake
09/30/2025, 1:55 AMmultiline_buffer_limit .
But this truncation should be effective for repeatedly continuous lines which are needed to concatenate.
It's weird. So, I'll dig on it.hhatake
09/30/2025, 8:12 AMGreg
09/30/2025, 8:24 AMmultiline to recombine them.Greg
09/30/2025, 8:24 AMGreg
09/30/2025, 8:25 AMpipeline:
inputs:
- name: forward
tag: logs
unix_path: /var/run/fluent.sock
mem_buf_limit: 32m
threaded: true
processors:
logs:
- name: multiline
multiline.key_content: log
mode: partial_message
- name: parser
key_name: log
parser: dockerGreg
09/30/2025, 8:28 AMlua filter later in the pipeline, as the downstream log storage can only handle 256KB logs)Greg
09/30/2025, 8:29 AMmultiline_buffer_limit, then truncate. is that right?hhatake
09/30/2025, 8:32 AMhhatake
09/30/2025, 8:34 AM[SERVICE]
storage.path /var/lib/fluent-bit/storage
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M # RAM used for backlog before spill
[INPUT]
Name forward
Tag logs
Unix_Path /var/run/fluent.sock
Mem_Buf_Limit 32M
storage.type filesystemGreg
09/30/2025, 9:17 AMYeah, it could be but the memory spikes should be caused by high volume ingestion on in_forward.๐ค Okay, in my simple testing, if I removed
multiline I didn't see memory usage increase so much.
To be clear, when I set mem_buf_limit: 32m I expect to see memory usage increase by 32MB due to back pressure etc, and I also expect to see some memory usage elsewhere. When I stress test with "small" log lines (up to around 250KB) I see memory increase, but only more or less up to a ceiling value of maybe 90MB.
However, when I stress test with large log lines of 100MB, I see memory usage grow unconstrained, limited only to the rate I can send the logs. I easily see memory usage of 200MB or more.Greg
09/30/2025, 9:19 AMGreg
09/30/2025, 9:20 AMGreg
09/30/2025, 9:21 AMmultiline_buffer_limit working well, as I assume it could help with at least part of the problem.Greg
09/30/2025, 9:22 AMAh, I got it. The current code could not truncate if the ingested logs were not met for the concatenating rules. When failing to concatenate, just pass-through as-is. So, there is no effect to handle with this feature.But doesn't this mean it should work for the docker use-case?
hhatake
09/30/2025, 10:17 AMhhatake
09/30/2025, 10:33 AMhhatake
09/30/2025, 10:43 AMMem_Buf_Limit parameter on input plugins when not using filesystem buffering mechanism.hhatake
09/30/2025, 10:45 AMhhatake
09/30/2025, 10:54 AMmultiline_buffer_limit: 0. This is because in_forward does not have flow control mechanism by itself. So, it could be easily to become a bottleneck of pipeline if it is enabled.Greg
09/30/2025, 11:10 AMBecause multiline truncation is mainly focused on using in_tail and I targeted to use with it. So, it's not effective for using with in_forward, I supposeis it technically feasible to work with
in_forward ? I could try to find some time to help with this, if it would be useful.Greg
09/30/2025, 11:12 AMThis is because in_forward does not have flow control mechanism by itself. So, it could be easily to become a bottleneck of pipeline if it is enabled.We currently configure
mem_buf_limit for in_forward. When we hit this level ingestion is paused, which at least protects FB (although we then get queuing upstream, in the docker log driver buffer)hhatake
09/30/2025, 11:12 AMhhatake
09/30/2025, 11:15 AMWe currently configureAh, sounds nice. Using withfor in_forward. When we hit this level ingestion is paused, which at least protects FB (although we then get queuing upstream, in the docker log driver buffer)mem_buf_limit
mem_buf_limit and some certain buffer limit of multiline could work for limiting the latter pipeline resource usages. But, temporary, it will increase memory usages on processing intermediate statuses of processing multilines.Greg
09/30/2025, 11:21 AMhhatake
09/30/2025, 11:28 AMGreg
09/30/2025, 11:40 AMlog fields into a buffer until it has reconstructed the original 100MB log line
7) This is passed to a LUA in the pipeline, which truncates the message to 250KB ๐คฆ
It seems it would be an improvement (for me) to truncate in multilineGreg
09/30/2025, 11:40 AMhhatake
09/30/2025, 11:42 AM