Hey All, I'm interested in the new FB 4.1 feature ...
# fluent-bit
g
Hey All, I'm interested in the new FB 4.1 feature
Multiline: Buffer truncation limit, metrics and diagnostics
. We have a particular problem where we see high memory usages when we receive large log lines, which appears to be linked to the multiline processor. Ideally we would fix this in the log source, and just never have to handle large log lines, but unfortunately this is out of our control. So if this "buffer truncation" feature works anything like I hope, it would be great for us. How do I configure this to be enabled? cc @hhatake
I didn't see anything regarding this in the docs so far
I tried adding
Copy code
service:
    multiline_buffer_limit: 200K
But it didn't seem to do anything
h
I added a configuration prameter for multiline truncation that is
multiline_buffer_limit
. But this truncation should be effective for repeatedly continuous lines which are needed to concatenate. It's weird. So, I'll dig on it.
Ah, I got it. The current code could not truncate if the ingested logs were not met for the concatenating rules. When failing to concatenate, just pass-through as-is. So, there is no effect to handle with this feature.
g
I am unsure what to do with this. Our use-case is for reading logs from docker, via the fluent forward log driver. Docker splits large logs into chunks (I think 64k), and we use
multiline
to recombine them.
We use it as an input processor
like this:
Copy code
pipeline:
  inputs:
    - name: forward
      tag: logs
      unix_path: /var/run/fluent.sock
      mem_buf_limit: 32m
      threaded: true
      processors:
        logs:
          - name: multiline
            multiline.key_content: log
            mode: partial_message

          - name: parser
            key_name: log
            parser: docker
When we receive a large log line, say 100MB, we see memory usage spike very high. Such large log line are effectively junk (and we would also truncate them in a
lua
filter later in the pipeline, as the downstream log storage can only handle 256KB logs)
As I understand this multiline truncate feature, it should help in our use-case, right? I thought it would start to recombine the docker chunks up to the
multiline_buffer_limit
, then truncate. is that right?
h
Yeah, it could be but the memory spikes should be caused by high volume ingestion on in_forward. When receiving large amount of data, you will encounter high memory usages because of using memory buffer that is default type of buffering system.
So, you probably need to adjust for memory usages something like to use filesystem buffering to avoid handling large amount of middle states of unprocessed buffers:
Copy code
[SERVICE]
    storage.path         /var/lib/fluent-bit/storage
    storage.sync         normal
    storage.checksum     off
    storage.backlog.mem_limit 5M   # RAM used for backlog before spill

[INPUT]
    Name                 forward
    Tag                  logs
    Unix_Path            /var/run/fluent.sock
    Mem_Buf_Limit        32M
    storage.type         filesystem
g
Yeah, it could be but the memory spikes should be caused by high volume ingestion on in_forward.
๐Ÿค” Okay, in my simple testing, if I removed
multiline
I didn't see memory usage increase so much. To be clear, when I set
mem_buf_limit: 32m
I expect to see memory usage increase by 32MB due to back pressure etc, and I also expect to see some memory usage elsewhere. When I stress test with "small" log lines (up to around 250KB) I see memory increase, but only more or less up to a ceiling value of maybe 90MB. However, when I stress test with large log lines of 100MB, I see memory usage grow unconstrained, limited only to the rate I can send the logs. I easily see memory usage of 200MB or more.
The problem for us is not so much "high" memory usage (although we would prefer it to be low), it is having a predictable ceiling usage, as FB is running as a sidecar container which shares the memory pool with the main app. If we can predict the maximum memory usage of FB, which can provision the shred memory correctly. If we can't predict the ceiling memory usage of FB, we will end up getting OOM ๐Ÿ˜ข
We did look at filesystem buffering, but as I recall it didn't solve the 100MB log line issue (I will double check this), and also made our throughput go down
Well, in any case, I would really like to know how to get the new
multiline_buffer_limit
working well, as I assume it could help with at least part of the problem.
Ah, I got it. The current code could not truncate if the ingested logs were not met for the concatenating rules. When failing to concatenate, just pass-through as-is. So, there is no effect to handle with this feature.
But doesn't this mean it should work for the docker use-case?
h
Off course. It does not mean it should work for the docker use-case.
Ah, I got realized some assumption glitches. Because multiline truncation is mainly focused on using in_tail and I targeted to use with it. So, it's not effective for using with in_forward, I suppose.
So, in your case, I supposed that logs ingestions of in_forward could be overloaded and this can be prevented with back pressures. To enable back pressures, we need to use
Mem_Buf_Limit
parameter on input plugins when not using filesystem buffering mechanism.
This is because Fluent Bit's memory usages are easily spikes when there is any bottlenecked pipelines without using filesystem buffering.
So, the current situation is: I'm not recommending to use multline truncation on in_forward pipeline. Just disabling with
multiline_buffer_limit: 0
. This is because in_forward does not have flow control mechanism by itself. So, it could be easily to become a bottleneck of pipeline if it is enabled.
g
Because multiline truncation is mainly focused on using in_tail and I targeted to use with it. So, it's not effective for using with in_forward, I suppose
is it technically feasible to work with
in_forward
? I could try to find some time to help with this, if it would be useful.
This is because in_forward does not have flow control mechanism by itself. So, it could be easily to become a bottleneck of pipeline if it is enabled.
We currently configure
mem_buf_limit
for in_forward. When we hit this level ingestion is paused, which at least protects FB (although we then get queuing upstream, in the docker log driver buffer)
h
On paper, it would work. But mechanism and assumptions are slightly glitches for my initial assumption to implement truncation feature of multiline. Just causes to generate bottleneck and memory usage spikes.
We currently configure
mem_buf_limit
for in_forward. When we hit this level ingestion is paused, which at least protects FB (although we then get queuing upstream, in the docker log driver buffer)
Ah, sounds nice. Using with
mem_buf_limit
and some certain buffer limit of multiline could work for limiting the latter pipeline resource usages. But, temporary, it will increase memory usages on processing intermediate statuses of processing multilines.
g
My understanding of how multiline works is as follows, is this correct? As each new line from a multiline log comes through, the contents are appended to the concatenation buffer. This keeps happening until it detects the "end" of the multiline log, at which point the buffer is released as a single log record. For your truncation feature, something similar happens, except it also checks the used size of the buffer, and once the limit is reached, no more data is appended. I assume following parts are continued to be read, but discarded, until the "end" of the multiline log is detected, at which point it restarts with a new, empty buffer. Is this the general contract for the behaviour?
๐Ÿ‘ 1
h
Yes. Itโ€™s general contract of behavior. And when using limit of multiline buffers, this could cause overloading and slowing down the processing speed. So, itโ€™ll become one of the bottlenecks.
g
To be clear, my current situation is 1) An app sends a 100MB log to the console 2) Docker reads this, and chops it up into 16KB chunks 3) Docker sends these chunks to a unix socket using the fluent forward protocol 4) FB reads from this socket using in_forward 5) Each 16KB chuck is passed to multiline 6) Multiline concatenates the
log
fields into a buffer until it has reconstructed the original 100MB log line 7) This is passed to a LUA in the pipeline, which truncates the message to 250KB ๐Ÿคฆ It seems it would be an improvement (for me) to truncate in multiline
If I ever get enough spare time, I may have a look at how to implement this. Although I am not a skilled C programmer, so I may need some hand holding...
๐Ÿ™‡ 1
h
Not sure I would help this but it sounds nice to me๐Ÿ˜€
๐Ÿ‘ 1