Hi everyone, I'm trying to setup multiline log par...
# fluent-bit
f
Hi everyone, I'm trying to setup multiline log parser in fluentbit. But I'm currently encountered a problem with parsing. Can anyone help me? I'm deploying Fluent Bit in Kubernetes and pipe the logs to Loki and Grafana. But below results are minimum configuration tested locally. Problem I tried to add
cri
parser and
log
key_content in my
[MULTILINE_PARSER]
configuration, but the resulting log looked like it's not parsed by the parser. Please see below setup for more information This is my
fluent-bit.conf
Copy code
[SERVICE]
    flush        1
    log_level    info
    parsers_file parsers_multiline.conf
[INPUT]
    Name                tail
    Tag                 kube.*
    Path                test.log
    multiline.parser    multiline_atlas
[OUTPUT]
    name             stdout
    match            *
parsers_multiline.conf
Copy code
[MULTILINE_PARSER]
    name          multiline_atlas
    type          regex
    parser        cri
    key_content   log
    flush_timeout 2000
    rule          "start_state"   "/^START .+/"      "cont"
    rule          "cont"          "/^(?!^START).+/"    "cont"

# below is the config from Fluent Bit GitHub parsers.conf
[PARSER]
    # <http://rubular.com/r/tjUt3Awgg4>
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z
    Time_Keep   On
test.log
Copy code
2025-08-07T07:10:16.421971894Z stdout F START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
2025-08-07T07:10:16.422018515Z stdout F    at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151
2025-08-07T07:10:16.422022754Z stdout F --- End of stack trace from previous location ---
2025-08-07T07:10:16.422026373Z stdout F    at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl
2025-08-07T07:10:11.421029029Z stdout F START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success
I ran below command to test my input
Copy code
fluent-bit -c fluent-bit.conf
This is the Fluent Bit result from the above log. As we can see, the logs are not parsed with cri parser (there's still
2025-08-07T07:10:16.421971894Z stdout F
text in the resulting log).
Copy code
[0] kube.test.log: [[1754554269.322906687, {}], {"log"=>"2025-08-07T07:10:16.421971894Z stdout F START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array."}]
[1] kube.test.log: [[1754554269.323039756, {}], {"log"=>"2025-08-07T07:10:16.422018515Z stdout F    at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151"}]
[2] kube.test.log: [[1754554269.323044768, {}], {"log"=>"2025-08-07T07:10:16.422022754Z stdout F --- End of stack trace from previous location ---"}]
[3] kube.test.log: [[1754554269.323049988, {}], {"log"=>"2025-08-07T07:10:16.422026373Z stdout F    at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl"}]
[4] kube.test.log: [[1754554269.323053486, {}], {"log"=>"2025-08-07T07:10:11.421029029Z stdout F START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success"}]
If I change my
[INPUT] multiline.parser
to
cri
, I got the expected result.
cri
parser grouping the content of the log
Copy code
[0] kube.test.log: [[1754550616.421971894, {}], {"time"=>"2025-08-07T07:10:16.421971894Z", "stream"=>"stdout", "_p"=>"F", "log"=>"START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array."}]
[1] kube.test.log: [[1754550616.422018515, {}], {"time"=>"2025-08-07T07:10:16.422018515Z", "stream"=>"stdout", "_p"=>"F", "log"=>"   at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151"}]
[2] kube.test.log: [[1754550616.422022754, {}], {"time"=>"2025-08-07T07:10:16.422022754Z", "stream"=>"stdout", "_p"=>"F", "log"=>"--- End of stack trace from previous location ---"}]
[3] kube.test.log: [[1754550616.422026373, {}], {"time"=>"2025-08-07T07:10:16.422026373Z", "stream"=>"stdout", "_p"=>"F", "log"=>"   at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl"}]
[4] kube.test.log: [[1754550611.421029029, {}], {"time"=>"2025-08-07T07:10:11.421029029Z", "stream"=>"stdout", "_p"=>"F", "log"=>"START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success"}]
Is there anything wrong with my
[MULTILINE_PARSER]
configuration? I would like to have my logs first parsed by the
cri
parser, then get the
log
part of the logs before sending it to Loki. Basically I just want to concatenate the
log
part of the logs without the timestamp.
Copy code
START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
   at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151
--- End of stack trace from previous location ---
   at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl
Is there something I'm missing? Thank you
p
I'd use the built in CRI parser rather than define your own. Is that what you mean when you say it works? I'm sure there was an option in the Loki output to specify certain keys to send only
Yeah there are some options here to only send a single key: https://docs.fluentbit.io/manual/data-pipeline/outputs/loki
So just tail input with the built in CRI parser per the docs then configure Loki output to only send whatever you want
f
Thanks for replying. But, I want to concatenate the multi line logs to one log. For this I need to setup Fluent Bit multiline parser right? The thing is, if I use my multiline parser, then there are 2 problems 1. The logs are not concatenated 2. The
cri
parser doesn't parse the logs first before my multiline parser concatenate the logs This is my multiline parser config. I assume that the logs are first parsed using
cri
parser and then concatenated using my multiline parser regex? But this doesn't happen
Copy code
[MULTILINE_PARSER]
    name          multiline_atlas
    type          regex
    parser        cri
    key_content   log
    flush_timeout 2000
    rule          "start_state"   "/^START .+/"      "cont"
    rule          "cont"          "/^(?!^START).+/"    "cont"
p
The built-in CRI parser is multiline and handles the potential multiline created by the kubelet independently of your app, i.e.
partial
vs
full
lines. After that you can then have another
multiline
filter to combine it further.
f
I'm trying to combine two or more
full
lines directly in the
tail
plugin without having to use filters. That's why I created
multiline
parser that uses
cri
as the first parser and then concatenate the lines based on the
start_state
and the
cont
state. But that doesn't work. I'm wondering if there's anything I missed
p
Honestly I'd simplify it first then only complicate it if you really need to.