fftrf
08/07/2025, 8:23 AMcri parser and log key_content in my [MULTILINE_PARSER] configuration, but the resulting log looked like it's not parsed by the parser. Please see below setup for more information
This is my fluent-bit.conf
[SERVICE]
flush 1
log_level info
parsers_file parsers_multiline.conf
[INPUT]
Name tail
Tag kube.*
Path test.log
multiline.parser multiline_atlas
[OUTPUT]
name stdout
match *
parsers_multiline.conf
[MULTILINE_PARSER]
name multiline_atlas
type regex
parser cri
key_content log
flush_timeout 2000
rule "start_state" "/^START .+/" "cont"
rule "cont" "/^(?!^START).+/" "cont"
# below is the config from Fluent Bit GitHub parsers.conf
[PARSER]
# <http://rubular.com/r/tjUt3Awgg4>
Name cri
Format regex
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
Time_Keep On
test.log
2025-08-07T07:10:16.421971894Z stdout F START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
2025-08-07T07:10:16.422018515Z stdout F at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151
2025-08-07T07:10:16.422022754Z stdout F --- End of stack trace from previous location ---
2025-08-07T07:10:16.422026373Z stdout F at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl
2025-08-07T07:10:11.421029029Z stdout F START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success
I ran below command to test my input
fluent-bit -c fluent-bit.conf
This is the Fluent Bit result from the above log.
As we can see, the logs are not parsed with cri parser (there's still 2025-08-07T07:10:16.421971894Z stdout F text in the resulting log).
[0] kube.test.log: [[1754554269.322906687, {}], {"log"=>"2025-08-07T07:10:16.421971894Z stdout F START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array."}]
[1] kube.test.log: [[1754554269.323039756, {}], {"log"=>"2025-08-07T07:10:16.422018515Z stdout F at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151"}]
[2] kube.test.log: [[1754554269.323044768, {}], {"log"=>"2025-08-07T07:10:16.422022754Z stdout F --- End of stack trace from previous location ---"}]
[3] kube.test.log: [[1754554269.323049988, {}], {"log"=>"2025-08-07T07:10:16.422026373Z stdout F at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl"}]
[4] kube.test.log: [[1754554269.323053486, {}], {"log"=>"2025-08-07T07:10:11.421029029Z stdout F START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success"}]
If I change my [INPUT] multiline.parser to cri , I got the expected result. cri parser grouping the content of the log
[0] kube.test.log: [[1754550616.421971894, {}], {"time"=>"2025-08-07T07:10:16.421971894Z", "stream"=>"stdout", "_p"=>"F", "log"=>"START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array."}]
[1] kube.test.log: [[1754550616.422018515, {}], {"time"=>"2025-08-07T07:10:16.422018515Z", "stream"=>"stdout", "_p"=>"F", "log"=>" at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151"}]
[2] kube.test.log: [[1754550616.422022754, {}], {"time"=>"2025-08-07T07:10:16.422022754Z", "stream"=>"stdout", "_p"=>"F", "log"=>"--- End of stack trace from previous location ---"}]
[3] kube.test.log: [[1754550616.422026373, {}], {"time"=>"2025-08-07T07:10:16.422026373Z", "stream"=>"stdout", "_p"=>"F", "log"=>" at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl"}]
[4] kube.test.log: [[1754550611.421029029, {}], {"time"=>"2025-08-07T07:10:11.421029029Z", "stream"=>"stdout", "_p"=>"F", "log"=>"START INFO 2025-07-18 08:47:18 [Test Log String Scope] Log String Success"}]
Is there anything wrong with my [MULTILINE_PARSER] configuration? I would like to have my logs first parsed by the cri parser, then get the log part of the logs before sending it to Loki. Basically I just want to concatenate the log part of the logs without the timestamp.
START ERROR 2025-07-18 08:47:39 Exception: System.IndexOutOfRangeException: Index was outside the bounds of the array.
at Test.Module.Service.Test.LogServiceTest.<>c.<<StartAsync_ShouldReturnFailure_WhenExceptionIsThrown>b__6_0>d.MoveNext() in /home/abcdef/core-module/module-test/Test.Module.Service.Test/LogServiceTest.cs:line151
--- End of stack trace from previous location ---
at Test.Module.Service.Implements.LogServiceBase.StartAsync(String logAction, Func1 func, String scope, Action onEndHook) in /home/abcdef/core-module/module/Test.Module.Service/ServiceImpl
Is there something I'm missing?
Thank youPat
08/07/2025, 10:31 AMPat
08/07/2025, 10:32 AMPat
08/07/2025, 10:33 AMfftrf
08/08/2025, 1:40 AMcri parser doesn't parse the logs first before my multiline parser concatenate the logs
This is my multiline parser config. I assume that the logs are first parsed using cri parser and then concatenated using my multiline parser regex? But this doesn't happen
[MULTILINE_PARSER]
name multiline_atlas
type regex
parser cri
key_content log
flush_timeout 2000
rule "start_state" "/^START .+/" "cont"
rule "cont" "/^(?!^START).+/" "cont"Pat
08/09/2025, 1:53 PMpartial vs full lines. After that you can then have another multiline filter to combine it further.fftrf
08/11/2025, 2:45 AMfull lines directly in the tail plugin without having to use filters. That's why I created multiline parser that uses cri as the first parser and then concatenate the lines based on the start_state and the cont state. But that doesn't work. I'm wondering if there's anything I missedPat
08/13/2025, 6:48 AM