So.. I've been trying to get fluent to process a m...
# fluent-bit
j
So.. I've been trying to get fluent to process a multiline tcp sys log for the better part of a whole day yesterday. I enabled tracing and removed all other parsers but the multi line but it still seemingly doesn't activate on the lines I'm trying to get it to activate on. I see this in the log though [2025/07/30 140240] [trace] [filtermultilinemultiline.0 at /src/fluent-bit/plugins/filter_multiline/ml.c:790] not processing records from the emitter <- I assume this means that it's not even processing with the multi line?
p
What's your actual config and version? Do you have some sample input and expected vs actual output? Making a local reproducer for it may be the best option - you should be able to trigger the same behaviour with other inputs that may be easier to debug than tcp socket streams
j
@Pat yes hold on let me get all of that
p
yeah, make it easy for folks to help and we will 🙂 I just do this in my spare time whilst things are "compiling" 😛
j
sample input <134> 2025/07/291525:58 Test-VPX01 0-PPE-1 : default REWRITE Message 5675605 0 : "RES: co-woots-sit-bye-imp-ssl HOSTNAME: a.a.a.com CLIENT-IP: 10.10.10.10 VIP: 10.10.10.10 : 443 TXID: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx Client Cert Serial Number: Client Cert Subject: Client Cert Issuer: Headers: HTTP/1.1 200 OK Connection: keep-alive axid: 000000000000000000000000000000 sit 2025-07-29-15 8785 Content-Type: application/json Content-Length: 52 Date: Tue, 29 Jul 2025 202558 GMT " regex
Copy code
[SERVICE]
    Log_Level          trace
    flush             1
    Multiline_Core     On
    parsers_file      parsers.conf
    parsers_file      multiline_parsers.conf
[INPUT]
    Name               TCP
    Listen             0.0.0.0
    Port               514
    Tag                netscaler
    Format             none
[FILTER]
    Name               multiline
    Match              *
    multiline.key_content  log
    multiline.parser       netscaler_rewrite_ml
    debug_flush           On
[OUTPUT]
    Name               opensearch
    Match              *
    Host               10.10.10.10
    Port               9200
    tls                On
    tls.verify         off
    HTTP_User          netscaler_ingest
    HTTP_Passwd        password
    Index              netscaler-%Y.%m.%d
    Suppress_Type_Name On
Copy code
[MULTILINE_PARSER]
    Name           netscaler_rewrite_ml
    key_content    log
    type           regex
    flush_Timeout  2000
    rule "start_state"  "<\\d+>\\s*\\d{4}\\/\\d{2}\\/\\d{2}:\\d{2}:\\d{2}:\\d{2}.*REWRITE\\s+Message.*\\\""  "cont"
    rule  "cont"   "^\\s*\"\\s*\\r?$"  "cont"
~
p
can you reproduce it with just a tail input instead of tcp?
j
let me try to do it.. I'll be brutally honest and say this whole thing is out of my wheel house and my hats are large storage, virtualization, and network. Somehow a syslog/docker hat got thrown on my head..
p
yeah I feel you
simplify is key though, you don't want to be debugging ten things at once like tcp sockets, syslog format, regex, etc.
usually I try to make a simple container that reproduces it all in one go with tail/dummy input to stdout then I can just iterate on getting that correct before integrating the rest of the stack
j
the trace shows the logs entering and such
just it says it's constantly flushing them and at the top of a large packet push to opensearch it says "[2025/07/30 140240] [trace] [filtermultilinemultiline.0 at /src/fluent-bit/plugins/filter_multiline/ml.c:790] not processing records from the emitter" <-
p
Copy code
mkdir /tmp/test
cd /tmp/test
vi fluent-bit.conf
everything else
docker run --rm -it -v $PWD:/fluent-bit/etc --network=host fluent/fluent-bit
add a stdout output and ignore opensearch for now
you need to get it right in FB first before starting involving that monstrosity
you can also do a stdout filter before your multiline filter to see the data before it goes in
usually there's some incorrect assumption in what the data actually looks like which breaks your regex
and as always, make sure you're on the latest version - 4.0.5 currently
j
yeah running 4.0.5 and there are other things in the sys log but I assumed that that I put the multi parser first and it grabs whatever it can and then passes the rest to the parsers in the line(which I removed to see if the not processing records) would go away
p
never assume, use stdout to print out the actual data you're getting
maybe the tcp stream has some other garbage in it
divide and conquer basically to figure out where the problem is, first step is to verify your input actually matches what you think it does with stdout to print it out
j
guess I need to capture it as it leaves the syslog into a file as the actual files of the system look even more different
p
just add a stdout for now without your multiline and check what that looks like
Copy code
[SERVICE]
    Log_Level          trace
    flush             1
[INPUT]
    Name               TCP
    Listen             0.0.0.0
    Port               514
    Tag                netscaler
    Format             none
[OUTPUT]
    Name stdout
    match *
j
looking at the outputs it's exactly like I was expecting
p
ok next step, write that to a file and use tail input so people can run it locally
j
damn.. I think I see it
p
What was it?
j
well still working on it but seems there are extra quotes/unprintable that I wasn't seeing
@Pat i wanted to give you an update since I didn't follow up yesterday(I ended up working on this at night sadly) turns out I had two issues.. one was formatting in the config sadly.. I think I need to switch to YAML instead of .conf because I'm clearly hitting issues with how .conf has to be formatted but I ended up copy/pasting the entire .conf from the site and then just changing the rules between the quotes and it started actually working.. Then I just needed to fix my closing.. my start rule was actually correct.. just the formatting wasn't correct and for some reason bitfluent doesn't like barf out anything about that in the logs sadly
thank you for helping me though.. using the stdout stuff I moved it to my local machine with about 15 lines of the sys log and constantly running over and over until I figured this all out
p
Glad you got it sorted mate