Hey everyone I am wondering if anyone has experien...
# fluent-bit
m
Hey everyone I am wondering if anyone has experience using fluentbit with SAP Audit logs. I am trying to use fluentbit tail input on SAP Audit logs on a linux operating system but i cannot get it to work. Fluentbit does not seem to see any file appending for them or at least seems to be unable to read even with a REGEX parser. Has anyone encountered this use case?
p
It sounds like a more generic issue to be honest, What's your actual config and version? Switch to debug log level and what does that say about the files?
Also by default tail will only read new data added to the file since it was started, and needs new lines.
Does it work with other files and if so, What's the difference?
Parser does not really matter, if a parser cannot match then you'll still get the whole line as a default key
m
Hey Pat this is my current config im using
Copy code
[INPUT]
    Name tail
    Tag SAPAudit/${HOSTNAME}
    Path /usr/sap/*/D*/log/audit*
    Ignore_Older 1d
    Read_from_Head True
    Skip_Long_Lines Off
    Refresh_Interval 10
    DB ${DB}
    DB.Sync normal
    Mem_Buff_Limit 10MB

[OUPUT]
    Name s3
    Match SAPAudit*
    bucket <redacted>
    region <redacted>
    s3_key_format <redacted>
    total_file_size 1M 
    upload_timeout 1m
theres no parser currently as im testing but we still cannot get tail to read the file
it works with all other files , its specifically not working with SAP Audit logs. they are in a weird format that has null bytes and its definitely not standard log format
p
Ah right, I don't think it can read them then. I'm sure there's a limited encoding so may be that. You could potentially use exec to drive them into Fluent Bit if there is a way to actually cat them
I've no idea what a SAP log file looks like so may be worth sharing an example but it also sounds like a proprietary thing designed to mess things up 😂
😂 1
Tail essentially consumes by new lines
Sometimes you need to disable inotify as well for weirder filesystems too, there's an option in the docs for it
m
ahh see that may lie the error, the SAP audit logs append to a single line constantly
so there is no new line, but one very long line unfortunately
p
Yeah that's not supported by tail so you'll need to use something else
m
For a solution with exec , what could I maybe do? I am not familiar with that input of fluentbit
p
Exec can run a binary or script and then the output is ingested basically: https://docs.fluentbit.io/manual/pipeline/inputs/exec Sometimes crappy proprietary logs have a custom binary you can use to
tail
them which can then be invoked this way.
😂 1