Hi! can i get help here with clusterfilters / clus...
# fluent-operatoro
Oron Cohen
05/15/2025, 2:40 PMHi! can i get help here with clusterfilters / clusterparsers?
j
Josh Baird
05/15/2025, 2:52 PMWhat is your question?
🙌 1
o
Oron Cohen
05/15/2025, 2:54 PMI have a fluent-bit daemonset with fluentd, which forwards to opensearch,
I try to parse a "java" format log log with clusterparser+clusterfilter,
i use a regex parser, which breaks the log into fields,
one if the fields is "logtime", which holds a timestamp.
Copy code
apiVersion: <http://fluentbit.fluent.io/v1alpha2|fluentbit.fluent.io/v1alpha2>
kind: ClusterParser
metadata:
name: regex-for-clusterfilter-nbo-job
labels:
<http://fluentbit.fluent.io/component|fluentbit.fluent.io/component>: logging
<http://fluentbit.fluent.io/enabled|fluentbit.fluent.io/enabled>: 'true'
spec:
regex:
regex: '^(?<containertime>\S+)\s+(?<stdtype>(stdout|stderr))\s+(F|P)\s+(?<logtime>\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2}\.\d*)\s+(?<level>.+)\s*---\s*\[\s*(?<thread>\S+)\]\s*(?<message>[.\s\S]+)'
timeKey: logtime
timeFormat: '%Y-%m-%d %H:%M:%S.%N'
timeKeep: falseOron Cohen
05/15/2025, 2:55 PMI can't get the "logtime" key to become the timestamp,
i do see the field within opensearch - it should be removed if the parser worked
Oron Cohen
05/15/2025, 2:55 PMi've made sure that the timeformat is correct
Oron Cohen
05/15/2025, 2:56 PMhere is the clusterfilter:
Copy code
kind: ClusterFilter
metadata:
name: nbo-job
labels:
<http://fluentbit.fluent.io/component|fluentbit.fluent.io/component>: logging
<http://fluentbit.fluent.io/enabled|fluentbit.fluent.io/enabled>: 'true'
spec:
ordinal: 999
match: nbo.job.*
filters:
- parser:
keyName: log
parser: regex-for-clusterfilter-nbo-job
reserveData: truec
Chengwei Guo
05/16/2025, 1:00 AMto verify your parser work or not, you can add stdout filter or output.
o
Oron Cohen
05/18/2025, 6:53 AMI do have an output, my log is sent parsed as expected, but "logtime" seem to not be parsed, and stays as a field within the record.
Oron Cohen
05/18/2025, 7:02 AMI have a similar setup for fluentd [ i'm migrating from a fluentd daemonset into the fluent-operator ]:
Copy code
<source>
@type tail
read_lines_limit 250
path_key tailed_path
path ...
tag ...
multiline_flush_interval 30s
<parse>
@type multiline
format_firstline /^\S+\s(stdout|stderr)\s(F|P)\s\d{4}-\d{1,2}-\d{1,2}\s\d{1,2}:\d{1,2}:\d{1,2}/
format1 /^(?<containertime>\S+)\s+(?<stdtype>(stdout|stderr))\s+(F|P)\s+(?<logtime>\d{4}-\d{1,2}-\d{1,2}\s+\d{1,2}:\d{1,2}:\d{1,2}\.\d*)\s+(?<level>.+)\s*---\s*\[\s*(?<thread>\S+)\]\s*(?<message>[.\s\S]+)/
time_key logtime
time_format %Y-%m-%d %H:%M:%S.%N
</parse>
</source>3 Views