hello Fluent-bit community, i am using the below ...
# fluent-bit
a
hello Fluent-bit community, i am using the below filter-kubernetes.conf: filters: | [FILTER] Name kubernetes Match kube.* Kube_URL https://kubernetes.default.svc:443 Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token Kube_Tag_Prefix kube.var.log.containers. Merge_Log On Merge_Log_Trim On Keep_Log Off K8S-Logging.Parser On K8S-Logging.Exclude Off Annotations Off Labels On [FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes Add_prefix kubernetes_ [FILTER] Name nest Match kube.* Operation lift Nested_under kubernetes_labels Add_prefix kubernetes_labels_ [FILTER] Name Lua Match kube.* Call replace_periods_in_key Code function replace_periods_in_key(tag, timestamp, record) for key, val in pairs(record) do if key:find("%.") then new_key = key:gsub("%.", "_") record[new_key] = record[key] record[key] = nil end end return 2, timestamp, record end while doing so some labels are getting parsed ( e.g app.kubernetes.io/instance is being parsed as kubernetes_labels_app_kubernetes_io/instance ) and some labels are not being parsed and they are getting dead lettered as below "io.confluent.connect.elasticsearch.ElasticsearchClient$ReportingException: Indexing failed: ElasticsearchException[Elasticsearch exception [type=document_parsing_exception, reason=[1:587] failed to parse field [kubernetes_labels_app.app.kubernetes.io/name] of type [keyword] in document with id '<kafka-topic-name>+0+3729109781'. Preview of field's value: '{kubernetes={io/name=<deployment-name>}}']]; nested: ElasticsearchException[Elasticsearch exception [type=illegal_state_exception, reason=Can't get text on a START_OBJECT at 1:544]];\n"
p
This sounds like an Elasticsearch issue - "parsed" here is nothing to do with FB parsing by the sounds of it which is adding things but something ES is doing on ingestion. The ES ingestion seems to be rejecting certain types of values - is this documented as to what restrictions are present, is it something with your schema or otherwise set up? What does the actual Fluent Bit data look like? Use Fluent Bit stdout output to show you the data structure of the relevant records but I think also the ES output has some
trace_
config options that help to debug this. I always say do not debug via the endpoint (ES), do it directly in FB to see what the data actually looks like. Once you have the details on what restrictions there are and what your data actually looks like then you can get FB to "massage" it so ES accepts it. For example you already have a LUA function in place (although there is a config option in ES to do the same) that could be extended to do more.
Also, provide your actual output config - it's impossible to really help without it - and also make sure you're on the latest OSS version which is 4.0.3.
Yeah I thought there was a config option... https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch#configuration-parameters Why are you using your own LUA script to do what
Replace_Dots
does? Was it not working for some reason? I would recommend using the output option rather than make your own one because maybe you are messing up something else causing the error.
I would start off then with the following: 1. Remove LUA filter and use
replace_dots on
2. Ensure using latest OSS version 3. Enable the
trace_
options for ES to get more information 4. Debug with
stdout
then if still hitting issues