Hi do fluent-bit internal metrics contain Kubernet...
# fluent-operatorh
Hristo Papazov
09/26/2023, 2:29 PMHi do fluent-bit internal metrics contain Kubernetes metadata, like k8s.pod.name, k8s.cluster.name, etc.? I do not see any so I am just wondering if I am missing configuration or the metadata is just not available.
l
lecaros
09/26/2023, 3:00 PMlecaros
09/26/2023, 3:01 PMyou can use the k8s filter for that
h
Hristo Papazov
09/26/2023, 3:03 PMAs far as I am aware FILTER does not work on internal metrics at the moment.
l
lecaros
09/26/2023, 3:04 PMooh, sorry I missed it’s for internal metrics
👍 1
p
Pat
09/27/2023, 2:31 PMyou can add labels to metrics though via remote write or prom exporter - however those labels would need to be from env vars passed in by DownwardAPI
Pat
09/27/2023, 2:32 PMAdding labels though is generally frowned upon as it can explode your storage for Prometheus.
What do you mean by does it contain K8S metdata, type of metric would contain that? A label on the metric or some additional metric?
h
Hristo Papazov
09/27/2023, 2:37 PMI am wondering if there is a way to enrich the internal metrics the same way I can enrich Log s with Kubernetes filter. I know filters were not supported for internal metrics in the past.
p
Pat
09/27/2023, 2:38 PMWhat do you mean by enrich though? The only real thing for Prometheus is adding labels and that is generally not recommended.
Pat
09/27/2023, 2:40 PMFor label issues - see here: https://prometheus.io/docs/practices/naming/#labels
Do you want the metric name to include the pod name or something similar? Which metric?
As it stands, metrics do not support running custom filters. Only processors to add labels or otherwise mutate them: https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/yaml/configuration-file#config_processors
h
Hristo Papazov
09/27/2023, 2:53 PMI will take a look at the document.