Hello, I'm looking for a good way to get custom metrics from my Pods in the node, to be collected by fluentbit, and then exported to prometheus outside the cluster. Since the prometheus scrape metrics input requires host and port, it doesn't really work well with pods exposing metrics. It could be exposed via the service (which works), but certain thirdparty operators (strimzi for kafka for instance), don't allow this to be exposed in the kafka broker service objects. Seeking advise on how to go about it. thanks

Why can't you make a separate service that selects those pods? Or do you need per-pod metrics? The other option is run a side car of fluent bit that just scrapes localhost and then does a remote-write or similar to Prometheus.

Why can't you make a separate service that selects those pods?

I have been. this won't work for jobs though, right? the current design was running the daemonset, and allow it to be like the pod metrics aggregator for the node

Ah ok, you didn't say a daemonset 🙂 However you can run a service that selects pods from a daemonset I think maybe... You may want to label the metrics with the node name (I typically do that via the DownwardAPI) if you do that though, otherwise just scrape the pods via something like podMonitor maybe - I'm pretty sure the chart supports pod + service monitors.

the current design was running the daemonset

oh. my bad. with the daemonset mentioned here, just to be clear, I meant the fluentbit pods are running as daemonset