Hi, I am trying to fetch flink monitoring metrics in prometheus ( mostly all

flink_*

metrics ). Following this doc. Followed all steps but i am not able to see the metrics in prometheus. Is this the correct method ? Am i missing something here ? Really appreciate the help. Please share how we can achieve this. We are running with flink version 1.16.2

Yes, im monitoring flink with prometheus. Can you share your reporter settings?

Hi @Mikhail Spirin, thanks. The setting is working with pushgateway configs. I am assuming this is because of the short lived jobs. We are also trying to setup alerting, if you can share any doc with important alerts we can add or any best practices on flink alerting please help.

Important ones are described here - https://flink.apache.org/2019/02/21/monitoring-apache-flink-applications-101/ Im not aware aware of any others which are interesting enough…

Thanks. Also, @Mikhail Spirin is there a way we can inject dynamic labels in flink metrics ( preferably on sessioncluster and task_manager ) ? Usecase is we want to route different taskmanager alerts to different teams in our org for which we need dynamic labels to be added to prometheus metrics.

im not aware of ways how to do so. Me personally, when i wanted metrics for specific operators, i used naming for datastreaming operators (maps specifically), and then based on those names set up prometheus/grafana, may be that can help

Got it. Thanks a lot for the help!