Hello, quick question
We utilize a HA setup in Flink distributing our tasks to multiple K8s pods. The problem is we want to set up additional observability on each pod beyond the builtin Flink metrics. So we need to
A. Initialize static objects
B. Startup local servers so they can be scraped
However since Java operators are serialized and distributed I don't think we can just create these objects on task initialization. Is there a recommended way to handle this?
m
M C (Mac)
09/02/2023, 3:32 AM
Have you tried adding custom metrics https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/ops/metrics/
?
In the past we would use the operators open function to do “init” things including (more for pocs) starting “embeded” server threads among other things. Not sure if that was recommended though haha.
m
Matthew Kerian
09/05/2023, 5:18 PM
Yeah, the problem then is we're forced to use the
Rich
functions for all metrics which isn't sufficient for us.