Hey Team, we are using Flink operator and have successfully moved all our jobs from standalone kubernetes to native kubernetes. However, custom metrics seems to be broken now. We still see JVM and flink native metrics. This is how we register metrics in job code -

import io.prometheus.client.Counter;

public static final Counter invalidEventCnt = Counter.build()
      .name("invalid_event_total")
      .help("Count of invalid event")
      .labelNames("reason")
      .register();

We do not see these metrics emitted any more. More context - • With standalone kubernetes deployment, we used to copy job jar (user code) directly to

/opt/flink/lib

, now we are copying to

/opt/lib

and using this path in

jarUri

in job spec. ◦ The job code is dynamically loaded by class loader now as it is not is the classpath like it used to be with standalone deployment. Could this be messing up the metrics? • We want to use metrics from

io.prometheus.client

as we have requirement of using dynamic labels which does not seem to be supported with Flink metrics. Any suggestions on what could be causing us to lose metrics? Any troubleshooting suggestions?