We are using the offlineSegmentDelayHours metric for monitoring if the RealtimeToOffline task is stuck, and since upgrading to 0.8.0 we see stale values for it. Prior to 0.8.0 the metrics were present only on one controller, but now they can be on multiple controllers. I've found that 0.8.0 enables Controller Resource by default, so the tables can have different controllers as leaders. We couldn't find a metric to decide which controller is the leader for a table, so we can't filter out the stale metric for alerts. IMO these metrics should be removed for the table once leadership is lost, or there should be a gauge which can be used to decide if a controller is leader for a table.

Good point. @Jack any thoughts on this?

Yeah, that sounds fair. Let me try to make the change

@Jack that may not help. I think we should have a solution around not emitting metrics for a table if the controller is not the leader for that table

We are using the JMX metrics with prometheus agent

@Kishore G That’s also true. We should also clean up metrics if the current controller is not the leader for that table. Since controller periodic tasks are run periodically, we can do the cleanup there

Thanks @Jack, mind filing an issue to capture the problem and track progress?

Sure, will file an issue

Thanks!

Here it is: https://github.com/apache/pinot/issues/7554