Is there a way or an API to get the latest offset consumed for a real-time table/segment?
12/10/2021, 4:08 PM
Under swagger Apis look for consumingSegmentsInfo API
12/10/2021, 4:40 PM
Ah sweet. The intention is to monitor if the consuming segment offset is in sync with the kafka partition offset. Does Pinot expose such a metric already?
@User alternatively, what would be your suggestion on monitoring this?
12/10/2021, 11:31 PM
this API is the only way to monitor exact offset consumed from Pinot side. Oe metric that is useful is LLC_PARTITION_CONSUMING. This is a gauge which will be 0 if the partition is not consuming for any reason. Monitoring this in a way such as “if 0 for more than 10 minutes, alert” would be good
@User just checking, we dont have any other ways to monitor lag betwee kafka latest offset ad consumer offset right?
12/10/2021, 11:46 PM
@User@User yes, I am not aware of any other existing ways to monitor.
12/11/2021, 2:58 PM
Thanks @User and @User. In my opinion offset or diff in the offset between Kafka and Pinot should be a metric that Pinot exposes. This can give users confidence in the freshness of data especially when it comes to real-time tables. LLC_PARTITION_CONSUMING doesn’t give as much insight and can be misleading there Kafka doesn’t stream messages 24/7