Is there a way or an API to get the latest offset ...
# generalm
Map
12/10/2021, 2:37 PMIs there a way or an API to get the latest offset consumed for a real-time table/segment?
n
Neha Pawar
12/10/2021, 4:08 PM
Under swagger Apis look for consumingSegmentsInfo API
m
Map
12/10/2021, 4:40 PMAh sweet. The intention is to monitor if the consuming segment offset is in sync with the kafka partition offset. Does Pinot expose such a metric already?
Map
12/10/2021, 11:26 PM@User alternatively, what would be your suggestion on monitoring this?
n
Neha Pawar
12/10/2021, 11:31 PM
this API is the only way to monitor exact offset consumed from Pinot side. Oe metric that is useful is LLC_PARTITION_CONSUMING. This is a gauge which will be 0 if the partition is not consuming for any reason. Monitoring this in a way such as “if 0 for more than 10 minutes, alert” would be good
Neha Pawar
12/10/2021, 11:32 PM
@User just checking, we dont have any other ways to monitor lag betwee kafka latest offset ad consumer offset right?
m
Mayank
12/10/2021, 11:46 PM
@User @User yes, I am not aware of any other existing ways to monitor.
m
Map
12/11/2021, 2:58 PMThanks @User and @User. In my opinion offset or diff in the offset between Kafka and Pinot should be a metric that Pinot exposes. This can give users confidence in the freshness of data especially when it comes to real-time tables. LLC_PARTITION_CONSUMING doesn’t give as much insight and can be misleading there Kafka doesn’t stream messages 24/7