Hello,
Is there a way to check the current kafka offset that a realtime table is reading on at a given point in time?
m
Mayank
05/10/2021, 1:51 PM
Not at the moment, but you can check the previous saved offset from segment Zk metadata
👍 1
p
Pedro Silva
05/10/2021, 1:52 PM
Thought as much, thank you @User
m
Mayank
05/10/2021, 2:47 PM
Could you tell us a bit more on how you wanted to use it?
p
Pedro Silva
05/10/2021, 2:49 PM
My realtime table wasn't consuming from kafka. These messages are stringified jsons which frequently change schemas or have may occasionally have some errors, I wanted to understand what was the last read offsets for each kafka partition to be able to accurately debug where Pinot stopped.
m
Mayank
05/10/2021, 2:57 PM
Got it. There are metrics emitted for consumption. You could see those going to zero when consumption would have stopped. That would give you the time-frame.
n
Neha Pawar
05/10/2021, 3:00 PM
You can use the consumingSegmentsInfo Api
p
Pedro Silva
05/10/2021, 3:00 PM
In my use-case ingestion time-frame in Pinot does not correlate kafka's message time. The kafka messages are computed in batch and sent in bulk to the pinot ingestion topic.
n
Neha Pawar
05/10/2021, 3:00 PM
You'll find it on swagger under segments section
p
Pedro Silva
05/10/2021, 3:03 PM
There seems to an issue, the request does not reach the cluster?