Hello, Is there a way to check the current kafka o...
# general
p
Hello, Is there a way to check the current kafka offset that a realtime table is reading on at a given point in time?
m
Not at the moment, but you can check the previous saved offset from segment Zk metadata
👍 1
p
Thought as much, thank you @User
m
Could you tell us a bit more on how you wanted to use it?
p
My realtime table wasn't consuming from kafka. These messages are stringified jsons which frequently change schemas or have may occasionally have some errors, I wanted to understand what was the last read offsets for each kafka partition to be able to accurately debug where Pinot stopped.
m
Got it. There are metrics emitted for consumption. You could see those going to zero when consumption would have stopped. That would give you the time-frame.
n
You can use the consumingSegmentsInfo Api
p
In my use-case ingestion time-frame in Pinot does not correlate kafka's message time. The kafka messages are computed in batch and sent in bulk to the pinot ingestion topic.
n
You'll find it on swagger under segments section
p
There seems to an issue, the request does not reach the cluster?
n
On the top left hand corner change http to https
p
Brilliant! Thank you very much Neha!
🙂 1