https://pinot.apache.org/ logo
Join Slack
Powered by
Hello - how do we reset the offset of a pinot tabl...
# general
s
Hello - how do we reset the offset of a pinot table (current segment using simple consumer) to say the latest ?
x
Just to clarify, the requirement is to: just update current segment consuming offset to latest? or you also want to truncate existing data?
s
yes truncating data in the kafka topic is what is needed
there is some bad formatted data in the topic that fails deserialization and are looking at ways to drain that data out...
x
got it. so it’s just for changing offset for existing segments but not delete existing data in Pinot
s
yes data is still in topic
where are offsets stored ? ZK ? can we update them there ?
x
also you don’t want to delete existing data in pinot table right
offset is stored in segmentMetadata under PropertyStore in Zookeeper
s
segment.realtime.startOffset
should that property be updated ?
x
yes
message has been deleted
find the IN_PROGRESS segment
and update the offset
s
do we need to restart pinot ?
x
then you can reload the table or restart the server
yes
s
the broker holding segment
x
broker doesn’t hold segment
only pinot server consumes
s
oh I see
ok let me try that
there seem to be 22 in progress segments ...
usually how many in progress segments are expected ?
x
it’s one per Kafka partition
the segment name will contain partition id
tablename_{kakfa_partition_id}_{seq_id}_ts
the max seq_id will point you to the latest current consuming segment
s
you are in the right track. If you know which partition you want to advance, then you can change the metadata of the latest segment of that partition, Reload will not help, you will need to restart all servers that consume that particular partition,
Open in Slack
PreviousNext
Powered by