https://pinot.apache.org/ logo
#general
Title
# general
s

Suraj

02/20/2020, 10:56 PM
Hello - how do we reset the offset of a pinot table (current segment using simple consumer) to say the latest ?
x

Xiang Fu

02/20/2020, 11:13 PM
Just to clarify, the requirement is to: just update current segment consuming offset to latest? or you also want to truncate existing data?
s

Suraj

02/20/2020, 11:15 PM
yes truncating data in the kafka topic is what is needed
there is some bad formatted data in the topic that fails deserialization and are looking at ways to drain that data out...
x

Xiang Fu

02/20/2020, 11:34 PM
got it. so it’s just for changing offset for existing segments but not delete existing data in Pinot
s

Suraj

02/20/2020, 11:35 PM
yes data is still in topic
where are offsets stored ? ZK ? can we update them there ?
x

Xiang Fu

02/20/2020, 11:46 PM
also you don’t want to delete existing data in pinot table right
offset is stored in segmentMetadata under PropertyStore in Zookeeper
s

Suraj

02/20/2020, 11:53 PM
segment.realtime.startOffset
should that property be updated ?
x

Xiang Fu

02/21/2020, 12:11 AM
yes
message has been deleted
find the IN_PROGRESS segment
and update the offset
s

Suraj

02/21/2020, 12:12 AM
do we need to restart pinot ?
x

Xiang Fu

02/21/2020, 12:12 AM
then you can reload the table or restart the server
yes
s

Suraj

02/21/2020, 12:12 AM
the broker holding segment
x

Xiang Fu

02/21/2020, 12:12 AM
broker doesn’t hold segment
only pinot server consumes
s

Suraj

02/21/2020, 12:13 AM
oh I see
ok let me try that
there seem to be 22 in progress segments ...
usually how many in progress segments are expected ?
x

Xiang Fu

02/21/2020, 12:16 AM
it’s one per Kafka partition
the segment name will contain partition id
tablename_{kakfa_partition_id}_{seq_id}_ts
the max seq_id will point you to the latest current consuming segment
s

Subbu Subramaniam

02/21/2020, 12:40 AM
you are in the right track. If you know which partition you want to advance, then you can change the metadata of the latest segment of that partition, Reload will not help, you will need to restart all servers that consume that particular partition,