https://pinot.apache.org/ logo
t

Tanmay Movva

02/08/2021, 12:50 PM
Hello, is there an API to reset a table when it’s offset is out of range for consumption? If not, does disabling and enabling the table trigger a offset reset?
/segments/{tableNameWithType}/reset
does this reset the consumption offset for the table?
k

Kishore G

02/08/2021, 2:33 PM
What do you mean out of range?
t

Tanmay Movva

02/08/2021, 2:34 PM
We have migrated to a new kafka cluster, but the pinot is trying to consume from the previously committed segments, which is expected. But in the new cluster, as the offsets again start from 0 we want pinot to reset and start consuming from the earliest(smallest) offset.
k

Kishore G

02/08/2021, 2:44 PM
Ah.. what about the previous data?
t

Tanmay Movva

02/08/2021, 2:45 PM
We drained out the data from previous cluster. If we start reading data from the starting offset of the new cluster then there shouldn’t be any data loss.
The only change I made to table config was for kafka bootstrap servers. It was an endpoint change.
This is what I see in the logs.
k

Kishore G

02/08/2021, 2:57 PM
Got it
Yeah, it’s behaving according to the design and as you mentioned it’s using the offset from the previous segment
Do you have hybrid table or real-time only?
t

Tanmay Movva

02/08/2021, 2:59 PM
real time only
k

Kishore G

02/08/2021, 3:00 PM
The number of partitions remain the same?
t

Tanmay Movva

02/08/2021, 3:00 PM
Yes. topic names and partitions are same as earlier.
k

Kishore G

02/08/2021, 3:02 PM
Ok, there is a way to achieve this but might require some complex sequences of steps
Can you file an issue.. I will add the steps there
t

Tanmay Movva

02/08/2021, 3:03 PM
Sure. Doing it.
k

Kishore G

02/08/2021, 3:03 PM
Thanks
Why did you change the Kafka cluster?
t

Tanmay Movva

02/08/2021, 3:04 PM
We have upgraded the kafka version.
k

Kishore G

02/08/2021, 3:08 PM
Okay.. upgrading resets the offsets?
t

Tanmay Movva

02/08/2021, 3:10 PM
Umm..not sure. But we created a new cluster with the newer version and are migrating applications to the new cluster. There are many applications still using the older versioned cluster, so had to change the endpoint and recreate the topics again in the new cluster.
This is exactly what I was looking for - https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors. I was assuming the periodic task, realtime segment validator would take care of these situations too.
k

Kishore G

02/08/2021, 3:15 PM
makes sense
@Subbu Subramaniam @Neha Pawar ^^
s

Subbu Subramaniam

02/08/2021, 7:19 PM
Disclaimer: This has never been tried before. @Tanmay Movva you probably have a bunch of segments in CONSUMING state right now. If you reset the start offset of these segments to 0 in the metadata and change the state from CONSUMING to OFFLINE, then the next time realtime segment data manager runs, it should create new segments at earliest offset available in each partition. Again, never tried before.
k

Kishore G

02/08/2021, 7:19 PM
I suggested that and it worked
s

Subbu Subramaniam

02/08/2021, 7:19 PM
ah ok nice
k

Kishore G

02/08/2021, 7:20 PM
what would have been better is to have an API around that - commit the existing segment and the new segment starts from the earliest or the latest
basically a way to override the offsets
it helps in cases where there was bad data
s

Subbu Subramaniam

02/08/2021, 7:27 PM
API is good, but we need to think through the case when there are multipole controllers. In a distributed controller system, the realtime segment repair job can run in one contoller, and the api fielded by another cntroller, causing potential race conditions. I think we need to think this through well before starting to build APIs. One way is to provide a primitive to disable all operations for a table -- automatic repair, segment completion, etc. -- do whatever we want to do, and then re-enable the operations. Again, this is just a top-level idea. Devil is in the details.
k

Kishore G

02/08/2021, 7:28 PM
yes, a global way to pause operations for a table is a good thing to have