https://pinot.apache.org/ logo
Join Slack
Powered by
Hello, is there an API to reset a table when it’s ...
# troubleshooting
t
Hello, is there an API to reset a table when it’s offset is out of range for consumption? If not, does disabling and enabling the table trigger a offset reset?
/segments/{tableNameWithType}/reset
does this reset the consumption offset for the table?
k
What do you mean out of range?
t
We have migrated to a new kafka cluster, but the pinot is trying to consume from the previously committed segments, which is expected. But in the new cluster, as the offsets again start from 0 we want pinot to reset and start consuming from the earliest(smallest) offset.
k
Ah.. what about the previous data?
t
We drained out the data from previous cluster. If we start reading data from the starting offset of the new cluster then there shouldn’t be any data loss.
The only change I made to table config was for kafka bootstrap servers. It was an endpoint change.
This is what I see in the logs.
Untitled
Untitled
k
Got it
Yeah, it’s behaving according to the design and as you mentioned it’s using the offset from the previous segment
Do you have hybrid table or real-time only?
t
real time only
k
The number of partitions remain the same?
t
Yes. topic names and partitions are same as earlier.
k
Ok, there is a way to achieve this but might require some complex sequences of steps
Can you file an issue.. I will add the steps there
t
Sure. Doing it.
k
Thanks
Why did you change the Kafka cluster?
t
We have upgraded the kafka version.
k
Okay.. upgrading resets the offsets?
t
Umm..not sure. But we created a new cluster with the newer version and are migrating applications to the new cluster. There are many applications still using the older versioned cluster, so had to change the endpoint and recreate the topics again in the new cluster.
This is exactly what I was looking for - https://druid.apache.org/docs/latest/development/extensions-core/kafka-ingestion.html#resetting-supervisors. I was assuming the periodic task, realtime segment validator would take care of these situations too.
k
makes sense
@Subbu Subramaniam @Neha Pawar ^^
s
Disclaimer: This has never been tried before. @Tanmay Movva you probably have a bunch of segments in CONSUMING state right now. If you reset the start offset of these segments to 0 in the metadata and change the state from CONSUMING to OFFLINE, then the next time realtime segment data manager runs, it should create new segments at earliest offset available in each partition. Again, never tried before.
k
I suggested that and it worked
s
ah ok nice
k
what would have been better is to have an API around that - commit the existing segment and the new segment starts from the earliest or the latest
basically a way to override the offsets
it helps in cases where there was bad data
s
API is good, but we need to think through the case when there are multipole controllers. In a distributed controller system, the realtime segment repair job can run in one contoller, and the api fielded by another cntroller, causing potential race conditions. I think we need to think this through well before starting to build APIs. One way is to provide a primitive to disable all operations for a table -- automatic repair, segment completion, etc. -- do whatever we want to do, and then re-enable the operations. Again, this is just a top-level idea. Devil is in the details.
k
yes, a global way to pause operations for a table is a good thing to have
Open in Slack
PreviousNext
Powered by