Hello,
Regarding kafka-based streaming ingestion. ...
# generalp
Pedro Silva
05/05/2021, 10:31 AMHello,
Regarding kafka-based streaming ingestion. When does pinot commit offsets to kafka? Is it after creating a segment? Can Pinot be configured to commit offsets only after a segment has been stored in deep storage to ensure no data is lost, in case segments in the server but not in deep storage are deleted?
k
Kishore G
05/05/2021, 10:40 AM
That’s the default behavior. Offsets are checkpointed only after segments are committed to deep store
p
Pedro Silva
05/05/2021, 10:41 AMWhat happens then when deep store is not configured?
k
Kishore G
05/05/2021, 10:41 AM
Note, Pinot does not use Kafka for checkpointing. Pinot has its own checkpointing mechanism
Kishore G
05/05/2021, 10:42 AM
It will always upload the segment to controller and controller will store it in its local directory
p
Pedro Silva
05/05/2021, 10:43 AMPinot segment checkpoint means controller storing segment in local directory?
Pedro Silva
05/05/2021, 10:43 AMIf the controller local directory is corrupted or deleted is the checkpoint invalid?
Pedro Silva
05/05/2021, 10:43 AMI.e: Is the controller dir a single point of failure?
k
Kishore G
05/05/2021, 10:52 AM
No
Kishore G
05/05/2021, 10:53 AM
p
Pedro Silva
05/05/2021, 11:15 AMIs there a way to tell Pinot to discard/delete consuming segments and re-read events from Kafka from the last committed offset?
Pedro Silva
05/05/2021, 11:15 AMIn the case where the consuming segment was deleted from local storage but the metadata about the segment still exists
m
Mayank
05/05/2021, 1:05 PM
Did you try to delete the segment using rest-api / swagger?
p
Pedro Silva
05/05/2021, 1:14 PMNot yet, I will try after configuring deep store
m
Mayank
05/05/2021, 1:39 PM
Ok