https://pinot.apache.org/ logo
#general
Title
# general
p

Pedro Silva

05/05/2021, 10:31 AM
Hello, Regarding kafka-based streaming ingestion. When does pinot commit offsets to kafka? Is it after creating a segment? Can Pinot be configured to commit offsets only after a segment has been stored in deep storage to ensure no data is lost, in case segments in the server but not in deep storage are deleted?
k

Kishore G

05/05/2021, 10:40 AM
That’s the default behavior. Offsets are checkpointed only after segments are committed to deep store
p

Pedro Silva

05/05/2021, 10:41 AM
What happens then when deep store is not configured?
k

Kishore G

05/05/2021, 10:41 AM
Note, Pinot does not use Kafka for checkpointing. Pinot has its own checkpointing mechanism
It will always upload the segment to controller and controller will store it in its local directory
p

Pedro Silva

05/05/2021, 10:43 AM
Pinot segment checkpoint means controller storing segment in local directory?
If the controller local directory is corrupted or deleted is the checkpoint invalid?
I.e: Is the controller dir a single point of failure?
k

Kishore G

05/05/2021, 10:52 AM
No
p

Pedro Silva

05/05/2021, 11:15 AM
Is there a way to tell Pinot to discard/delete consuming segments and re-read events from Kafka from the last committed offset?
In the case where the consuming segment was deleted from local storage but the metadata about the segment still exists
m

Mayank

05/05/2021, 1:05 PM
Did you try to delete the segment using rest-api / swagger?
p

Pedro Silva

05/05/2021, 1:14 PM
Not yet, I will try after configuring deep store
m

Mayank

05/05/2021, 1:39 PM
Ok