Hi team, I want to understand how the consumption flow works during a server restart after a crash or dead state . So, for my usecase each individual record is very critical and i want to have no duplicate in my pinot . As per my understanding, when a server crashes, the segment which is actively consuming is paused . And when the server gets restarted the paused segment should start reconsuming from last committed offset from ZK , this way duplicates might intrude. Is it the correct flow and if yes, are there any ways to avoid duplicates without losing any record ?.

What is committed is the offset and the statement corresponding to it. When a server restarts it will consume from previous committed offset and segment and hence there will never be duplicates

So, a new segment starts consuming from previous committed offset after the server is restarted ?..

When a server crashes there is no chance to do things like pause. Also pause implies commit if server didn’t crash, so either way no duplicates, the design takes care of all that