Hello we are seeing issue where replication is failing at times for few segments after a data load. Also few segments had 3 replicas instead of 2 causing it to be still in updating state. The table is set to have 2 replica. I had deleted all segments of the table using swagger and reloaded, still same issue. We are using pinot 1.4 So if we query the table based on which replica group it hits we are seeing different results of the query, which is a big issue. Its not showing any error while querying. (image attached) Can you please confirm if we update the table to use atomic data load, will it replace all the segments of the table only if replication is also successful or will it only do the replacement at a segment level without checking the replica status. Thank you. cc: @Jackie @Xiang Fu @Mayank @Johan Adami

where replication is failing at times for few segments after a data load

I’m not sure I understand this statement. Pinot itself is not actively replicating the data. Each replica is independently downloading the new segment. If it is failing to download, the server should have logs explaining why

Also few segments had 3 replicas instead of 2 causing it to be still in updating state

that sounds like either your table config is incorrect (there are really 3 replicas instead of 2) or some other process is potentially running a rebalance. I can’t think of another operation that would add a segment to a server temporarily.

Can you please confirm if we update the table to use atomic data load

I believe you’re referring to https://docs.pinot.apache.org/operators/operating-pinot/consistent-push-and-rollback. Yes, I believe this will only update segments if all of them succeed

thank you..for this just to confirm.. does it also ensure replicas are loaded for all segments or does it replace even if one replica is available? I believe you’re referring to https://docs.pinot.apache.org/operators/operating-pinot/consistent-push-and-rollback. Yes, I believe this will only update segments if all of them succeed

this works at the broker level. so it switches which segments are being queried. I’m fairly certain it will only succeed in completing the ingestion + atomically switching once all servers/replicas have gotten the new segments. We use this feature internally and have not had reports of inconsistency

thank you.. will test it out.. hopefully it resolves our data corruption issue due to replication failures

you should use an empty/new directory for each data load.

clean the output dir before your run 🙂