Hello, do we know when we might have consistent data load for spark execution. https://github.com/apache/pinot/issues/12941#event-12498082012 Currently we a table that is pretty huge and sometimes it takes time for all segments to be refreshed and if a user queries the table during this time we might show incorrect results if only some segments are refreshed. cc: @Jackie

I believe that is already supported. Check the PRs linked to the issue

I dont see it working. We are using spark job to load the data. I am using below configs. I dont see the timestamp added to the segment names. As per document consistentdatapush should add a timestamp. I tried using both SegmentCreationAndMetadataPush and SegmentMetadataPush jobType. Please let me know if i am doing anything wrong.

Not sure if this is the root cause, but it might be caused by

REFRESH

ingestion type

@Priyen Patel Can you please suggest on the above issue. Is atomic data load enabled for spark ingestion https://docs.pinot.apache.org/release-1.4.0/manage-data/data-import/batch-ingestion/spark.