Duplicate Records Issue in Pinot Segment PUSH Hello Team, I am encountering a duplicate record issue in the Pinot segment PUSH process. I am loading the same file with incremental data into an offline table, but I am receiving duplicate records. Details: Columns: state identifier event_time (this is the segment column) Issue: I am observing that each segment's file has the same epoch milliseconds: 1749007120076. I would like to know if there is a way to create hourly segments to avoid this issue. Example: First Push (record1.csv): Outstanding, 9937825, 1749007120076 Query from the Pinot table returns 1 record. Second Push (secondfile.csv): Outstanding, 919210623, 1739852763827 Outstanding, 9937825, 1748987320076 // Previous record Now, the query returns 3 records instead of the expected 2. Job Specification: Here is the job specification I am using: executionFrameworkSpec: name: 'standalone' segmentGenerationJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentGenerationJobRunner' segmentTarPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentTarPushJobRunner' segmentUriPushJobRunnerClassName: 'org.apache.pinot.plugin.ingestion.batch.standalone.SegmentUriPushJobRunner' jobType: SegmentCreationAndTarPush inputDirURI: '$input_dir_path' includeFileNamePattern: 'glob:**/*.csv' outputDirURI: '$out_dir_path' overwriteOutput: true pushJobSpec: pushFileNamePattern: 'glob:**/*.tar.gz' pushParallelism: 10 pushAttempts: 2 pushRetryIntervalMillis: 1000 pinotFSSpecs: - scheme: file className: org.apache.pinot.spi.filesystem.LocalPinotFS recordReaderSpec: dataFormat: 'csv' className: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReader' configClassName: 'org.apache.pinot.plugin.inputformat.csv.CSVRecordReaderConfig' tableSpec: tableName: '$dest_table' schemaURI: '$pinot_ctl_url/tables/$dest_table/schema' tableConfigURI: '$pinot_ctl_url/tables/$dest_table' pinotClusterSpecs: - controllerURI: '$pinot_ctl_url' Could you please help me understand why this process is producing duplicate records? Any insights or suggestions on how to resolve this issue would be greatly appreciated.