Hello Team we created offline table and segment column is in miliseconds epoch time.When we push the record we see that 1 segment is created per miliseconds. we have planned for 20 billion record ..As per my understanding 20billion segments file will created..How we can optimize.is it limitation on pinot?

I don’t follow, why is one segment created per millisecond? Is this something you configured? How are you generating the segments

we are using SQL quer "select original_event_time,state,identifier, alarm_identifier from alarm_sink" table every 15 minutes and push to pinot offline table by above jobspec file. . offline table we have below configuration "segmentsConfig": { "replication": "1", "retentionTimeUnit": "DAYS", "retentionTimeValue": "180", "minimizeDataMovement": false, "timeColumnName": "original_event_time" }, main aim is if same original_event_time and identifier received with state changes from status A to B then we will replace that in pinot past record..As pinot table does not support UPSERT on offline table only.we are doing segment replacement.How ever we are getting duplicate data of two state when we checked segment file we see that segment files created with miliseconds "segment.start.time": "1748987320076", "segment.time.unit": "MILLISECONDS", "segment.size.in.bytes": "14900", "segment.end.time": "1748987320076", "segment.total.docs": "1", "segment.creation.time": "1749124814338", "segment.push.time": "1749124814654", "segment.end.time.raw": "1748987320076", "segment.start.time.raw": "1748987320076", "segment.index.version": "v3", "custom.map": "{\"input.data.file.uri\":\"file:/tmp/input/MFAS_ONE_ROW_PER_ALARM/2025/06/03/21/1748984400.csv\"}", "segment.crc": "2873039884", "segment.partition.metadata": "{\"columnPartitionMap\":{\"operation_context\":{\"numPartitions\":64,\"functionName\":\"Murmur\",\"partitions\":[46],\"functionConfig\":null}}}", addition configuration in offline table is "tableIndexConfig": { "segmentPartitionConfig": { "columnPartitionMap": { "operation_context": { "functionName": "Murmur", "numPartitions": 64 } } }, is it the issue because of above as i see metadata segment file as "segment.partition.metadata": "{\"columnPartitionMap\":{\"operation_context\":{\"numPartitions\":64,\"functionName\":\"Murmur\",\"partitions\":[46],\"functionConfig\":null}}}", not understanding why it is not contains timestamp column

How do you know which record is in which segment? And how are you replacing segments? Also, still not clear what this has to do with one segment per millisecond

we get records from SQL query "select original_event_time,state,identifier, alarm_identifier from alarm_sink" and write to a file and using jobSpeckSummerizationTemplate.yaml that already I had share,When we push and went to table segment in UI,we are seing segments are in miliseconds.