Dear Team, This is regarding Pinot server catchup mechanism. I am using single node setup where I have a downtime of pinot server for about 1 hour..... But during this window I have a kafka stream data where the source system keep on publishing the data in KAFKA pipe and I see around 10milion records. Once the server started, to read all the OFFSET datasets from kafka is not all working and I am not able to see the KAFKA topic data is consumed, segments created. What are the best options to improve the catch-up mechanism here. Good options are welcomed. Partition: Current Partition count is = 0, i shall be okay to go with 5 partitions as well. Configurations used so far to improve the catch-up as follows. "stream.kafka.consumer.prop.max.poll.records": "5000", "stream.kafka.consumer.prop.auto.commit.interval.ms": "1000", "stream.kafka.num.consumer.fetch.threads": "5", "realtime.segment.flush.threshold.segment.rows": "10000000" Need some good options to fix the catch-up issue. Please don't say, use cluster or move the OFFSET to LATEST and let the data shall get LOST.... Thanks

See: https://docs.pinot.apache.org/configuration-reference/ingestion#streamconfigmaps You can set

stream.[streamType].consumer.prop.auto.offset.reset

to

smallest

to let partition offset fetch to use the last offset if exists or smallest if it’s a new table or offset reset. For Pinot 1.3+ Pinot will automatically repair data pause and those offsets stuffs. For Pinot 1.2 and lower version, you need to trigger RealtimeSegmentValidationManager to fix this manually.