Hi. Pinot team. I'm receiving a "Message loss detected" error on a server in a Pino 1.2 cluster. Does this error mean that the segment failed to start consuming from startOffset and instead started consuming from batchFirstOffset? Does this mean data loss occurred between startOffset: 769410318 and batchFirstOffset: 769525930? However, when I query the data corresponding to the offset in Pino, the data corresponding to the offset was not lost. What is this error log? 2025/09/12 150935.996 ERROR [debug_tiara_core__1__1__20250912T0608Z] org.apache.pinot.core.data.manager.realtime.RealtimeSegmentDataManager_debug_tiara_core__1__1__20250912T0608ZreportDataLoss958 Message loss detected in stream partition: 1 for table: debug_tiara_core_REALTIME startOffset: 769410318 batchFirstOffset: 769525930 select $segmentName, * from debug_tiara_core where __metadata$partition='1' and __metadata$offset >= '769410318' and __metadata$offset <= '769525930 ' -- Data exists, no loss. -- debug_tiara_core__1__1__20250912T0608Z { "segment.realtime.startOffset": "769410318", "segment.realtime.endOffset": "769571100", "segment.creation.time": "1757657338493", "segment.start.time": "1757635200000", "segment.end.time": "1757635200000", "segment.total.docs": "150,000", }

This message just means that Pinot server consumer tried to fetch messageBatch from upstream starting from the

startOffset

but in the response received from upstream, the first message offset (

batchFirstOffset

) is greater than the

startOffset

@harnoor There are two replicas. The

batchFirstOffset

value is slightly different, but the same error occurs in the replica segments within the same segment. Even though error logs appear in all replicas within the segment, there is no data loss. When I query

__metadata$offset

, I see the

offset

that was mentioned as

loss

in the log. This log is being written to about a thousand times per second. Why is this error occurring even though there is no data loss?

That's interesting. if no isolation config is set, then Pinot consumer will read all messages. Does your pinot version includes this log fix: https://github.com/apache/pinot/pull/16030/files The message loss log line you pointed out had a bug which we fixed in June.