Apache Pinot

Hello!
We’re running *Apache Pinot 1.3.0* consuming from Kafka topics produced by *Flink 1.19 KafkaSink* with *EXACTLY_ONCE* delivery guarantee. Pinot tables use Kafka isolation level `read_committed`.
On low-volume topics (can be idle for ~30 minutes), during server restart Pinot’s startup readiness using the `FreshnessBasedConsumptionStatusChecker` gets stuck forever and never turns `GOOD`. Freshness-based check does the fallback to offset-based check, but it fails as well. The log repeats with a stable “+1” offset gap, log example in thread.
Manually consuming last hour of user messages shows last user record offset is 18895 (no record at 18896 visible to a normal consumer)

This blocks server startup readiness and cause restart loops.
It looks like it possibly caused by kafka transactional markers/control records at the tail that aren’t visible for consumer, but still advance the offset.

We fixed this issue using `realtimeFreshnessIdleTimeoutMs`, but as I understand it's not recommended to enable it in production environment.
Is there any better options to let Pinot skip transactional markers instead of enabling `realtimeFreshnessIdleTimeoutMs`?
Using `enableRealtimeOffsetBasedConsumptionStatusChecker` produced the same results

Log example from: `org.apache.pinot.server.starter.helix.FreshnessBasedConsumptionStatusChecker`

```Segment xxx__5__25__20260303T1524Z with freshness 1993405ms has not caught up within min freshness 10000. At offset 18896. Earliest offset 14662. Latest offset 18897.
Segment xxx__5__25__20260303T1524Z with freshness 2003675ms has not caught up within min freshness 10000. At offset 18896. Earliest offset 14662. Latest offset 18897.
Segment xxx__5__25__20260303T1524Z with freshness 2005995ms has not caught up within min freshness 10000. At offset 18896. Earliest offset 14662. Latest offset 18897.
Segment xxx__5__25__20260303T1524Z with freshness 2013937ms has not caught up within min freshness 10000. At offset 18896. Earliest offset 14662. Latest offset 18897.
Segment xxx__5__25__20260303T1524Z with freshness 2024211ms has not caught up within min freshness 10000. At offset 18896. Earliest offset 14662. Latest offset 18897.```

At this point, `realtimeFreshnessIdleTimeoutMs`  seems to be the only option. May be file an issue for a better handling, perhaps using Kafka's LSO in read_committed mode?

this one troubled us for quite a while.
• we have `realtimeFreshnessIdleTimeoutMs` set to 3 minutes
• <https://github.com/apache/pinot/pull/17598> should also make sure the freshness tracking always trends to lower freshness, not sawtoothing
we did a good but of research, but for transaction publishes, kafka will never give you the “latest” message because it’s a “control” message. We couldn’t come up with a better way to distinguish between
• this is the very special transaction publish with large checkpoint interval case vs
• pinot is actually behind

Thank you for the replies! I’ll create a new issue to track it and post it here

Issue: <https://github.com/apache/pinot/issues/17962>