https://pinot.apache.org/ logo
Join Slack
Powered by
Hey all, Since migrating 1.4 → 1.5 and pinot-kafk...
# troubleshooting
p
Hey all, Since migrating 1.4 → 1.5 and pinot-kafka-2.0 → pinot-kafka-3.0, realtime tables consuming from transactional Kafka topics (AWS MSK) get stuck. With 3 partitions × 3 replicas, one consuming segment per table wedges in an internal re-seek loop and never advances. - Reset segment: does not help - Pause / resume: does not help - Adding 
stream.kafka.isolation.level: read_committed
(and confirming the table picks it up): does not help — segment still gets stuck - Only workaround: delete table, set ``auto.offset.reset=smallest``, backfill. Works until the next segment rolls — then a new segment gets stuck again at a commit boundary. It only affects transactional topics. Non-transactional topics are fine. Do you have any suggestions for me?
Exact error log (repeats forever, ~2/sec)
Copy code
2026/06/26 09:17:34.313 INFO [RealtimeSegmentDataManager_mytable__1__2__20260626T0908Z] [mytable__1__2__20260626T0908Z] Consumed 0 events from (rate:0.0/s), currentOffset=14674563, numRowsConsumedSoFar=216152, numRowsIndexedSoFar=216152
2026/06/26 09:17:57.480 INFO [ClassicKafkaConsumer] [mytable__1__2__20260626T0908Z] [Consumer clientId=mytable_REALTIME-endur_mytable-1, groupId=null] Seeking to offset 14674563 for partition endur_mytable-1
2026/06/26 09:17:57.982 INFO [ClassicKafkaConsumer] [mytable__1__2__20260626T0908Z] [Consumer clientId=mytable_REALTIME-endur_mytable-1, groupId=null] Seeking to offset 14674563 for partition endur_mytable-1
(Earlier, before we set `read_committed`, we also saw — a +1 gap, same partition:)
ERROR [RealtimeSegmentDataManager_pwr_phys_fullsupply_timeseries__0__8__...] Message loss detected in stream partition: 0 for table: myanothertable_REALTIME startOffset: 79028081 batchFirstOffset: 79028082
Environment - Apache Pinot ***1.5.0***, LOWLEVEL consumers, 3 replicas, 3 partitions - Stream plugin ***pinot-kafka-3.0*** (
kafka30.KafkaConsumerFactory
) - ***AWS MSK***, SASL_SSL / SCRAM-SHA-512, ***transactional*** topics, tiered storage Sample realtime table config
Copy code
{
  "REALTIME": {
    "tableName": "mytable_REALTIME",
    "tableType": "REALTIME",
    "segmentsConfig": {
      "timeColumnName": "timestamp_col",
      "replication": "3",
      "retentionTimeUnit": "DAYS",
      "retentionTimeValue": "15",
      "replicasPerPartition": "3",
      "segmentPushType": "APPEND",
      "segmentAssignmentStrategy": "BalanceNumSegmentAssignmentStrategy",
      "minimizeDataMovement": false
    },
    "tenants": {
      "broker": "DefaultTenant",
      "server": "DefaultTenant"
    },
    "tableIndexConfig": {
      "nullHandlingEnabled": true,
      "rangeIndexVersion": 2,
      "loadMode": "MMAP",
      "columnMajorSegmentBuilderEnabled": true,
      "noDictionarySizeRatioThreshold": 0.85
    },
    "ingestionConfig": {
      "streamIngestionConfig": {
        "streamConfigMaps": [
          {
            "streamType": "kafka",
            "stream.kafka.topic.name": "mytopic",
            "stream.kafka.consumer.type": "LOWLEVEL",
            "stream.kafka.consumer.factory.class.name": "org.apache.pinot.plugin.stream.kafka30.KafkaConsumerFactory",
            "stream.kafka.consumer.prop.auto.offset.reset": "smallest",
            "stream.kafka.isolation.level": "read_committed",
            "stream.kafka.decoder.class.name": "org.apache.pinot.plugin.stream.kafka.KafkaJSONMessageDecoder",
            "stream.kafka.decoder.prop.format": "JSON",
            "sasl.mechanism": "SCRAM-SHA-512",
            "security.protocol": "SASL_SSL",
            "realtime.segment.flush.threshold.rows": "0",
            "realtime.segment.flush.threshold.segment.size": "200M",
            "realtime.segment.flush.threshold.time": "24h"
          }
        ],
        "pauselessConsumptionEnabled": false,
        "disasterRecoveryMode": "DEFAULT",
        "columnMajorSegmentBuilderEnabled": true,
        "trackFilteredMessageOffsets": false,
        "enforceConsumptionInOrder": false,
        "useIdealStateToCalculatePreviousSegment": false
      },
      "continueOnError": true,
      "rowTimeValueCheck": true,
      "segmentTimeValueCheck": false,
      "maxConsecutiveRecordFetchFailuresAllowed": 0
    },
    "isDimTable": false
  }
}
IMG_8774.png,IMG_8775.png
j
one consuming segment per table wedges in an internal re-seek loop and never advances.
Looks similar to https://github.com/apache/pinot/issues/18663 Fixed by https://github.com/apache/pinot/pull/18669
p
Thanks for the links. I think that is exactly my issue. Is there a docker image for the fix? We can't wait until 1.6 is released and this must be a critical bug for others as well since kafka20 has been removed from 1.5
j
@Xiang Fu @Mayank for the release timelines ^^
👍 1
x
The nightly image should have it
You can also cherry pick to 1.5.0 and build your own image
p
Thanks! I will check out the commit history later. I tried this tag from 4 days ago "1.6.0-SNAPSHOT-dc4e957717-20260625-21-amazoncorretto" it fixed our issue but we faced another issue with equality operators for timestamp columns The following syntax does not work anymore. It used to work in 1.5 
AND processing_time = 1761667561482
AND processing_time = fromTimestamp(fromISO8601('2025-10-28T14:06:01.482Z'));
processing_time is a datetime column of type TIMESTAMP
Copy code
{
  "name": "processing_time",
  "dataType": "TIMESTAMP",
  "format": "1:MILLISECONDS:EPOCH",
  "granularity": "1:MILLISECONDS"
}
The following syntax works: 
AND CAST(processing_time AS LONG) = 1761667561482;
AND datetrunc('millisecond', processing_time) = datetrunc('millisecond', fromTimestamp(fromISO8601('2025-10-28T16:06:01.482000Z')));
Seems like issue came from this PR github.com/apache/pinot/pull/18396 I have raised the following issue github.com/apache/pinot/issues/18881
y
@piby the workaround to add the explicit LONG / BIGINT cast to the column is a valid one, any issues with using that? https://github.com/apache/pinot/pull/18396 fixes an important performance issue when comparing a column with a literal (avoid per row casts + enable index usage). This timestamp truncation issue was an existing unrelated one that just gets surfaced in some additional cases by the change. In your issue's suggested fix, option 1 regresses the performance fix. Option 2 seems to make sense though, I'll explore that further. Thanks for reporting!
p
@Yash Mayya Thanks for the reply. Unfortunately the workaround (CAST(... AS LONG) or datetrunc('millisecond', ...)) isn't feasible for us — it would require changing every downstream service that queries Pinot. The reason we're on this snapshot in the first place is that we need the realtime-tables fix in #18669. The problem is that #18669 lands after #18396 in history, so any nightly that includes the fix we need also pulls in this TIMESTAMP truncation regression. We can't take one without the other from available images. So our only options today are: live with the truncation bug, or cherry-pick #18669 onto a pre-#18396 base and build/maintain a custom image — neither of which is ideal. We are directly using release-1.5.0-11-amazoncorretto in production It would be a critical issue for other users as well. Any TIMESTAMP column compared to an epoch-millis literal will now silently return no rows, with no error to indicate anything is wrong. Silent incorrect results on a common query pattern are easy to miss and hard to debug. It would be great if we could get the underlying precision fixed. Happy to test a build once it's available. Thanks again!
y
github.com/apache/pinot/pull/18883 should fix the issue, I'll get someone to review it soon.
thanks 1
❤️ 1
p
Thanks 🙂
We tested the change on our end and it fixed the issue. Thanks for the quick fix 🙂
👍 1
2 Views
Open in Slack
PreviousNext
Powered by