https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Team, We are facing an issue with MergeRollupTa...
# general
v
Hi Team, We are facing an issue with MergeRollupTask in our Pinot cluster. After the task runs, the original segments are not getting deleted, and we end up with both the original and the merged segments in the table. Retention properties: left as default. Any guidance on what we might be missing would be super helpful. Adding task, table and segments related details in the thread.
Copy code
{
  "state": "COMPLETED",
  "timestamp": 1756200611573,
  "segmentsFrom": [
    "test__0__1__20250725T1033Z",
    "test__3__1__20250725T1033Z",
    "test__2__1__20250725T1033Z",
    "test__1__1__20250725T1033Z",
    "test__3__2__20250729T0926Z",
    "test__0__2__20250729T0926Z",
    "test__2__2__20250729T0926Z",
    "test__1__2__20250729T0926Z"
  ],
  "segmentsTo": [
    "merged_1day_1756200600019_0_test_1753780306840_1753833429919_0",
    "merged_1day_1756200600019_0_test_1753833669122_1753867602971_1"
  ]
}
Copy code
"taskTypeConfigsMap": {
        "MergeRollupTask": {
          "schedule": "0 0/30 * * * ?",
          "1day.mergeType": "concat",
          "1day.bucketTimePeriod": "1d",
          "1day.bufferTimePeriod": "1d"
        }
      }

        "realtime.segment.flush.threshold.rows": "0",
        "realtime.segment.flush.threshold.time": "24h",
        "realtime.segment.flush.threshold.segment.size": "500M"



1. MetaData for merged segment : merged_1day_1756200600019_0_test_1753780306840_1753833429919_0

{
  "segment.start.time": "1753780306840",
  "segment.time.unit": "MILLISECONDS",
  "segment.size.in.bytes": "401877",
  "segment.end.time": "1753833429919",
  "segment.total.docs": "1713",
  "segment.creation.time": "1756200606891",
  "segment.push.time": "1756200609360",
  "segment.end.time.raw": "1753833429919",
  "segment.start.time.raw": "1753780306840",
  "segment.index.version": "v3",
  "custom.map": "{\"MergeRollupTask.mergeLevel\":\"1day\",\"MergeRollupTask.time\":\"1756200609237\"}",
  "segment.crc": "1353549389",
  "segment.realtime.status": "UPLOADED",
}

2. MetaData for original segment : test__0__1__20250725T1033Z

{
  "segment.realtime.endOffset": "8398078872",
  "segment.start.time": "1753780306840",
  "segment.time.unit": "MILLISECONDS",
  "segment.flush.threshold.size": "10000",
  "segment.realtime.startOffset": "8336899940",
  "segment.end.time": "1753780623351",
  "segment.total.docs": "5",
  "segment.realtime.numReplicas": "1",
  "segment.creation.time": "1753439584529",
  "segment.index.version": "v3",
  "segment.crc": "101520278",
  "segment.realtime.status": "DONE"
}
r
Are you also using the 
SegmentGenerationAndPushTask
task?
v
no this is a realtime table ..
for a segment we can see below logs where a segment is getting transitioned from OFFLINE -> DROPPED
Copy code
2025-08-29 14:57:33.312 INFO  [HelixTaskExecutor-message_handle_thread_4] Closed segment: test__0__13__20250809T0927Z of table: test_REALTIME
2025-08-29 14:57:33.312 INFO  [HelixTaskExecutor-message_handle_thread_4] Removed segment: test__0__13__20250809T0927Z from table: test_REALTIME
2025-08-29 14:57:33.315 INFO  [HelixTaskExecutor-message_handle_thread_4] Message 59edab39-68e0-40e8-ba39-18e0b98925c6 completed.
2025-08-29 14:57:33.315 INFO  [HelixTaskExecutor-message_handle_thread_4] Delete message 59edab39-68e0-40e8-ba39-18e0b98925c6 from zk!
2025-08-29 14:57:33.315 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] 96 START: CallbackHandler 0, INVOKE /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb type: CALLBACK
2025-08-29 14:57:33.315 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] CallbackHandler 0 subscribing changes listener to path: /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb, watchChild: false
2025-08-29 14:57:33.316 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] CallbackHandler0, Subscribing to path: /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES took: 0
2025-08-29 14:57:33.316 INFO  [HelixTaskExecutor-message_handle_thread_4] MBean CLMParticipantReport:Cluster=pinot-stage-nb6,Transition=ONLINE--OFFLINE has been registered.
2025-08-29 14:57:33.316 INFO  [HelixTaskExecutor-message_handle_thread_4] message finished: 59edab39-68e0-40e8-ba39-18e0b98925c6, took 11
2025-08-29 14:57:33.316 INFO  [HelixTaskExecutor-message_handle_thread_4] Message: 59edab39-68e0-40e8-ba39-18e0b98925c6 (parent: null) handling task for test_REALTIME:test__0__13__20250809T0927Z completed at: 1756459653316, results: true. FrameworkTime: 1 ms; HandlerTime: 10 ms.
2025-08-29 14:57:33.316 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] No Messages to process
2025-08-29 14:57:33.316 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] 96 END:INVOKE CallbackHandler 0, /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb type: CALLBACK Took: 1ms
2025-08-29 14:57:33.350 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] 96 START: CallbackHandler 0, INVOKE /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb type: CALLBACK
2025-08-29 14:57:33.350 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] CallbackHandler 0 subscribing changes listener to path: /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES, callback type: CALLBACK, event types: [NodeChildrenChanged], listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb, watchChild: false
2025-08-29 14:57:33.350 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] CallbackHandler0, Subscribing to path: /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES took: 0
2025-08-29 14:57:33.351 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] The latency of message d1813cd6-c48a-4cb1-a35e-2038e6431696 is 6 ms
2025-08-29 14:57:33.352 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] Scheduling message d1813cd6-c48a-4cb1-a35e-2038e6431696: test_REALTIME:test__0__13__20250809T0927Z, OFFLINE->DROPPED
2025-08-29 14:57:33.352 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] Submit task: d1813cd6-c48a-4cb1-a35e-2038e6431696 to pool: java.util.concurrent.ThreadPoolExecutor@233dfb73[Running, pool size = 40, active threads = 0, queued tasks = 0, completed tasks = 754]
2025-08-29 14:57:33.352 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] Message: d1813cd6-c48a-4cb1-a35e-2038e6431696 handling task scheduled
2025-08-29 14:57:33.352 INFO  [ZkClient-EventThread-96-pinot101.xxx.nb6:2181,pinot102.xxx.nb6:2181,pinot103.xxx.nb6:2181] 96 END:INVOKE CallbackHandler 0, /pinot-stage-nb6/INSTANCES/Server_pinot101.xxx.nb6_8098/MESSAGES listener: org.apache.helix.messaging.handling.HelixTaskExecutor@18aab6eb type: CALLBACK Took: 2ms
2025-08-29 14:57:33.352 INFO  [HelixTaskExecutor-message_handle_thread_25] handling task: d1813cd6-c48a-4cb1-a35e-2038e6431696 begin, at: 1756459653352
2025-08-29 14:57:33.352 INFO  [HelixTaskExecutor-message_handle_thread_25] handling message: d1813cd6-c48a-4cb1-a35e-2038e6431696 transit test_REALTIME.test__0__13__20250809T0927Z|[] from:OFFLINE to:DROPPED, relayedFrom: null
2025-08-29 14:57:33.352 INFO  [HelixTaskExecutor-message_handle_thread_25] Instance Server_pinot101.xxx.nb6_8098, partition test__0__13__20250809T0927Z received state transition from OFFLINE to DROPPED on session 30004849dd500d0, message id: d1813cd6-c48a-4cb1-a35e-2038e6431696
2025-08-29 14:57:33.352 INFO  [HelixTaskExecutor-message_handle_thread_25] SegmentOnlineOfflineStateModel.onBecomeDroppedFromOffline() : ZnRecord=d1813cd6-c48a-4cb1-a35e-2038e6431696, {CREATE_TIMESTAMP=1756459653345, ClusterEventName=MessageChange, EXECUTE_START_TIMESTAMP=1756459653352, EXE_SESSION_ID=30004849dd500d0, FROM_STATE=OFFLINE, MSG_ID=d1813cd6-c48a-4cb1-a35e-2038e6431696, MSG_STATE=read, MSG_TYPE=STATE_TRANSITION, PARTITION_NAME=test__0__13__20250809T0927Z, READ_TIMESTAMP=1756459653351, RESOURCE_NAME=test_REALTIME, RESOURCE_TAG=test_REALTIME, RETRY_COUNT=3, SRC_NAME=pinot102.xxx.nb6_9000, SRC_SESSION_ID=10004b951e40100, STATE_MODEL_DEF=SegmentOnlineOfflineStateModel, STATE_MODEL_FACTORY_NAME=DEFAULT, TGT_NAME=Server_pinot101.xxx.nb6_8098, TGT_SESSION_ID=30004849dd500d0, TO_STATE=DROPPED}{}{}, Stat=Stat {_version=0, _creationTime=1756459653349, _modifiedTime=1756459653349, _ephemeralOwner=0}
2025-08-29 14:57:33.355 INFO  [HelixTaskExecutor-message_handle_thread_25] Merging with delta list, recordId = test_REALTIME other:test_REALTIME
2025-08-29 14:57:33.356 INFO  [HelixTaskExecutor-message_handle_thread_25] Message d1813cd6-c48a-4cb1-a35e-2038e6431696 completed.
2025-08-29 14:57:33.357 INFO  [HelixTaskExecutor-message_handle_thread_25] Delete message d1813cd6-c48a-4cb1-a35e-2038e6431696 from zk!
2025-08-29 14:57:33.357 INFO  [HelixTaskExecutor-message_handle_thread_25] message finished: d1813cd6-c48a-4cb1-a35e-2038e6431696, took 5
2025-08-29 14:57:33.357 INFO  [HelixTaskExecutor-message_handle_thread_25] Message: d1813cd6-c48a-4cb1-a35e-2038e6431696 (parent: null) handling task for test_REALTIME:test__0__13__20250809T0927Z completed at: 1756459653357, results: true. FrameworkTime: 1 ms; HandlerTime: 4 ms.
r
iirc, the issue is that the merged segments are too large
v
The actual size of the merged segments is around 5 MB ...
r
in that case, you have a different issue
v
will threshold.segment.size be an issue as it is set as 500 MB in the tableConfig ...
r
no
Open in Slack
PreviousNext
Powered by