Rajat
07/08/2025, 5:57 AMMayank
Rajat
07/08/2025, 6:08 AMster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.285 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__11__116__20250708T0330Z] Controller response {"streamPartitionMsgOffset":"99496930","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324
2025/07/08 06:04:36.380 INFO [HttpClient] [universal_cookie__16__104__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=102005105&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__16__104__20250708T0304Z&rowCount=218574&memoryUsedBytes=180087680 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.380 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__16__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"102005105","buildTimeSec":-1,"status":"HOLD","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=102005105&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__16__104__20250708T0304Z&rowCount=218574&memoryUsedBytes=180087680
2025/07/08 06:04:36.479 INFO [HttpClient] [universal_cookie__14__107__20250708T0346Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.479 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__14__107__20250708T0346Z] Controller response {"streamPartitionMsgOffset":"99529897","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955
2025/07/08 06:04:36.484 INFO [HttpClient] [universal_cookie__17__104__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.484 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__17__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"99417039","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552
2025/07/08 06:04:36.485 INFO [HttpClient] [universal_cookie__4__103__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=289193733&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__4__103__20250708T0304Z&rowCount=217503&memoryUsedBytes=179682548 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.485 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__4__103__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"289193733","buildTimeSec":-1,"status":"HOLD","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=289193733&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__4__103__20250708T0304Z&rowCount=217503&memoryUsedBytes=179682548
2025/07/08 06:04:36.584 INFO [HttpClient] [universal_cookie__11__116__20250708T0330Z] Sending request: /segmentCommitStart?segmentSizeBytes=157937940&reason=rowLimit&buildTimeMillis=7208&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.584 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__11__116__20250708T0330Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=157937940&reason=rowLimit&buildTimeMillis=7208&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324
2025/07/08 06:04:36.590 INFO [HttpClient] [universal_cookie__14__107__20250708T0346Z] Sending request: /segmentCommitStart?segmentSizeBytes=155989099&reason=rowLimit&buildTimeMillis=6499&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.590 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__14__107__20250708T0346Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=155989099&reason=rowLimit&buildTimeMillis=6499&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955
2025/07/08 06:04:36.590 INFO [HttpClient] [universal_cookie__17__104__20250708T0304Z] Sending request: /segmentCommitStart?segmentSizeBytes=160407999&reason=rowLimit&buildTimeMillis=6691&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.590 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__17__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=160407999&reason=rowLimit&buildTimeMillis=6691&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552
2025/07/08 06:04:37.385 INFO [HttpClient] [universal_cookie__10__105__20250708T0346Z] Sending request: /segmentUpload?segmentSizeBytes=152700817&reason=rowLimit&buildTimeMillis=6946&streamPartitionMsgOffset=102152907&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__10__105__20250708T0346Z&rowCount=216550&memoryUsedBytes=179899095 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:37.385 INFO [RealtimeSegmentDataManager_universal_cookie__2__111__20250708T0350Z] [universal_cookie__10__105__20250708T0346Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"FAILED","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentUpload?segmentSizeBytes=152700817&reason=rowLimit&buildTimeMillis=6946&streamPartitionMsgOffset=102152907&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__10__105__20250708T0346Z&rowCount=216550&memoryUsedBytes=179899095
2025/07/08 06:04:37.385 WARN [RealtimeSegmentDataManager_universal_cookie__10__105__20250708T0346Z] [universal_cookie__10__105__20250708T0346Z] Controller response was FAILED and not COMMIT_SUCCESS
2025/07/08 06:04:37.385 INFO [RealtimeSegmentDataManager_universal_cookie__10__105__20250708T0346Z] [universal_cookie__10__105__20250708T0346Z] Could not commit segment. Retrying after hold
these are the logs that I got, repeated Could not commit segment. Retrying after holdMayank
Rajat
07/08/2025, 6:19 AMCaught exception while uploading segment: universal_cookie__10__105__20250708T0346Z from instance: Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098
java.io.IOException: No space left on device
After inspection inside pod, I saw that the deleted segments were taking up space and pinot couldn;t copy new segments tar filesRajat
07/08/2025, 6:19 AMMayank
Rajat
07/08/2025, 3:45 PMRajat
07/08/2025, 3:47 PMRajat
07/08/2025, 3:47 PMMayank
Rajat
07/08/2025, 3:55 PMRajat
07/18/2025, 10:12 AMMayank
Rajat
07/22/2025, 10:38 AMRajat
07/22/2025, 10:40 AMMayank
Rajat
07/22/2025, 12:45 PMMayank
Rajat
07/22/2025, 12:57 PMMayank
Mayank