Hi team, in one of my cluster, the ingestion rate ...
# troubleshooting
r
Hi team, in one of my cluster, the ingestion rate decreases after some time of rebalancing and it comes to zero after a few time. what could be the reason? I am running realtime table only but this bug is happening a lot of time
m
We rebalance RT tables all the time, there isn’t such a bug. You probably want to dig deeper into logs on what is stopping the ingestion for partitions
r
Copy code
ster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.285 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__11__116__20250708T0330Z] Controller response {"streamPartitionMsgOffset":"99496930","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324
2025/07/08 06:04:36.380 INFO [HttpClient] [universal_cookie__16__104__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=102005105&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__16__104__20250708T0304Z&rowCount=218574&memoryUsedBytes=180087680 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.380 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__16__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"102005105","buildTimeSec":-1,"status":"HOLD","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=102005105&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__16__104__20250708T0304Z&rowCount=218574&memoryUsedBytes=180087680
2025/07/08 06:04:36.479 INFO [HttpClient] [universal_cookie__14__107__20250708T0346Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.479 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__14__107__20250708T0346Z] Controller response {"streamPartitionMsgOffset":"99529897","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955
2025/07/08 06:04:36.484 INFO [HttpClient] [universal_cookie__17__104__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.484 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__17__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"99417039","buildTimeSec":235,"controllerVipUrl":"http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000","status":"COMMIT","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552
2025/07/08 06:04:36.485 INFO [HttpClient] [universal_cookie__4__103__20250708T0304Z] Sending request: /segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=289193733&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__4__103__20250708T0304Z&rowCount=217503&memoryUsedBytes=179682548 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.485 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__4__103__20250708T0304Z] Controller response {"streamPartitionMsgOffset":"289193733","buildTimeSec":-1,"status":"HOLD","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=289193733&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__4__103__20250708T0304Z&rowCount=217503&memoryUsedBytes=179682548
2025/07/08 06:04:36.584 INFO [HttpClient] [universal_cookie__11__116__20250708T0330Z] Sending request: /segmentCommitStart?segmentSizeBytes=157937940&reason=rowLimit&buildTimeMillis=7208&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.584 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__11__116__20250708T0330Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=157937940&reason=rowLimit&buildTimeMillis=7208&streamPartitionMsgOffset=99496930&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__11__116__20250708T0330Z&rowCount=217111&memoryUsedBytes=179613324
2025/07/08 06:04:36.590 INFO [HttpClient] [universal_cookie__14__107__20250708T0346Z] Sending request: /segmentCommitStart?segmentSizeBytes=155989099&reason=rowLimit&buildTimeMillis=6499&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.590 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__14__107__20250708T0346Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=155989099&reason=rowLimit&buildTimeMillis=6499&streamPartitionMsgOffset=99529897&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__14__107__20250708T0346Z&rowCount=216490&memoryUsedBytes=179903955
2025/07/08 06:04:36.590 INFO [HttpClient] [universal_cookie__17__104__20250708T0304Z] Sending request: /segmentCommitStart?segmentSizeBytes=160407999&reason=rowLimit&buildTimeMillis=6691&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:36.590 INFO [ServerSegmentCompletionProtocolHandler] [universal_cookie__17__104__20250708T0304Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"COMMIT_CONTINUE","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=160407999&reason=rowLimit&buildTimeMillis=6691&streamPartitionMsgOffset=99417039&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__17__104__20250708T0304Z&rowCount=218962&memoryUsedBytes=266580552
2025/07/08 06:04:37.385 INFO [HttpClient] [universal_cookie__10__105__20250708T0346Z] Sending request: /segmentUpload?segmentSizeBytes=152700817&reason=rowLimit&buildTimeMillis=6946&streamPartitionMsgOffset=102152907&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__10__105__20250708T0346Z&rowCount=216550&memoryUsedBytes=179899095 to controller: pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local, version: Unknown
2025/07/08 06:04:37.385 INFO [RealtimeSegmentDataManager_universal_cookie__2__111__20250708T0350Z] [universal_cookie__10__105__20250708T0346Z] Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"status":"FAILED","isSplitCommitType":true} for http://pinot-pinot-controller-1.pinot-pinot-controller-headless.pinot-engage.svc.cluster.local:9000/segmentUpload?segmentSizeBytes=152700817&reason=rowLimit&buildTimeMillis=6946&streamPartitionMsgOffset=102152907&instance=Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098&name=universal_cookie__10__105__20250708T0346Z&rowCount=216550&memoryUsedBytes=179899095
2025/07/08 06:04:37.385 WARN [RealtimeSegmentDataManager_universal_cookie__10__105__20250708T0346Z] [universal_cookie__10__105__20250708T0346Z] Controller response was FAILED and not COMMIT_SUCCESS
2025/07/08 06:04:37.385 INFO [RealtimeSegmentDataManager_universal_cookie__10__105__20250708T0346Z] [universal_cookie__10__105__20250708T0346Z] Could not commit segment. Retrying after hold
Copy code
these are the logs that I got, repeated Could not commit segment. Retrying after hold
m
Check controller log
r
in that I could see
Copy code
Caught exception while uploading segment: universal_cookie__10__105__20250708T0346Z from instance: Server_pinot-pinot-server-2.pinot-pinot-server-headless.pinot-engage.svc.cluster.local_8098
java.io.IOException: No space left on device
After inspection inside pod, I saw that the deleted segments were taking up space and pinot couldn;t copy new segments tar files
but my question is why the deleted segments were not getting deleted from the controller?
m
Are you not using deepstore, why are segments on controller?
r
yeah not using deepstore
what other things that deepstore offers?
in managing pinot
m
If you have > 1 controllers (that you should have for HA) you need a common store shared across all controllers. This is needed for functionality - all controllers need to see the same view of back up data. It is also needed for DR (deepstore is the golden backup of the data).
r
okay cool
@Mayank why is the deleted segments getting stored in controller. Controller doesn't suppose to store any data, right?
m
Sounds like you haven’t setup deepstore? Is controller.datadir pointing to a local controller disk?
r
we haven't set it in the values.yaml when running in kubernetes. But by default does any deleted segment get stored in controller disk? we are still yet to do poc deepstore, is there any other config that that can delete the deleted segments??
also, how long does the data gets stored in the Deleted segments?
m
It is not just deleted segments, all pushed data will be on controller (as backup), if you don’t setup deepstore
r
yeah but my question is , controller shouldn't be set for data storage right? server is for that role. Or am I getting something wrong about the role of controller
m
Servers have a local copy. Servers can be recycled, scaled up / down, etc. So you also need a durable store for backup. This is deepstore, and since you didn’t configure it, it is controller
r
oh okay. How long does the data stay there?
m
You can configure
But more importantly, note that if you have more than one controllers you definitely need a shared storage for functionality