Hi Team We are facing an issue at production s realtime inge Apache Pinot #troubleshooting

Hi Team. We are facing an issue at production's r...

Arkoprovo Sircar

09/26/2025, 11:22 AM

Hi Team. We are facing an issue at production's realtime ingestion table. No ingestion is having. controller's up checks show down for

controller-1

It seems some issue with controller trying to commit segments to our S3 deepstore? And we had enabled table setting and server setting to commit directly from server. Can somebody please help why this has happened? We have tried restarting all controller and servers these are the logs for

Controller

Copy code

2025/09/26 05:53:56.694 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-1] Processing segmentCommitEndWithMetadata:Offset: -1,Segment name: GatewayTest__9__2651__20250922T1206Z,Instance Id: Server_pinot-server-3.pinot-server-headless.pinot-prod.svc.cluster.local_8098,Reason: rowLimit,NumRows: 64257,BuildTimeMillis: 17312,WaitTimeMillis: 0,ExtraTimeSec: -1,SegmentLocation: <s3://zpinot-prod/GatewayTest/GatewayTest__9__2651__20250922T1206Z.tmp.016a1c02-a9e6-465d-aa20-01fe6c76fe0d,MemoryUsedBytes>: 0,SegmentSizeBytes: 305005624,StreamPartitionMsgOffset: 340975150
2025/09/26 05:53:56.699 INFO [SegmentCompletionFSM_GatewayTest__9__2651__20250922T1206Z] [grizzly-http-server-1] Processing segmentCommitEnd(Server_pinot-server-3.pinot-server-headless.pinot-prod.svc.cluster.local_8098, 340975150)
2025/09/26 05:53:56.699 INFO [SegmentCompletionFSM_GatewayTest__9__2651__20250922T1206Z] [grizzly-http-server-1] Committing segment GatewayTest__9__2651__20250922T1206Z at offset 340975150 winner Server_pinot-server-3.pinot-server-headless.pinot-prod.svc.cluster.local_8098
2025/09/26 05:53:56.699 INFO [PinotLLCRealtimeSegmentManager] [grizzly-http-server-1] Committing segment file for segment: GatewayTest__9__2651__20250922T1206Z
2025/09/26 05:53:56.700 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-0] Processing segmentConsumed:Offset: -1,Segment name: FintechOnboarding__24__437__20250922T1517Z,Instance Id: Server_pinot-server-7.pinot-server-headless.pinot-prod.svc.cluster.local_8098,Reason: timeLimit,NumRows: 39,BuildTimeMillis: -1,WaitTimeMillis: -1,ExtraTimeSec: -1,SegmentLocation: null,MemoryUsedBytes: 0,SegmentSizeBytes: -1,StreamPartitionMsgOffset: 1546
2025/09/26 05:53:56.700 ERROR [SegmentCompletionFSM_GatewayTest__9__2651__20250922T1206Z] [grizzly-http-server-1] Caught exception while committing segment file for segment: GatewayTest__9__2651__20250922T1206Z
java.lang.IllegalStateException: Connection pool shut down
	at org.apache.pinot.shaded.org.apache.http.util.Asserts.check(Asserts.java:34) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.requestConnection(PoolingHttpClientConnectionManager.java:269) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$DelegatingHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:75) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.software.amazon.awssdk.http.apache.internal.conn.ClientConnectionManagerFactory$InstrumentedHttpClientConnectionManager.requestConnection(ClientConnectionManagerFactory.java:57) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:176) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[pinot-all-1.2.0-jar-with-dependencies.jar:1.2.0-cc33ac502a02e2fe830fe21e556234ee99351a7a]
	at org.apache.pinot.shaded.software.amazon.awssdk.http.apache.internal.impl.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72) ~[pinot-all-

Arkoprovo Sircar

09/26/2025, 11:31 AM

Server Logs

Copy code

2025/09/26 05:49:35.836 INFO [PinotFSSegmentUploader] [GatewayTest__1__2651__20250922T1143Z] Successfully upload segment GatewayTest__1__2651__20250922T1143Z to <s3://zpinot-prod/GatewayTest/GatewayTest__1__2651__20250922T1143Z.tmp.2056d53d-b794-4f56-bd40-460994d54966>.
2025/09/26 05:49:35.853 INFO [HttpClient] [GatewayTest__1__2651__20250922T1143Z] Sending request: <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentCommitEndWithMetadata?segmentSizeBytes=304532696&reason=rowLimit&buildTimeMillis=14228&streamPartitionMsgOffset=341061784&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&offset=-1&name=GatewayTest__1__2651__20250922T1143Z&location=s3%3A%2F%2Fzpinot-prod%2FGatewayTest%2FGatewayTest__1__2651__20250922T1143Z.tmp.2056d53d-b794-4f56-bd40-460994d54966&rowCount=63615> to controller: pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local, version: Unknown
2025/09/26 05:49:35.853 INFO [ServerSegmentCompletionProtocolHandler] [GatewayTest__1__2651__20250922T1143Z] Controller response {"status":"FAILED","isSplitCommitType":true,"streamPartitionMsgOffset":null,"buildTimeSec":-1} for <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentCommitEndWithMetadata?segmentSizeBytes=304532696&reason=rowLimit&buildTimeMillis=14228&streamPartitionMsgOffset=341061784&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&offset=-1&name=GatewayTest__1__2651__20250922T1143Z&location=s3%3A%2F%2Fzpinot-prod%2FGatewayTest%2FGatewayTest__1__2651__20250922T1143Z.tmp.2056d53d-b794-4f56-bd40-460994d54966&rowCount=63615>
2025/09/26 05:49:35.853 WARN [RealtimeSegmentDataManager_GatewayUsageAggregation__14__106__20250924T2152Z] [GatewayTest__1__2651__20250922T1143Z] CommitEnd failed with response {"status":"FAILED","isSplitCommitType":true,"streamPartitionMsgOffset":null,"buildTimeSec":-1}
2025/09/26 05:49:35.853 WARN [RealtimeSegmentDataManager_GatewayTest__1__2651__20250922T1143Z] [GatewayTest__1__2651__20250922T1143Z] Controller response was FAILED and not COMMIT_SUCCESS
2025/09/26 05:49:35.853 INFO [RealtimeSegmentDataManager_GatewayTest__1__2651__20250922T1143Z] [GatewayTest__1__2651__20250922T1143Z] Could not commit segment. Retrying after hold
2025/09/26 05:49:35.945 INFO [HttpClient] [GatewayUsageAggregation__16__105__20250922T1447Z] Sending request: <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=343033094&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&offset=-1&name=GatewayUsageAggregation__16__105__20250922T1447Z&rowCount=637958> to controller: pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local, version: Unknown
2025/09/26 05:49:35.945 INFO [ServerSegmentCompletionProtocolHandler] [GatewayUsageAggregation__16__105__20250922T1447Z] Controller response {"status":"COMMIT","isSplitCommitType":true,"controllerVipUrl":"<http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000>","streamPartitionMsgOffset":"343033094","buildTimeSec":126} for <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=343033094&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&offset=-1&name=GatewayUsageAggregation__16__105__20250922T1447Z&rowCount=637958>
2025/09/26 05:49:35.948 INFO [HttpClient] [GatewayUsageAggregation__16__105__20250922T1447Z] Sending request: <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=158477&reason=rowLimit&buildTimeMillis=95&streamPartitionMsgOffset=343033094&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&waitTimeMillis=1&offset=-1&name=GatewayUsageAggregation__16__105__20250922T1447Z&rowCount=637958> to controller: pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local, version: Unknown
2025/09/26 05:49:35.948 INFO [ServerSegmentCompletionProtocolHandler] [GatewayUsageAggregation__16__105__20250922T1447Z] Controller response {"status":"COMMIT_CONTINUE","isSplitCommitType":true,"streamPartitionMsgOffset":null,"buildTimeSec":-1} for <http://pinot-controller-1.pinot-controller-headless.pinot-prod.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=158477&reason=rowLimit&buildTimeMillis=95&streamPartitionMsgOffset=343033094&instance=Server_pinot-server-2.pinot-server-headless.pinot-prod.svc.cluster.local_8098&waitTimeMillis=1&offset=-1&name=GatewayUsageAggregation__16__105__20250922T1447Z&rowCount=637958>

Open in Slack

Previous Next