https://pinot.apache.org/ logo
Join Slack
Powered by
Hi Team, We initially had 3 controller nodes and r...
# troubleshooting
v
Hi Team, We initially had 3 controller nodes and recently added 3 more. Following this, we stopped the controller service on the 3 old nodes. However, after about a week, the ingestion process started failing because the Pinot server was attempting to connect to one of the stopped controller nodes. Interestingly, we did not encounter this issue in another cluster where the same approach was implemented. While I understand that we need to remove the stopped instances from Zookeeper using the Swagger API, ideally, the Pinot server should fail over and connect to an active/leader controller automatically. Am I missing something ?
Copy code
2025-11-12 16:11:54.373 ERROR [sample__121__287__20251105T1324Z] LLRealtimeSegmentDataManager_sample__66__339__20251105T1654Z - Could not send request <http://pinot-controller-01.local:9000/segmentUpload?segmentSizeBytes=1073117101&buildTimeMillis=91591&streamPartitionMsgOffset=10967889843&instance=pinot-server-01.local_8098&offset=-1&name=sample__121__287__20251105T1324Z&rowCount=73431535&memoryUsedBytes=2015467107>
org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to pinot-controller-01.local:9000 [pinot-controller-01.local/10.10.10.56] failed: Connection refused (Connection refused)
  at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
  at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
x
interesting, can you check from the controller, which controller is the leader for this table ?
technically those should already be moved
v
Even I had the same thought but the leader has been moved to newer node
x
can you try to pause then unpause ingestion
v
We haven't tried that but from the logs I presume that the the server's have cached the controller information and post server restart I don't see the same connection error.
Copy code
ControllerLeaderLocator - Millis since last controller cache value invalidate 14222 is less than allowed frequency 30000. Skipping invalidate.
------
ControllerLeaderLocator - Invalidating cached controller leader value
@Xiang Fu rolling restart did not help. I am still seeing the connection refused error.
Hi @Xiang Fu We tried pause and unpause of consumption but the issue still persists. I also removed the controller entry using drop but the server is still looking for old controller which is in stopped state and removed from ZK. Appreciate your inputs
x
can you provide more stacktrace from the server side ?
v
@Xiang Fu We also tried following approaches and we ran out of options. • Paused consumption of all the tables. • Stopped old controller pinot_101.example • Removed the controller from Zk • Performed rolling restart of Controller's, Servers and brokers • Resumed the consumption. • The logs were fine for sometime but we are encountering the same issue again. Here is the complete stacktrace
Copy code
2025-11-27 12:57:38.741 INFO  [sample__48__386__20251126T1110Z] ControllerLeaderLocator - Millis since last controller cache value invalidate 21715 is less than allowed frequency 30000. Skipping invalidate.
2025-11-27 12:57:38.741 WARN  [sample__48__386__20251126T1110Z] LLRealtimeSegmentDataManager_sample__48__386__20251126T1110Z - Controller response was FAILED and not COMMIT_SUCCESS
2025-11-27 12:57:38.741 INFO  [sample__48__386__20251126T1110Z] LLRealtimeSegmentDataManager_sample__48__386__20251126T1110Z - Could not commit segment. Retrying after hold
2025-11-27 12:57:38.793 INFO  [sample__78__349__20251126T1110Z] HttpClient - Sending request: <http://pinot_002.example:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=15519012948&instance=Server_pinot107.example_8098&offset=-1&name=sample__78__349__20251126T1110Z&rowCount=72111655&memoryUsedBytes=2058896507> to controller: pinot_002.example, version: Unknown
2025-11-27 12:57:38.793 INFO  [sample__78__349__20251126T1110Z] ServerSegmentCompletionProtocolHandler - Controller response {"streamPartitionMsgOffset":"15519012948","buildTimeSec":555,"isSplitCommitType":true,"controllerVipUrl":"<http://pinot_101.example:9000>","status":"COMMIT","offset":15519012948} for <http://pinot_002.example:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=15519012948&instance=Server_pinot107.example_8098&offset=-1&name=sample__78__349__20251126T1110Z&rowCount=72111655&memoryUsedBytes=2058896507>
2025-11-27 12:57:38.796 INFO  [sample__78__349__20251126T1110Z] HttpClient - Sending request: <http://pinot_002.example:9000/segmentCommitStart?segmentSizeBytes=1089028855&buildTimeMillis=88766&streamPartitionMsgOffset=15519012948&instance=Server_pinot107.example_8098&waitTimeMillis=355019&offset=-1&name=sample__78__349__20251126T1110Z&rowCount=72111655&memoryUsedBytes=2058896507> to controller: pinot_002.example, version: Unknown
2025-11-27 12:57:38.796 INFO  [sample__78__349__20251126T1110Z] ServerSegmentCompletionProtocolHandler - Controller response {"streamPartitionMsgOffset":null,"buildTimeSec":-1,"isSplitCommitType":false,"status":"COMMIT_CONTINUE","offset":-1} for <http://pinot_002.example:9000/segmentCommitStart?segmentSizeBytes=1089028855&buildTimeMillis=88766&streamPartitionMsgOffset=15519012948&instance=Server_pinot107.example_8098&waitTimeMillis=355019&offset=-1&name=sample__78__349__20251126T1110Z&rowCount=72111655&memoryUsedBytes=2058896507>
2025-11-27 12:57:38.796 ERROR [sample__78__349__20251126T1110Z] LLRealtimeSegmentDataManager_sample__98__359__20251126T1110Z - Could not send request <http://pinot_101.example:9000/segmentUpload?segmentSizeBytes=1089028855&buildTimeMillis=88766&streamPartitionMsgOffset=15519012948&instance=Server_pinot107.example_8098&waitTimeMillis=355019&offset=-1&name=sample__78__349__20251126T1110Z&rowCount=72111655&memoryUsedBytes=2058896507>
org.apache.pinot.shaded.org.apache.http.conn.HttpHostConnectException: Connect to pinot_101.example:9000 [pinot_101.example/10.106.152.64] failed: Connection refused (Connection refused)
        at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.examplemon.utils.http.HttpClient.sendRequest(HttpClient.java:282) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.examplemon.utils.FileUploadDownloadClient.uploadSegment(FileUploadDownloadClient.java:670) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.Server2ControllerSegmentUploader.uploadSegmentToController(Server2ControllerSegmentUploader.java:95) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.Server2ControllerSegmentUploader.uploadSegment(Server2ControllerSegmentUploader.java:72) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.Server2ControllerSegmentUploader.uploadSegment(Server2ControllerSegmentUploader.java:67) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.SplitSegmentCommitter.uploadSegment(SplitSegmentCommitter.java:95) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.SplitSegmentCommitter.examplemit(SplitSegmentCommitter.java:76) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.examplemit(LLRealtimeSegmentDataManager.java:1067) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.examplemitSegment(LLRealtimeSegmentDataManager.java:1035) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:760) [pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.ConnectException: Connection refused (Connection refused)
        at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:?]
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) ~[?:?]
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) ~[?:?]
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) ~[?:?]
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:?]
        at java.net.Socket.connect(Socket.java:609) ~[?:?]
        at org.apache.pinot.shaded.org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        at org.apache.pinot.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[pinot-all-1.0.0-jar-with-dependencies.jar:1.0.0-b6bdf6c9686b286a149d2d1aea4a385ee98f3e79]
        ... 20 more
2025-11-27 12:57:38.796 INFO  [sample__78__349__20251126T1110Z] ControllerLeaderLocator - Millis since last controller cache value invalidate 21770 is less than allowed frequency 30000. Skipping invalidate.
2025-11-27 12:57:38.796 WARN  [sample__78__349__20251126T1110Z] LLRealtimeSegmentDataManager_sample__78__349__20251126T1110Z - Controller response was FAILED and not COMMIT_SUCCESS
2025-11-27 12:57:38.796 INFO  [sample__78__349__20251126T1110Z] LLRealtimeSegmentDataManager_sample__78__349__20251126T1110Z - Could not commit segment. Retrying after hold
2025-11-27 12:57:38.884 INFO  [sample__113__314__20251126T1110Z] HttpClient - Sending request: <http://pinot_002.example:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=12882574995&instance=Server_pinot107.example_8098&offset=-1&name=sample__113__314__20251126T1110Z&rowCount=71701378&memoryUsedBytes=2058896339> to controller: pinot_002.example, version: Unknown
2025-11-27 12:57:38.884 INFO  [sample__113__314__20251126T1110Z] ServerSegmentCompletionProtocolHandler - Controller response {"streamPartitionMsgOffset":"12882574995","buildTimeSec":555,"isSplitCommitType":true,"controllerVipUrl":"<http://pinot_101.example:9000>","status":"COMMIT","offset":12882574995} for <http://pinot_002.example:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=12882574995&instance=Server_pinot107.example_8098&offset=-1&name=sample__113__314__20251126T1110Z&rowCount=71701378&memoryUsedBytes=2058896339>
x
how long for you to be ok then failed? has a new segment ever succcessfully committed ? I saw this in the stacktrac,eis the ip correct? 
pinot_101.example/10.106.152.64] failed: Connection refused (Connection refused)
Open in Slack
PreviousNext
Powered by