Hey Team, We are seeing timeout errors in servers...
# troubleshooting
r
Hey Team, We are seeing timeout errors in servers when commiting segment and calling controller APIs. The errors logs are below:
Copy code
2025/05/11 05:34:48.702 ERROR [ServerSegmentCompletionProtocolHandler] [metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z] Could not send request http://pinot-controller-2.pinot-controller-headless.d3-pinot-cluster.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=535035644&reason=rowLimit&buildTimeMillis=5958&streamPartitionMsgOffset=1113047209&instance=Server_pinot-server-17.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098&name=metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z&rowCount=747316&memoryUsedBytes=661429872
On controller logs I am unable to find any errors but build time seems high to me.
Copy code
2025/05/11 05:34:56.303 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-16] Processing segmentCommitStart: Segment name: metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z,Instance Id: Server_pinot-server-17.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098,Reason: null,NumRows: 747316,BuildTimeMillis: 5958,WaitTimeMillis: 0,ExtraTimeSec: -1,SegmentLocation: null,MemoryUsedBytes: 661429872,SegmentSizeBytes: 535035644,StreamPartitionMsgOffset: 1113047209
Due to above issue our ingestion rate is slowing down. Any recommendations on how to fix this? Controller has high heap all the times. I have also done the changes to remove controller from ingestion path.
how many kafka partitions are committing segments at the same time, as well how many segments in your table?
we observed the commit slow issue due to too many IdealState updates and there was a fix in 1.3.0 release to use group commit for this.
r
Yes, I have decoupled controller from ingestion path. We have around 60k segments and ingestion rate peaks at 600k rps with 72 partitions. We have 20-30 segments commiting at the same time. I think we are on latest release, let me verify once.
Are you talking about this? https://github.com/apache/pinot/pull/14170
x
got it, then it’s likely the segment commit to IdealState issue. There are a few PRs https://github.com/apache/pinot/pull/13976 and https://github.com/apache/pinot/pull/14237
We observed similar issue when IdealState grows large and each segment push commit time may take 5 seconds to zookeeper, then it means each minute you can commit at most 12 segments. Those PRs make the IS commit to batch all the segments changes in one zk write.
r
Thanks for the links! Do we need any configuration to enable this?
x
No, this is improvement in 1.3
r
We have made this change, but still we are getting timeout error. Any other way to fix this?
Controller logs
Copy code
2025/05/13 14:00:09.483 INFO [PinotLLCRealtimeSegmentManager] [grizzly-http-server-31] Finished committing segment metadata for segment: metric_numerical_agg_1hour_partition__34__37097__20250513T1349Z. Time taken for updating committing segment metadata: 45ms; creating new consuming segment (metric_numerical_agg_1hour_partition__34__37098__20250513T1400Z) metadata: 24ms; updating ideal state: 8539ms; total: 8609ms
x
do you have deep store where servers all directly upload segments to ?
this one is info log, seems to be ok
can you share the error logs regarding the timeout
r
Yes we have S3 deep store.
Error logs:
Copy code
2025/05/14 02:34:59.414 ERROR [ServerSegmentCompletionProtocolHandler] [metric_numerical_agg_1hour_partition__35__36798__20250514T0231Z] Could not send request http://pinot-controller-2.pinot-controller-headless.d3-pinot-cluster.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2466593373&instance=Server_pinot-server-35.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098&name=metric_numerical_agg_1hour_partition__35__36798__20250514T0231Z&rowCount=1608178&memoryUsedBytes=1333916768
java.net.SocketTimeoutException: Read timed out
        at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[?:?]
        at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[?:?]
        at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[?:?]
        at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
        at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
        at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
        at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
        at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:331) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
        at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:196) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
x
seems http timeout for commit. Do you have controller side logs and errors
r
There are no errors on controller. Let me share logs.
x
could it be too many requests to controllers and controller http server got throttled
how many cpu you give for controller
r
Hey Xiang, apologies for delay in response. We have 12 vCPU, controller CPU reaches 500%.
These are the logs. Below is summary of observations. Summary of Observations • Segment Reaches Row Limit and Triggers Commit: • The Pinot server consumes events until it hits the configured row limit (1,396,003 rows), at which point it stops consumption and initiates a segment commit to the controller1. • Repeated HOLD Responses from Controller: • After sending the
/segmentConsumed
request to the controller, the server repeatedly receives a
{"status":"HOLD"}
response. This indicates that another server has been selected as the committer for this segment, and this server must wait for the commit to complete1. • Committer Selection and Coordination: • Controller logs confirm that two servers (Server_pinot-server-0 and Server_pinot-server-1) are assigned to the same segment. The controller notifies one server as the committer (
COMMIT
) and instructs the other(s) to `HOLD`1. • Server Experiences Communication Errors: • After several HOLD responses, the server logs errors such as
Could not send request ...
and
Invalidating cached controller leader value
. This suggests intermittent communication or leader resolution issues between the server and controller1. • NOT_SENT Status and Controller Cache Invalidation: • The server receives a
NOT_SENT
status from the controller and attempts to refresh its controller leader cache, indicating it temporarily lost track of the controller leader, possibly due to network issues or a controller failover1. • Eventually, Segment Commit Succeeds: • After the commit completes on the winning server, the controller responds with
{"status":"KEEP"}
to the waiting server. The server then transitions the segment from CONSUMING to ONLINE, downloads the committed segment from S3, and loads it successfully1. • No Evidence of Deep Store (S3) Failures: • Segment upload and download operations to S3 complete successfully, as shown by the logs indicating the segment is fetched, untarred, and loaded without error1. •
pinot server controller logs