raghav
05/11/2025, 5:42 AM2025/05/11 05:34:48.702 ERROR [ServerSegmentCompletionProtocolHandler] [metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z] Could not send request http://pinot-controller-2.pinot-controller-headless.d3-pinot-cluster.svc.cluster.local:9000/segmentCommitStart?segmentSizeBytes=535035644&reason=rowLimit&buildTimeMillis=5958&streamPartitionMsgOffset=1113047209&instance=Server_pinot-server-17.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098&name=metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z&rowCount=747316&memoryUsedBytes=661429872
On controller logs I am unable to find any errors but build time seems high to me.
2025/05/11 05:34:56.303 INFO [LLCSegmentCompletionHandlers] [grizzly-http-server-16] Processing segmentCommitStart: Segment name: metric_numerical_agg_1hour_partition__7__54636__20250511T0533Z,Instance Id: Server_pinot-server-17.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098,Reason: null,NumRows: 747316,BuildTimeMillis: 5958,WaitTimeMillis: 0,ExtraTimeSec: -1,SegmentLocation: null,MemoryUsedBytes: 661429872,SegmentSizeBytes: 535035644,StreamPartitionMsgOffset: 1113047209
Due to above issue our ingestion rate is slowing down. Any recommendations on how to fix this? Controller has high heap all the times. I have also done the changes to remove controller from ingestion path.Xiang Fu
Xiang Fu
Xiang Fu
raghav
05/12/2025, 2:54 PMraghav
05/12/2025, 2:56 PMXiang Fu
Xiang Fu
raghav
05/12/2025, 4:08 PMXiang Fu
raghav
05/13/2025, 2:02 PMraghav
05/13/2025, 2:07 PM2025/05/13 14:00:09.483 INFO [PinotLLCRealtimeSegmentManager] [grizzly-http-server-31] Finished committing segment metadata for segment: metric_numerical_agg_1hour_partition__34__37097__20250513T1349Z. Time taken for updating committing segment metadata: 45ms; creating new consuming segment (metric_numerical_agg_1hour_partition__34__37098__20250513T1400Z) metadata: 24ms; updating ideal state: 8539ms; total: 8609msXiang Fu
Xiang Fu
Xiang Fu
raghav
05/14/2025, 2:36 AMraghav
05/14/2025, 2:36 AM2025/05/14 02:34:59.414 ERROR [ServerSegmentCompletionProtocolHandler] [metric_numerical_agg_1hour_partition__35__36798__20250514T0231Z] Could not send request http://pinot-controller-2.pinot-controller-headless.d3-pinot-cluster.svc.cluster.local:9000/segmentConsumed?reason=rowLimit&streamPartitionMsgOffset=2466593373&instance=Server_pinot-server-35.pinot-server-headless.d3-pinot-cluster.svc.cluster.local_8098&name=metric_numerical_agg_1hour_partition__35__36798__20250514T0231Z&rowCount=1608178&memoryUsedBytes=1333916768
java.net.SocketTimeoutException: Read timed out
at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346) ~[?:?]
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796) ~[?:?]
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099) ~[?:?]
at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:149) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.hc.core5.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:250) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.hc.core5.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:56) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:331) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]
at org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:196) ~[pinot-all-1.4.0-SNAPSHOT-jar-with-dependencies.jar:1.4.0-SNAPSHOT-eb9c759344502969c80e3e9ec00fe67bd24d2965]Xiang Fu
raghav
05/14/2025, 7:31 AMXiang Fu
Xiang Fu
raghav
05/22/2025, 6:43 AMraghav
05/22/2025, 6:44 AM/segmentConsumed request to the controller, the server repeatedly receives a {"status":"HOLD"} response. This indicates that another server has been selected as the committer for this segment, and this server must wait for the commit to complete1.
• Committer Selection and Coordination:
• Controller logs confirm that two servers (Server_pinot-server-0 and Server_pinot-server-1) are assigned to the same segment. The controller notifies one server as the committer (COMMIT) and instructs the other(s) to `HOLD`1.
• Server Experiences Communication Errors:
• After several HOLD responses, the server logs errors such as Could not send request ... and Invalidating cached controller leader value. This suggests intermittent communication or leader resolution issues between the server and controller1.
• NOT_SENT Status and Controller Cache Invalidation:
• The server receives a NOT_SENT status from the controller and attempts to refresh its controller leader cache, indicating it temporarily lost track of the controller leader, possibly due to network issues or a controller failover1.
• Eventually, Segment Commit Succeeds:
• After the commit completes on the winning server, the controller responds with {"status":"KEEP"} to the waiting server. The server then transitions the segment from CONSUMING to ONLINE, downloads the committed segment from S3, and loads it successfully1.
• No Evidence of Deep Store (S3) Failures:
• Segment upload and download operations to S3 complete successfully, as shown by the logs indicating the segment is fetched, untarred, and loaded without error1.
•raghav
05/22/2025, 6:45 AM