Hi Team we are facing one issue with realtime consumption fr

Question

Hi Team we are facing one issue with realtime consumption from kafka We have 4 pinot servers 7 real time tables Every few hours one of the pinot server stops consuming from one kafka topic In logs server unable to connect to controller while finalising the segment However controller is up and running Should pinot server retry for such recoverable errors Are there any config levers retries on error etc to fix this Any thoughts

Xiang Fu · Answer

so it s not connecting to controller do you have stacktrace for the controller call error

Ravi Singal · Answer

```2021 02 01 17 34 58 141 ERROR ServerSegmentCompletionProtocolHandler spanEventView 24 625 20210201T1714Z Could not send request <http pinot controller 1 pinot controller traceable svc cluster local 9000 segmentConsumed name=spanEventView 24 625 20210201T1714Z amp offset=19967548 amp instance=Server pinot server 0 pinot server traceable svc cluster local 8098 amp reason=rowLimit amp memoryUsedBytes=50846737 amp rowCount=31250 amp streamPartitionMsgOffset=19967548> java net SocketTimeoutException Read timed out at java net SocketInputStream socketRead0 Native Method ~ at java net SocketInputStream socketRead SocketInputStream java 115 ~ at java net SocketInputStream read SocketInputStream java 168 ~ at java net SocketInputStream read SocketInputStream java 140 ~ at shaded org apache http impl io SessionInputBufferImpl streamRead SessionInputBufferImpl java 137 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl io SessionInputBufferImpl fillBuffer SessionInputBufferImpl java 153 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl io SessionInputBufferImpl readLine SessionInputBufferImpl java 282 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl conn DefaultHttpResponseParser parseHead DefaultHttpResponseParser java 138 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl conn DefaultHttpResponseParser parseHead DefaultHttpResponseParser java 56 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl io AbstractMessageParser parse AbstractMessageParser java 259 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl DefaultBHttpClientConnection receiveResponseHeader DefaultBHttpClientConnection java 163 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl conn CPoolProxy receiveResponseHeader CPoolProxy java 157 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http protocol HttpRequestExecutor doReceiveResponse HttpRequestExecutor java 273 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http protocol HttpRequestExecutor execute HttpRequestExecutor java 125 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl execchain MainClientExec execute MainClientExec java 272 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl execchain ProtocolExec execute ProtocolExec java 186 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl execchain RetryExec execute RetryExec java 89 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl execchain RedirectExec execute RedirectExec java 110 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl client InternalHttpClient doExecute InternalHttpClient java 185 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 83 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at shaded org apache http impl client CloseableHttpClient execute CloseableHttpClient java 108 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot common utils FileUploadDownloadClient sendRequest FileUploadDownloadClient java 357 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot common utils FileUploadDownloadClient sendSegmentCompletionProtocolRequest FileUploadDownloadClient java 631 ~ pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot server realtime ServerSegmentCompletionProtocolHandler sendRequest ServerSegmentCompletionProtocolHandler java 207 pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot server realtime ServerSegmentCompletionProtocolHandler segmentConsumed ServerSegmentCompletionProtocolHandler java 174 pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot core data manager realtime LLRealtimeSegmentDataManager postSegmentConsumedMsg LLRealtimeSegmentDataManager java 930 pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at org apache pinot core data manager realtime LLRealtimeSegmentDataManager$PartitionConsumer run LLRealtimeSegmentDataManager java 557 pinot all 0 6 0 jar with dependencies jar 0 6 0 bb646baceafcd9b849a1ecdec7a11203c7027e21 at java lang Thread run Thread java 834 ```

Ravi Singal · Answer

The readiness probe of pinot controller pod is failing As a result k8s does not allow connection to controller leader pod pinot server connection fails with following error message

Ravi Singal · Answer

Following is the readiness probe on pinot controller ``` readinessProbe failureThreshold 3 httpGet path pinot controller admin port 9000 scheme HTTP initialDelaySeconds 60 periodSeconds 10 successThreshold 1 timeoutSeconds 1```

Xiang Fu · Answer

I see have you try to enter into the pod and check if the http endpoint is up

Xiang Fu · Answer

also you can use ` health` endpoint to check