Hi Team,
We are doing a POC for using Pinot for o...
# generalg
Gaurav Sinha
11/09/2022, 8:41 AMHi Team,
We are doing a POC for using Pinot for our company. We have it setup on K8S on GCP. Today i am observing an error while trying to query through Pinot UI [4 out of 6 segments are showing as unavailable]-
Copy code
[
{
"message": "null:\n4 segments [user_impressions_v1_stg__3__0__20221107T1247Z, user_impressions_v1_stg__0__0__20221107T1247Z, user_impressions_v1_stg__1__0__20221107T1247Z, user_impressions_v1_stg__4__0__20221107T1247Z] unavailable",
"errorCode": 305
}
]Gaurav Sinha
11/09/2022, 4:37 PMDebug API response is as below -
Copy code
[
{
"tableName": "user_impressions_v1_stg_REALTIME",
"numSegments": 6,
"numServers": 3,
"numBrokers": 3,
"segmentDebugInfos": [
{
"segmentName": "user_impressions_v1_stg__0__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-2.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-1.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
},
{
"segmentName": "user_impressions_v1_stg__1__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-0.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-2.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
},
{
"segmentName": "user_impressions_v1_stg__2__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-0.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-1.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
},
{
"segmentName": "user_impressions_v1_stg__3__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-2.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-1.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
},
{
"segmentName": "user_impressions_v1_stg__4__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-0.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-2.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
},
{
"segmentName": "user_impressions_v1_stg__5__0__20221107T1247Z",
"serverState": {
"Server_pinot-poc-server-0.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
},
"Server_pinot-poc-server-1.pinot-poc-server-headless.pinot-poc.svc.cluster.local_8098": {
"idealState": null,
"externalView": null,
"segmentSize": null,
"consumerInfo": null,
"errorInfo": null
}
}
}
],
"serverDebugInfos": [],
"brokerDebugInfos": [],
"tableSize": {
"reportedSize": "0 bytes",
"estimatedSize": "0 bytes"
},
"ingestionStatus": {
"ingestionState": "HEALTHY",
"errorMessage": ""
}
}
]Gaurav Sinha
11/11/2022, 8:53 AMHi @Neha Pawar @Kishore G Can you please take a look ?
cc: @Ranveer Singh
n
Neha Pawar
11/11/2022, 3:28 PM
This looks like the first segment of every partition. Likely the other two will also go into error state soon.
Look at server log. Grep for the segment name on that server, and grep for ERROR and Exception. Most likely, some exception is happening when trying to go from consuming to completed segment. Could be some time column format issue
2 Views