Apache Pinot

image.png

Hi, some segments of one pinot table is bad status. When I call /tables/{realtimeTableName}/consumingSegmentsInfo api to see segment consuming info, I found these segments have no consuming info. What’s possible reasons for this error?

It seems a little weird that 10-0 segment is consuming and 10-1 has no consuming info.

What’s the status of segment in external view, and what does debug api say 

debug api says:
```"ingestionStatus": {
      "ingestionState": "UNHEALTHY",
      "errorMessage": "Not all servers responded for segment: table_02__0__3__20220712T1501Z Missing servers : [Server_pinot-server-7.pinot-server-headless.pinot.svc.cluster.local_8098]"
    }```
:rolling_on_the_floor_laughing:

But external view shows:
```"table_02__0__3__20220712T1501Z": {
      "Server_pinot-server-8.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },```


And idealstate shows:
```"table_02__0__3__20220712T1501Z": {
      "Server_pinot-server-7.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING",
      "Server_pinot-server-8.pinot-server-headless.pinot.svc.cluster.local_8098": "CONSUMING"
    },```


Some record transformation exceptions were thrown, and couldn’t locate other error that may cause server 7 dead. And from top pods command, server 7 is working, no high cpu usage, no high memory usage.