👋 hey there I have a strange situation going on. I have 2 servers setup up. Eventually, they had a problem and restarted and still running fine.I can see in the logs that they are consuming data normally:

Consumed 261 events from (rate:3.1030054/s), currentOffset=763096, numRowsConsumedSoFar=288096, numRowsIndexedSoFar=288096
....
[Consumer clientId=consumer-455, groupId=] Discovered group coordinator <redacted> (id: 2147483646 rack: null)

But the controller still show them with

dead

status and when I try to query the data, I see in the Broker log:

No server found for request 1: select responseId from responseCount limit 1

And this is the response from the query API:

{
  "exceptions": [],
  "numServersQueried": 0,
  "numServersResponded": 0,
  "numSegmentsQueried": 0,
  "numSegmentsProcessed": 0,
  "numSegmentsMatched": 0,
  "numConsumingSegmentsQueried": 0,
  "numDocsScanned": 0,
  "numEntriesScannedInFilter": 0,
  "numEntriesScannedPostFilter": 0,
  "numGroupsLimitReached": false,
  "totalDocs": 0,
  "timeUsedMs": 0,
  "offlineThreadCpuTimeNs": 0,
  "realtimeThreadCpuTimeNs": 0,
  "segmentStatistics": [],
  "traceInfo": {},
  "minConsumingFreshnessTimeMs": 0,
  "numRowsResultSet": 0
}

How can I make the Controller see they are alive? 👀

Can you check the idealstate and externalview in ZK browser? If server shows up there then may be the broker routing table needs to be rebuild (swagger api).

/tables/{tableName}/rebuildBrokerResourceFromHelixTags

that one right?

yep, the servers show up there. I tried that one, the response is

{
  "status": "Broker resource is not rebuilt because ideal state is the same for table: responseCount_REALTIME"
}

does idealstate matches externalview?

Yes, please check if idealstate matches externalview - You can do so manually, or via debug endpoint.

Although

/tables/{tableName}/rebuildBrokerResourceFromHelixTags

gave me that response, I can see that they are not the same.

/tables/<tableName>/idealstate

shows all partitions on each server (there are 2) should be online or consuming, but

/tables/<tableName>/externalview

show all the servers as offline.

Diana I did a Harddrive increase exercise today and seems like I’m getting the same issue I’m gonna do some stuff and will share what I did hopefully it helps

is this problem resolved? did you find out why the servers were in offline state?

Thanks, @Luis Fernandez I didn't try adding new servers, I wanted to make the controller recognize the current ones as alive (because they were up and consuming). We just deleted the whole deployment to try other stuff out 🤷

Do ideal state and external view show the servers and brokers for the table?

The externalview for the table only show segments and servers the external view for the broker show both brokers

does this show the servers and segments?

curl  -X GET "<broker_url>/debug/routingTable/GithubEventsTier" | jq

I'm assuming

GithubEventsTier

is the table name, so I should put my table's name, right? I don't see this

<broker>/debug/rountingTable

url in the Swagger and when I try to fetch it I get a 404. In zk, all servers have

"shutdownInProgress": "false"

I have 3 servers running now, but still the broker says

No server found for request

Its been too long, I'll restart all the pods 😞