https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
You can take a look at the requests and responses that the UI is submitting and see the errors in the Developer view on the browser (F12 on Chrome). Here's what I see in terms of listing the services:
the response will likely shed some light on why it isn't working for you.
a
Thank you. I'll take a look.
The SQL query the UI uses to retrieve information about running the instances under Services returns 
is_leader: 0
for all the coordinators, which contradicts what the coordinators return via ReST. Is that stored on ZooKeeper or PostgreSQL?
b
Actually, neither. It goes against sys.servers, which is a system table. Those exist in pieces of code on each server, and get populated and maintained in memory. I don't know what the API does, and why it works. (Or why the system tables are wrong.) I can't think of a way to pin this down atm for a good bug report, but if you want to fix it, it's possible that restarting things might help.
a
Good to know about the "sys.servers" table. I tried roll restarting the coordinators (I'm running on K8s), but that didn't work. I just tried deleting the Pod that was the leader, and I saw the following on the "new" leader:
Copy code
2023-02-28T19:52:25,886 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.server.coordinator.DruidCoordinator - I am the leader of the coordinators, all must bow! Starting coordination in [PT10S].
2023-02-28T19:52:25,908 INFO [LeaderSelector[/druid/coordinator/_COORDINATOR]] org.apache.druid.curator.discovery.CuratorServiceAnnouncer - Announcing service[DruidNode{serviceName='druid/coordinator', host='10.244.34.4', bindOnHost=false, port=-1, plaintextPort=8081, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}]
2023-02-28T19:52:25,914 INFO [LeaderSelector[/druid/overlord/_OVERLORD]] org.apache.druid.indexing.overlord.TaskMaster - By the power of Grayskull, I have the power!
So, leader election happens but this "sys.servers" table is not updated correctly, and the logs have no errors.
b
Coordinators seem OK, I thought maybe restarting the router might help?
a
I tried that, but unfortunately didn't help.
s
could this be a permission issue on the Metadata DB?
g
@Alejandro Galue when you say "router cannot detect the coordinator/overlord leader for some reason" what do you mean exactly? (like what API or UI error are you getting?)
I don't think it'd be related to the metadata db. Possibly something related to leader election; although it'd be odd for the server to be discovered and leader election not also work.
@Alejandro Galue fyi the 
is_leader
in 
sys.segments
for coordinator is populated by calling 
/druid/coordinator/v1/leader
on any one of the coordinators and comparing it to the host/port advertised by each coordinator
a
Hello @Gian Merlino The SQL query that the Router uses to retrieve the information from 
sys.servers
returns 0 for the 
is_leader
column for all the coordinators. However, the endpoint you mentioned returns the correct result (
/druid/coordinator/v1/leader
). Leader election works (I tested it). That's why I don't understand what's happening.
Today, I made some changes to the Indexers and Historicals and roll-restarted their StatefulSets, and suddenly, now the leaders appear correctly. Not sure how this influenced something managed by Coordinators and Overlords.
s
It's good that it is working, but still very weird. @Gian Merlino what process is in charge of updating the sys.servers table?
Perhaps there is a log entry there that explains this.
g
the Broker updates it based on its opinion of who the leader is: I wonder if the Broker lost the thread?
Open in Slack
PreviousNext
Powered by