https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
a
It needs to go to the same broker where the state is stored. That documentation you linked to shows how the router does this for you automatically:
If no 
druid.router.avatica.balancer
property is set, the Router will also default to using the Rendezvous Hash Balancer.
If you didn't have a router you would need a solution to ensure the connections go to the same broker.
d
There is no way to make that driver stateless?
I guess if user uses the http endpoint, then it will be stateless.
I see this logic:
Copy code
String connectionId = getAvaticaConnectionId(requestMap);
targetServer = hostFinder.findServerAvatica(connectionId);
byte[] requestBytes = objectMapper.writeValueAsBytes(requestMap);
request.setAttribute(AVATICA_QUERY_ATTRIBUTE, requestBytes);
LOG.debug("Forwarding JDBC connection [%s] to broker [%s]", connectionId, targetServer.getHost());
hm… looks like router does the right thing though.
Copy code
public Collection<Server> getAllServers()
  {
    return hostSelector.getAllBrokers().values().stream()
                       .flatMap(Collection::stream)
                       .collect(Collectors.toList());
  }
and then
Copy code
public Server findServerAvatica(String connectionId)
  {
    Server chosenServer = avaticaConnectionBalancer.pickServer(getAllServers(), connectionId);
  }
a
I will try to be clearer as I think maybe I added to your confusion. If you are using the router service everything will work for you automatically and there is nothing you need to do. IF you are making connections directly to brokers via JDBC you need to make sure you hit the same broker every time. The router does this for you automatically. So if you hit the router first you don't need to worry about it.
d
ah, I think I saw the root cause, it’s another problem caused by non-existent IP addresses when deploying in a dynamic environment like Kubernetes.
in the old ways, our 
druid.host=xxx
is ip address based. That caused this code here:
Copy code
public Collection<Server> getAllServers()
  {
    return hostSelector.getAllBrokers().values().stream()
                       .flatMap(Collection::stream)
                       .collect(Collectors.toList());
  }
To possible contain non-existent IP addresses because sometimes pods disappeared and ZK somehow still keep track of that IP addresses.
But that’s fine, we have an answer already. Our new clusters uses DNS for ZK memberships, so this problem will go away.
a
I don't know if that state is actually persisted anywhere so if the Pod gets replaced even with a consistent name I don't think it would fully solve the problem of brokers being re-created mid JDBC session. Though maybe it would end up being more graceful as it would at least be forwarded to a broker that exists?
a
Right now, if the brokers membership changes in a cluster (new one comes or old one dies), some connections can be re-routed to a different broker. This is because there is no centralized routing of Avatica JDBC connections. Different routers use hashing to arrive at the destination broker independently. This logic works fine as long as the broker set doesn't change.
d
My customer said that as long as he doesn’t keep the connectionId, e.g. just close and don’t reuse connection), he no longer face issues. But that’s not ideal, preferably he wants to pool the connection still.
a
Yup. Not reusing the connection is as close as we can get to stateless communication.
Open in Slack
PreviousNext
Powered by