How do I make spicedb react to pod failure faster for dispatch? any config available? eg I have 2 nodes A & B with dispatch enabled, B is suddenly killed I notice that query to A will fail with deadline_exceeded, I suspect because A is trying to dispatch to B which is no longer alive I am testing this case out in a local cluster using dns resolver for dispatch upstream address, I notice if I wait for more time after B is removed (eg > 30s) then all query will succeed In k8s we are using Operator to deploy spicedb, I am observing some brief spike of deadline_exceeded error when we rollout restart spicedb deployment, I am trying to figure out if this issue is due to intra-spicedb networking, how is the kuberesolver different? (if any)

if you're using the operator and letting it configure the pods, it should use the kuberesolver by default, which should quickly remove the killed pods from the dispatch hashring. i wouldn't expect to see deadline exceeded in that case. i'd also expect that if a pod goes down, the http2 connection that carries the gRPC traffic would be severed, in which case the calling node should receive an error. when you're getting deadline_exceeded, have you been able to isolate which component is producing the error? is it the node that receives the initial request?

apology, I checked it again, in prod/Operator case, I don't see deadline_exceeded error during rollout restart, only massively increased latency (our deadline is 10s) https://cdn.discordapp.com/attachments/1468871231255085088/1469162331761148006/Screenshot_2026-02-06_at_10.45.54_AM.png?ex=6986a78b&is=6985560b&hm=906df4cf505ed0ac23eda7fcaa30ce17a85ab607bdd0fe128ea84e3bf977d91d&

yes - ideally that metric is around zero all the time