https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
d
If you have a multi-node Kube cluster, I recommend draining/updating the nodes 1 by 1 to prevent these prolonged outages.
f
This is a fair point. I need to double check with the SREs doing the upgrade. Assumed they were doing this though.
Yeah, looks like we are doing this.
d
Do you have anti-affinity configured for the broker pods? to prevent all of them being assigned to a single host?
What errors are you seeing in the client application logs during these outages?
f
I would think we are using anti-affinity to spread out the workloads but if the node it has affinity for is drainined I wopuld still expect it to be deployed to another node unless the resources are too limited? things like
Copy code
unable to subscribe to topic=<persistent://public/default/topic_name>: connection error
Copy code
failed to ack message: consumer state is closed
Copy code
failed to create producer at reconnect: connection timed out
The scaled is pretty high, so that may be slowing things down
d
As long as one broker remained active on the non-drained node, it should detect this and take ownership of the topics. You will also need as many active bookie pods as you have your ack quorum set to. Otherwise, the producers won’t receive acks back and will stop publishing.
These error messages indicate an issue at the network layer. How big is your workload in terms of throughput? How many brokers do you have total? How many after shutting down a single K8s node?
f
maybe 250,000 messages published per second, some 36 bookies
we are gearing up to do another kube update so I will have to gather more information during that process
d
How many nodes are in your resource pool?
Open in Slack
PreviousNext
Powered by