I ve noticed that if I m using ZooKeeper for HA and I kill a Apache Flink #troubleshooting

I've noticed that if I'm using ZooKeeper for HA an...

Shahid Chohan

07/12/2023, 12:56 AM

I've noticed that if I'm using ZooKeeper for HA and I kill a ZK node that's in the quorum, the JM will restart. If I've specified multiple ZK nodes in the quorum string (say 5 for example), is there a way to get the JM to connect to a different node without triggering a restart? I've seen the`high-availability.zookeeper.client.tolerate-suspended-connections` config option, but I'm not sure if it actually applies since the node itself should be lost.

Alex Cruise

07/13/2023, 8:36 PM

Do you have multiple addresses in the
high-availability.zookeeper.quorum
config item? If not the JM might not be aware of any other members of the ZK cluster whoops nm, reread the question

Alex Cruise

07/13/2023, 8:40 PM

Do you actually have >1 JM running? 🙂

Alex Cruise

07/13/2023, 8:41 PM

That shouldn't be necessary when it comes to tolerating a killed ZK node though

Alex Cruise

07/13/2023, 8:42 PM

It's often surprisingly difficult to distinguish between outright failure and slow response though; I would try turning on the

tolerate-suspended-connections

and see how that goes

Alex Cruise

07/13/2023, 8:44 PM

e.g. if you never try to write to a socket, you may never be notified that the remote is dead, so a lot of protocols use heartbeats

Alex Cruise

07/13/2023, 8:56 PM

Looks like ZK clients do send heartbeats anyway

Alex Cruise

07/13/2023, 8:59 PM

https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/HighAvailabilityOptions.java#L153-L168 --

retry-wait

defaults to 5000ms, and

max-retry-attempts

defaults to 3

2 Views

Open in Slack

Previous Next