Consul is on my list, but it's got a few things that are dead on arrival for what I need I think, but will give it a fair shake.

I guess to throw numbers on it. Cold start 1k node cluster and time to broadcast changes are across the board 2-3x faster on etcd. Mind you, that's not usually particularly important. But it's interesting. I think heartbeat traffic on large clusters will get more interesting as heartbeat tracking purportedly doesn't have to go through the ensemble leader (zk to my recollection has to do session syncing from peers to the leader).

hrm. Whelp. etcd was cleaning house on zk up to 25k hosts. But when we went to 50k it crapped the bed

Zk took it like a champ though. Thundering herd cold start 50k hosts, watches propogated completely in about 30 seconds.

Though this doesn't have helix style external views in the mix (which is a big gut punch)

Honestly. I'm shocked. I thought for sure the newer technology would win out. etcd has more modern pieces that make it significantly faster at scale, but it eats shit well before ZK does.

So don't let anyone tell you newer is strictly better 😂

Honestly, I think is just telling me that I don't need to replace zk. I just need a modern rewrite of ZK with Raft and HTTP2, but keep everything else achitecturally the same otherwise.

Managed to make etcd hit the 50k mark, but it needs way bigger boxes then ZK does. Fascinating