This message was deleted.
# generals
Slackbot
09/05/2023, 12:19 AMThis message was deleted.
✅ 1
i
Ilya Esin
09/05/2023, 7:22 AMYes, I'd like to go deeper in this topic. Because, over the time we obtained some experience with data volume loss, instance disconnection and reboots on AWS. All of this is recent with not-so-big voldemort RO1 clusters.
f
Felix GV
09/05/2023, 1:24 PMYeah so... Voldemort did not have self-healing. That was one of the motivations for building Venice on top of Apache Helix.
Depending how Helix is configured, you can have it repair under-replicated partitions instantly or after some delay.
For us, IIRC, we set it to repair after 30m if one replica is down, and immediately if more than one is down. This avoids useless repair traffic during transient issues...
Felix GV
09/05/2023, 1:27 PMWith smaller clusters, the cost to repair will be higher... for example, if you have a 3 nodes cluster, and 1 node goes down, then each of the 2 remaining nodes need to ingest 50% of what the 3rd node carried. That may take a while and means you cannot run your nodes near full capacity. Whereas if you have a 25 nodes cluster and one goes down, then the remaining 24 nodes need to ingest only 4% each, which is quick and requires less headroom.
That is one of the motivations for having multi-tenant clusters.
i
Ilya Esin
09/05/2023, 3:07 PMWell, of course we are not anyhow near to LinkedIn scale. Our total dataset served from Voldemorts is something around 5-7TiB of data.
And, instead of using Apache Helix (which I wasn't aware of till you mentioned it) we wrote our own manager for controlling lifecycle and managing deployments of voldemort clusters.
As far as I get it, you have Venice nodes that carrying not just their primary partitions but also a backup ones? Or backup partitions/portions need to be provisioned at the moment of failure detected?
f
Felix GV
09/05/2023, 3:11 PMBoth Venice and Voldemort have many replicas of each partition, and in the two systems, all replicas can serve read traffic.
The difference is that when a replica goes down in Venice, the Helix controller embedded inside Venice will take care of reallocating that replica elsewhere in the fleet, automatically bootstrap its data, and eventually make it part of serving the read traffic when it's ready.
Felix GV
09/05/2023, 3:13 PMVoldemort does not have such self-healing capability, so it has to be done from the outside... we used to do it manually back when we operated Voldemort. Sounds like you built some automation to do it (which is great...!)
i
Ilya Esin
09/05/2023, 3:56 PMOur current data format is RO1 (for sub-ms query response reasons) so no replicas is allowed 😉
Our home-grown solution is far from being great — automates only storage version swap.
😮 1
f
Felix GV
09/05/2023, 4:22 PMWow... I didn't recall those details. Sounds fragile 🫠
i
Ilya Esin
09/05/2023, 4:34 PMYup, it is. With some exceptions and workarounds, of course.
This is one of the reasons we are talking right now. We'd like to make things a bit more self-healing, friction-less and up-to-date.
👌 1
3 Views