Reminder: Monday is our community syncup meeting at 9AM PST/12 PM Eastern/6 PM Central Europe. Here is the Microsoft Teams Link to join the meeting.

I suspect I'm going to be late for this one. Some morning conflicts will make this tight

I won’t be able to make it today. Not feeling too well. Please use the Slack huddle if no other LinkedIn employees can admit into the Teams meeting. My update is that the oncall runbook is now published to our site.

Oh nice!

Yeah… unfortunately there isn't a setting to override that restriction.

@Koorous Vargha hope you get well soon! Let's cancel today.

Feel better, Koorous!

Thank you @Sergey Makagonov @Hubert Puszklewicz ❤️

Oh... just checked the channel. I've a few questions and though I can ask them.

Please post them here. I can try to answer them later today.

Really comprehensive oncall runbook. Appreciate it Koorous!

No problem. If you have any questions, let me know. I tried to generalize it for OSS as much as possible

1. Let's say we run a multi-DC setup where AZ in AWS equals DC. We have three DSs. Our application is deployed in all three AZs and uses fast-client. Is it possible: * route requests from application to nodes in the same AZ? We can see now only the LEAST_LOADED or HELIX_ASSISTED way of routing * failover to other AZ on client level? 2. Do you have any process to upscale or downscale a cluster? As far as I understood from the documentation, it's pretty static 3. We are running Venice in AWS EKS. Given the cloud's nature, an instance can go down at any time. If we have RF=1, this will make the cluster unreachable. Will RF=2 make the cluster more resilient, and will we still be able to query data?

1.

HELIX_ASSISTED_ROUTING

would be what you're looking for same AZ routing. For fail-over, the functionality you're describing doesn't exist today. a. At LinkedIn, we have a 3 DC setup and when we need to fail out, it is done at the D2 level. So the failed out DC doesn't receive any outside traffic, including the Venice backend and the clients. b. For the AWS equivalent, the fail-out would need to be done at the Application Load Balancer / Route 53 level. 2. Upscale is automatic. Internally in LinkedIn, Helix has a wrapper that makes downscaling automatic. I don't think that's open sourced. a. If you know what node you're going to evict you can you can use

--remove-node-from-cluster

from the admin tool 3. Yes. In LinkedIn, we use RF=3. But RF=2 should be enough.

2. Does it mean after running command to

--remove-node-from-cluster

, cluster will be rebalanced automatically? I mean data from node I'm going to evict will be removed.

Yes. Helix will take of that automatically