Hi @Koorous Vargha, Hope you had a great long weekend! I have a few questions about Venice, hope you could take a look some time. 1. Safe deployment Curious if you guys have any recommendations for safe deployment of Venice? We had our ingestion jobs failing when servers were being rolled out:

03:48:28 ERROR VenicePushJob:        Kill check monitor detected that push job for store: ..., version: 19 has been killed. Status: ERROR

OfflinePushMonitor

flips an in-flight push to terminal

ERROR

when it briefly sees a replica go

OFFLINE

. To mitigate this, I added retries of the

VenicePushJob

inside of our Whatnot-specific Spark Job, and it helped. I was also looking at

node_removable

API on the controller side and considered adding it into

preStop

hook of servers, - but it looks like it's mostly for decommissioning servers and making sure that the stores remain healthy (it can return

WILL_LOSE_DATA

,

WILL_TRIGGER_LOAD_REBALANCE

). 2. Partition skew mitigation We used replication factor of 3 on our load-test setup with 10 venice servers. There were cases when some servers could have all 3 partitions of present for a particular store - not all active, but still, results in 33% of all reads for such store to be served from this particular server. 3. Per-store RCU quota and per-router quota Looks like the RCU quota is a combination of per-store and per-router quota (hav a write-up on this). I'm curious, what is the

max.read.capacity

config that you use in production for routers? And what are the practical maximum per-store RCUs limits do you set?

I can answer the first 2 right now. The third one I need to investigate a bit. 1. Safe Deployment a. The

node_removable

API is also used for determining whether it's safe to take down a node for deployment. b. There are two ways to address this: i. Set deployment concurrency to 1 (quickest way to mitigate but leads to slow deployment times) ii. Build an orchestration service that leverages the

node_removable

API that tells K8s whether or not it is safe to take the node down for deployment. We have something internally called Helix ACM that hooks this in, but it is not open-source right now and I don't know if they will since it has a lot LinkedIn specific implementation details. 2. Partition skew mitigation a. The issue is that the default in the OSS repo for

REBALANCE_STRATEGY

is

CrushRebalanceStrategy

which doesn't do placement evenly.

CrushEdRebalanceStrategy

was created to address this. There is a new version called

CrushEd2RebalanceStrategy

but it doesn't look like it is open-source right now.

CrushEdRebalanceStrategy

should be good enough to address the issue you're facing. b. I will update the default later today.

this.helixRebalanceAlg = props.getString(HELIX_REBALANCE_ALG, CrushRebalanceStrategy.class.getName());

Thanks a lot @Koorous Vargha.

Thanks for the detailed analysis for #3. > I'm curious, what is the

max.read.capacity

config that you use in production for routers? It ranges from 20M-100M in production. Ali would be better able to provide some historical context on this. > The metric name

read_quota_usage_kps

is also misleading. It's a

Total

JMX attribute (cumulative running total of per-second usage observations), not a rate. Dashboards have to wrap it with

per_second()

to see actual rate. The name implies it IS a rate. This is unfortunately tech debt 😢. There was some system at LinkedIn that couldn't read rate metrics that we depended on (it's decommissioned now), so we had to use Total and do the conversion as you described. Long-term we will be moving away from Tehtui based metrics over to OTEL which should fix this.