Reminder: Monday is our community syncup meeting at 9AM PST/12 PM Eastern/6 PM Central Europe. Here is the Microsoft Teams Link to join the meeting.

Subject for next sync is revisiting the Discord migration as there is more interest on infinite message retention again

Thanks @Koorous Vargha, will try that. btw, do you mind sharing what configs you have for: •

server_promotion_to_leader_replica_delay_seconds

•

offline_job_start_timeout_ms

•

delay_to_rebalance_ms

and what are your thoughts on setting

default_offline_push_strategy

to

WAIT_N_MINUS_ONE_REPLCIA_PER_PARTITION

? For context - we faced an issue with rolling restarts of servers while under 50 concurrent VPJs running at all time. I could make it work to avoid cluster getting into a bad state AND to not have any VPJs failing, but I changed a quite a few things and now try to minimize the changes that drift off defaults. https://docs.google.com/document/d/1kmysPVYJRkmhQz4v7mozLU0Sq3j717w3oYbBwpNuJAc/edit?tab=t.0

I checked ours against OSS defaults. There are two notions of "OSS default": the code default when the property is unset, and the OSS standalone/docker sample configs. - `server.promotion.to.leader.replica.delay.seconds`: code default is

300s

, which is what we use since we do not override it. OSS standalone/docker examples set this to

1s

. - `offline.job.start.timeout.ms`: OSS code default is `120000ms`; OSS standalone/docker examples set

60000ms

. We set

960000ms

. - `delay.to.rebalance.ms`: OSS code default is

0

, disabled. OSS standalone/docker examples also use

0

. We set

1800000ms

. - `default.offline.push.strategy`: OSS code default is

WAIT_ALL_REPLICAS

when unset. We explicitly set

WAIT_N_MINUS_ONE_REPLCIA_PER_PARTITION

. So the main drift from OSS is that we delay rebalance for 30 minutes, wait longer for offline jobs to start, and allow N-1 replica readiness per partition for offline push completion. The last one is the setting I'd consider most relevant to rolling restarts with many concurrent VPJs.

appreciate quick response. This is very helpful. Tells me that I was on the right track with some of those configs

Give those a try and lmk if it helps. If it does, then I can change the defaults in OSS

I bet that typo in

WAIT_N_MINUS_ONE_REPLCIA_PER_PARTITION

is annoying for you too 😅

I just realized it

Thanks again @Koorous Vargha. I updated the doc with the minimum changes that helped us survive multiple full set of servers restarting without disruption in VPJs and without getting the cluster into a bad state. @Koorous Vargha I think we're desperately in need for vetting the configs, at least to know what you are setting on your production clusters. The

single-dc/*properties

appeared with some dangerous configs yet again (

<http://server.promotion.to|server.promotion.to>.leader.replica.delay.seconds

being 1 seconds vs 300 being default). cc @Hubert Puszklewicz

I will try to prioritize it next week. Contributing back defaults post-ramp is something Venice needs to improve on