Hi everyone !  Using Kubernetes deployment and having increased the number of worker, I noticed that, depending on the node where workers were started, performances could be affected.  For instance, I have currently 4 workers replicas distributed on 3 nodes (

node A : 1 worker

,

node B : 1 worker

and

node C : 2 workers

). When syncing, if my

source-worker

and

destination-worker

are correctly distributed, I’m able to process about 25M rows in one hour (with a well distributed CPU load). However, if by any chance

source-worker

and

destination-worker

are both started in

node C

, then the node’s CPU goes up to 190% (against 10% and 10% for the two others) and time processing is much more slower as I’m only able to process about 15M rows within an hour.  Not sure if it's an actual request as I don't know if there is an existing strategy to avoid this situation but is there a way to force the parallelization of workers on different nodes to maximize performance ?

Hi, we currently don’t have a way to force parallelisation. We view this as part of K8s scheduling so this is opaque today

Within the next month, we will be working on combining the source/worker containers into a single pod. This should keep networking to a single node and should make things more efficient

we are still benchmarking and working on things

Great ! Thank you for the feedback 🙂