Is this your first time deploying Airbyte: Yes OS Version / Instance: e2-standard-2 Memory / Disk: 32Gb Deployment: Kubernetes Airbyte Version: 0.30.19-alpha Source name/version: NA Destination name/version: NA Description: What is the effect of increasing the replicas of the workers on workers.yaml? Does it have impact on performance when moving data? Although there are more worker pods with more replicas, every time a job of moving data starts, new pods for source and destination are created, so it is note clear if more worker nodes impact on the performance of the operations.

You should scale the number of worker nodes based on the throughput of data being synced simultaneously. You could probably run 10 rate limited api syncs on a single worker node but you may want multiple nodes if you’re doing file or db syncs.

ok. and the way to get the data moving faster would be to have more powerful machines on the node pools?

Yeah, or more pods, depending on if your jobs are computationally expensive. Are you maxing out CPU or mem for the worker pods?

So, i tried to, but then i got an issue which i reported here: https://airbytehq.slack.com/archives/C01MFR03D5W/p1634164642470700

The speed of the sync most of the cases depends on the source connector. I don't think the worker is a bottleneck when you're thinking in increase the performance. For database you can increase the batchsize, this will reduce the total time but will require more memory/cpu to execute the job. For API sources you're limited from the provider side.