https://linen.dev logo
#ask-community-for-troubleshooting
Title
# ask-community-for-troubleshooting
j

Jeferson Machado Santos

10/19/2021, 3:25 PM
Is this your first time deploying Airbyte: Yes OS Version / Instance: e2-standard-2 Memory / Disk: 32Gb Deployment: Kubernetes Airbyte Version: 0.30.19-alpha Source name/version: NA Destination name/version: NA Description: What is the effect of increasing the replicas of the workers on workers.yaml? Does it have impact on performance when moving data? Although there are more worker pods with more replicas, every time a job of moving data starts, new pods for source and destination are created, so it is note clear if more worker nodes impact on the performance of the operations.
j

Jared Rhizor (Airbyte)

10/19/2021, 3:38 PM
You should scale the number of worker nodes based on the throughput of data being synced simultaneously. You could probably run 10 rate limited api syncs on a single worker node but you may want multiple nodes if you’re doing file or db syncs.
The worker is basically a bus that handles the record verification/logging/state management for the source and destination pods.
j

Jeferson Machado Santos

10/19/2021, 3:44 PM
ok. and the way to get the data moving faster would be to have more powerful machines on the node pools?
j

Jared Rhizor (Airbyte)

10/19/2021, 3:45 PM
Yeah, or more pods, depending on if your jobs are computationally expensive. Are you maxing out CPU or mem for the worker pods?
j

Jeferson Machado Santos

10/19/2021, 3:59 PM
So, i tried to, but then i got an issue which i reported here: https://airbytehq.slack.com/archives/C01MFR03D5W/p1634164642470700
but still not figured it out...
u

[DEPRECATED] Marcos Marx

10/19/2021, 10:36 PM
The speed of the sync most of the cases depends on the source connector. I don't think the worker is a bottleneck when you're thinking in increase the performance. For database you can increase the batchsize, this will reduce the total time but will require more memory/cpu to execute the job. For API sources you're limited from the provider side.
this 1