https://linen.dev logo
r

Raj

12/29/2021, 9:13 AM
Hi all, we are running airbyte(0.29.22) on a ubuntu with 16GB and 4core cpu. We found that airbyte is running atmost 4 jobs even when the resources are available. we changed SUBMITTER_NUM_THREADS to 100 just to check if this helps. Can you please suggest what might be the problem? How can we increase the number of parallel jobs ? After changing SUBMITTER_NUM_THREADS to 100, all the jobs(less than 100) status change to running but only 4 of them are actually running.
Hi @Raj, I'd suggest you tweak
MAX_SYNC_WORKERS
to increase parallelism in the job run. You can also edit:
MAX_DISCOVER_WORKERS
,
MAX_CHECK_WORKERS
and
MAX_SPEC_WORKERS
All of these default to 5.
@[DEPRECATED] Augustin Lafanechere thanks for the reply. I should add them in the .env file right ?
k

Kashif Vikaas

12/30/2021, 2:32 PM
Yes, at the moment we do not expose these variables to the docker-compose file but we are working on it.
j

Johan Wärlander

01/03/2022, 5:04 PM
Hi @Raj the PR above was just merged, it'll soon be shipped in the next Airbyte version. Do you mind upgrading your Airbyte version? Keep in mind we have a mandatory intermediate upgrade as mentioned here.
Thanks for the update @[DEPRECATED] Augustin Lafanechere will explore this option.