Hi all, we are running airbyte(0.29.22) on a ubunt...
# contributing-to-airbyter
Raj
12/29/2021, 9:13 AMHi all, we are running airbyte(0.29.22) on a ubuntu with 16GB and 4core cpu. We found that airbyte is running atmost 4 jobs even when the resources are available. we changed SUBMITTER_NUM_THREADS to 100 just to check if this helps. Can you please suggest what might be the problem? How can we increase the number of parallel jobs ?
After changing SUBMITTER_NUM_THREADS to 100, all the jobs(less than 100) status change to running but only 4 of them are actually running.
Raj
12/29/2021, 2:18 PMHi @Raj, I'd suggest you tweak
MAX_SYNC_WORKERSMAX_DISCOVER_WORKERSMAX_CHECK_WORKERSMAX_SPEC_WORKERSRaj
12/29/2021, 2:19 PMAll of these default to 5.
Raj
12/29/2021, 6:28 PM@[DEPRECATED] Augustin Lafanechere thanks for the reply. I should add them in the .env file right ?
k
Kashif Vikaas
12/30/2021, 2:32 PMYes, at the moment we do not expose these variables to the docker-compose file but we are working on it.
j
Johan Wärlander
01/03/2022, 5:04 PMHi @Raj the PR above was just merged, it'll soon be shipped in the next Airbyte version. Do you mind upgrading your Airbyte version? Keep in mind we have a mandatory intermediate upgrade as mentioned here.
Johan Wärlander
01/03/2022, 5:04 PMThanks for the update @[DEPRECATED] Augustin Lafanechere will explore this option.