Hi all we are running airbyte 0 29 22 on a ubuntu with 16GB airbytehq #contributing-to-airbyte

Channels

p0-source-airtable-stream-failures

airbyte-enterprise-pricing

advice-data-warehouses

well-seeded-accounts

help-api-cli-orchestration

advice-data-orchestration

advice-data-quality

movedata-general

infra-dev-alerts

infra-dev-alerts-webhook

prod1-alerts-test

airbyte-for-power-users

memes-and-banter

movedata-future-evolution-of-the-data-stack

p0-amazon-ads-03-30

marcos-test-discourse

airbyte-plus-airflow

write-for-the-community

advice-reverse-etl

movedata-speakers

partner-productpair-hightouch

databricks-airbyte316

community-strategy

license-questions

movedata-building-connectors-in-minutes

movedata-navigating-the-modern-data-stack-with-open-source-headless-bi

movedata-data-orchestration-is-not-just-running-jobs-on-a-schedule

movedata-using-airbyte-on-day-1of-our-startup

movedata-using-airbyte-to-build-your-dremio-open-data-lakehouse

movedata-ingesting-data-with-airbyte-into-a-high-speed-data-lakehouse-using-trin

movedata-in-2023-data-trust-will-be-more-important-than-ever-before

p0-instagram-incompatible-metrics

movedata-mobilize-the-worlds-data

connector-builds

movedata-ducky-data-crunching-on-the-laptop---the-pendulum-swings

movedata-traditional-data-catalogs-will-be-replace-by-active-metadata-platforms

movedata-prefect-ing-self-hosted-airbyte

movedata-if-you-build-it-will-they-come-how-to-activate-your-modern-data-stack

movedata-data-analysts-are-setup-to-fail-and-its-our-fault

movedata-guardrails-not-stop-signs

movedata-dataops-on-the-open-modern-data-stack

movedata-better-data-testing-with-the-data-error-generating-process

movedata-open-source-communities-shape-modern-data-stacks

movedata-moving-data-reliably-ingestion-observability-are-better-together

movedata-airbyte-connection-management-scale

movedata-prep-your-pipelines---reverse-etl-and-the-coming-great-flood

movedata-cicd-for-data-building-devtest-data-environments-with-lakefs

movedata-why-bi-is-not-enough

office-hour-12-october

movedata-the-data-ecosystem-is-ready-for-etl-to-be-dead

understanding-airbyte

p0-cloud-syncs-failing-dec-15

movedata-data-engineering-is-software-engineering-and-software-engineering-is-da

p1-auto-detect-schema-issues-2-1-23

oss-master-build-failure

example-of-channel

advice-data-visualization

movedata-who-needs-a-traditional-data-warehouse-when-you-can-get-real-time-analy

advice-data-privacy

movedata-the-end-of-the-pipeline

movedata-keynote-building-the-foundations-of-data-movement

movedata-the-best-data-warehouse-is-a-lakehouse

movedata-five-causes-of-data-quality-issues

movedata-streaming-made-modern

movedata-modern-data-management---how-to-achieve-data-discovery

movedata-dev-first-open-source-data-observability-solution

movedata-building-a-real-time-user-facing-dashboard-with-airbyte-redpanda-and

movedata-a-different-way-to-work

movedata-let-your-data-team-choose-their-own-tools

movedata-bring-your-own-infra

movedata-airbyte-orchestration-in-gcp

airbyte-plus-dagster

feedback-and-requests

help-connector-development

advice-data-transformation

hacktoberfest-2022

dev-frontend6323

help-infrastructure-deployment

events-and-conferences

airbyte-cloud-pricing

help-connections-issues

shameless-plugs

advice-data-ingestion

advice-data-architecture

oss-master-build-failure

cloud-master-build-failure

contributing-to-airbyte

troubleshooting

connector-development

#contributing-to-airbyte

Title

r

Raj

12/29/2021, 9:13 AM

Hi all, we are running airbyte(0.29.22) on a ubuntu with 16GB and 4core cpu. We found that airbyte is running atmost 4 jobs even when the resources are available. we changed SUBMITTER_NUM_THREADS to 100 just to check if this helps. Can you please suggest what might be the problem? How can we increase the number of parallel jobs ? After changing SUBMITTER_NUM_THREADS to 100, all the jobs(less than 100) status change to running but only 4 of them are actually running.

Hi @Raj, I'd suggest you tweak

MAX_SYNC_WORKERS

to increase parallelism in the job run. You can also edit:

MAX_DISCOVER_WORKERS

,

MAX_CHECK_WORKERS

and

MAX_SPEC_WORKERS

All of these default to 5.

@[DEPRECATED] Augustin Lafanechere thanks for the reply. I should add them in the .env file right ?

k

Kashif Vikaas

12/30/2021, 2:32 PM

Yes, at the moment we do not expose these variables to the docker-compose file but we are working on it.

j

Johan Wärlander

01/03/2022, 5:04 PM

Hi @Raj the PR above was just merged, it'll soon be shipped in the next Airbyte version. Do you mind upgrading your Airbyte version? Keep in mind we have a mandatory intermediate upgrade as mentioned here.

Thanks for the update @[DEPRECATED] Augustin Lafanechere will explore this option.