https://linen.dev logo
k

kananindzya

02/02/2022, 11:24 AM
Hi, I wanted to clarify whether airbyte allows you to create more than 1_000_000 connections? Is there a lot of hardware needed so that, for example, more than 1_000_000 jobs work in parallel?
Hi @kananindzya, I'm curious about your use case, in which context would you need to create a million of connections? There's virtually no limit to the connection number, you can increase parallelism by tweaking the following env variables:
Copy code
### APPLICATIONS ###
# Scheduler #
# Relevant to scaling.
SUBMITTER_NUM_THREADS=10

# Worker #
# Relevant to scaling.
MAX_SYNC_WORKERS=5
MAX_SPEC_WORKERS=5
MAX_CHECK_WORKERS=5
MAX_DISCOVER_WORKERS=5
You can find more details about scaling Airbyte here
8 Views