https://linen.dev logo
v

Vijay

10/25/2021, 8:51 PM
How many connections can we have in airbyte? Are there inherent limitations?
Hi, the scheduler does a loop across all the connections to schedule which is 0(n) on the total number of connections. We’ve seen this take up to 20 - 30 seconds at 2.5k total connections. Depending on your scheduling latencies, this might be a limitation.
u

user

10/26/2021, 2:57 AM
For now we are pulling once per day, per connection. So it should not be a problem. But in terms of number of defined connections.. can we have 100k connections?
u

user

10/26/2021, 2:58 AM
By the way, great work! Kudos to the team. Just trying out Airbyte and great experience so far!
u

user

10/26/2021, 6:48 AM
In theory you can. I wouldn’t be surprised to see high scheduling latencies at that level
u

user

10/26/2021, 6:48 AM
@Harshith (Airbyte) can you weigh in from previous experience?
n

Nikhil Jain

10/26/2021, 7:07 AM
Hey Vijay we had about 6k connections and we had seen it takes 1-2 minutes before it schedules by going through all the connections 1. We have used custom scheduler to distribute the sync load across the day 2. Also I think moving to EKS made running multiple jobs / horizontal scaling much easier.
4 Views