Channels
v
Vijay
10/25/2021, 8:51 PMHow many connections can we have in airbyte? Are there inherent limitations?
Hi, the scheduler does a loop across all the connections to schedule which is 0(n) on the total number of connections.
We’ve seen this take up to 20 - 30 seconds at 2.5k total connections. Depending on your scheduling latencies, this might be a limitation.
u
user
10/26/2021, 2:57 AMFor now we are pulling once per day, per connection. So it should not be a problem. But in terms of number of defined connections.. can we have 100k connections?
u
user
10/26/2021, 2:58 AMBy the way, great work! Kudos to the team. Just trying out Airbyte and great experience so far!
u
user
10/26/2021, 6:48 AMIn theory you can. I wouldn’t be surprised to see high scheduling latencies at that level
u
user
10/26/2021, 6:48 AM@Harshith (Airbyte) can you weigh in from previous experience?
n
Nikhil Jain
10/26/2021, 7:07 AMHey Vijay we had about 6k connections and we had seen it takes 1-2 minutes before it schedules by going through all the connections
1. We have used custom scheduler to distribute the sync load across the day
2. Also I think moving to EKS made running multiple jobs / horizontal scaling much easier.
4 Views