How many connections can we have in airbyte? Are t...
# contributing-to-airbyte
v
How many connections can we have in airbyte? Are there inherent limitations?
Hi, the scheduler does a loop across all the connections to schedule which is 0(n) on the total number of connections. We’ve seen this take up to 20 - 30 seconds at 2.5k total connections. Depending on your scheduling latencies, this might be a limitation.
u
For now we are pulling once per day, per connection. So it should not be a problem. But in terms of number of defined connections.. can we have 100k connections?
u
By the way, great work! Kudos to the team. Just trying out Airbyte and great experience so far!
u
In theory you can. I wouldn’t be surprised to see high scheduling latencies at that level
u
@Harshith (Airbyte) can you weigh in from previous experience?
n
Hey Vijay we had about 6k connections and we had seen it takes 1-2 minutes before it schedules by going through all the connections 1. We have used custom scheduler to distribute the sync load across the day 2. Also I think moving to EKS made running multiple jobs / horizontal scaling much easier.