Hello, I have started using Airbyte to integrate social media data and I like it so far. I'm not sure about how Airbyte workers scale ? We may have multiple social media connections per user and a lot of users in the near future, is it safe to use Airbyte on GCP as described in the doc to handle it ?

Glad to hear you are enjoying Airbyte! Our current architecture scales vertically. Our core systems are quite efficient so the bottleneck is often the amount of data being synced.

I would think in terms of the amount of data being synced rather than the number of connections. What kind of volume are you expecting and what's a rough estimate of daily syncs?

Fyi, a more horizontally scalable architecture with K8s is on our roadmap and should be available within the next few months.

Thanks for your answer ! I just did the exercise of estimating the amount of data being synced, it would be lower than 3GB per month at the beginning. Data will be synced by calling periodically social media apis.

Nice! Our current architecture will easily support that. What about an estimate of the maximum data synced at once? (If all the connections happen to trigger at once) Would that still be 3GB?