I have a few general questions and I am unsure if this is the appropriate place to ask.
We are trying to build a data warehouse by syncing our customers shopify data to our hosted db, the problem is I am unsure how to scale this on AWS. We have 100s of customers so we would have 100s of connections and sources with a single destination and this will only grow. Using the cloud version of airbyte would be ideal if we didn’t have to worry about this scaling (cost dependent I guess) but api access is a must as we cannot manually set up and maintain these connections/sources . we typically on board a number of new customers every day and need to automate the set up of these these connections/sources as part of that process. Ideally we would be able to run workers on their own instances , Does anyone have any suggestions or examples of how this could be achieved? or maybe its just not possibly right now and I need to find another solution