Has anyone implemented a connector that accepts mu...
# connector-contributions
a
Has anyone implemented a connector that accepts multiple sources? Something that could accept a list of stores (or their respective API credentials) as input, and then iterate over this list to establish connections to each store?
✅ 1
p
With e.g. Python CDK it shouldn't be a problem. Every stream can use different source. https://docs.airbyte.com/connector-development/cdk-python/
👀 1
e
Curious as to what you're trying to achieve here. Is it that you want to be able to bulk schedule them all to run at once (and have changes reflected)? What's the difference from configuring a connection per source?
a
Thank you for responding @Ella Rohm-Ensing (Airbyte). We have a legacy Shopify feed that sources data from about 500 stores currently and hoping to increase our business to be able to onboard more stores in the future. From the Shopify connector set up, it seems that I will need to create a new source and connection per store. We're looking to onboard and then maintain quite a few additional networks after onboarding Shopify. From the connection maintainance point of view, it may be upto 20 connections for those networks + 500-1000 for shopify. Hence was wanting to explore solutions so that Shopify can be maintained under 1 connection somehow. Do you have any suggestions/thoughts on this approach?
p
There is an existing Shopify connector, so I'd recommend using separate source and connection for every store. You can use Airbyte terraform provider to configure all connections. terraform has for_each loop, so you iterate over set of credentials to configure your sources & connections. With Kubernetes deployment and autoscaling you can have many connections running in parallel. Putting so many sources into one connection feels like a huge bottleneck. If you write your own connector, are you going to run 500 threads or processes in single Docker container? Probably not, so data synchronization will be running only for smaller part of those stores. What will happen in case of any network issues with one of those stores? Or corrupted state? How long will it take to synchronize the data from all 500 stores?
e
Ah I see - multiple accounts to connect to per one source. I thought you were looking to connect to completely different APIs in one source - this makes a lot more sense 😄
a
I would also recommend using terraform to provision one connection per store. The Shopify connector does not support multiple shops because different shops typically require a different set of credentials