Hey! I'm currently using the self hosted version, ...
# replication-troubleshooting
c
Hey! I'm currently using the self hosted version, building some custom connectors for the Heartland Point of Sale API. Heartland issues a custom subdomain to each of it's customers, so the base_url of the API changes with each storefront we have. We have a database we can pull each store's subdomain, ID, and apiKey from, but I'm not sure how to automate the ETL process in Airbyte. I need a way to request: 1. Foreach storefront a. Request a bunch of different streams from https://{subdomain}.retail.heartland.us/api b. Transform each subdomain data by adding a store database ID c. Load the streams' data into the DB. I'm brand new to Airbyte. I've built a working connector with 1 fixed subdomain, but I'm not sure how to do what I need to without creating 50+ nearly identical custom connectors for each store subdomain.
a
This tutorial shows how to loop over a response using the python cdk - https://airbyte.com/tutorials/extract-data-from-the-webflow-api This talks about the ELT approach where transformation is done after loading into the destination https://airbyte.com/blog/elt-pipeline
💯 1
c
Thank you my good man!