Hi @kapa.ai I have an array type user input. For each value in the input I need to run a stream: Stream name "Orders" Url parameter: user input (account id) For each account ID I need to run that stream sync. So I'm looking for something along the lines of a for loop or something else. What would be my best way to deal with this?

Based on your question, it seems you're looking to run a stream sync for multiple account IDs using Airbyte. This is a common scenario, especially when dealing with APIs that require different account IDs for data extraction. Here's how you can approach this: 1. Parameterized Requests: Airbyte's Connector Builder UI supports parameterized requests, which allows you to run a stream for multiple values. This is exactly what you need for your use case. Here's how you can set it up: - Set "Parameter Values" to "User input" - Create a new user input named "Account IDs", set type to

array

- Set "Current Parameter Value Identifier" to a relevant name (e.g.,

account

) - Configure "Inject Parameter Value into outgoing HTTP Request" to inject the account ID into your URL parameter For example, if your Orders stream URL is something like

<https://api.example.com/orders>

, you would set the "URL Path" to

/orders?account_id={{ stream_partition.account }}

. This setup is similar to the example provided in the [Airbyte documentation for parameterized requests](https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#example). 2. Stream Slicing: If you're developing a custom connector, you can use the concept of stream slicing. This allows you to break down a large sync into smaller chunks, which in your case would be individual account IDs. You would implement this in the

stream_slices

method of your stream class. For each account ID, you'd generate a slice, and then use that slice in your

path

or

request_params

method to construct the appropriate URL. This approach is mentioned in the [Airbyte documentation on stream slices](https://docs.airbyte.com/connector-development/cdk-python/stream-slices). 3. API Approach: If you're using Airbyte in a more programmatic way, you could potentially use the Airbyte API to create multiple sources, one for each account ID. However, this might be more complex and less maintainable than the above options. Given your specific requirements, I would recommend using the Parameterized Requests approach if you're using the Connector Builder UI, or the Stream Slicing approach if you're developing a custom connector. These methods will allow you to efficiently sync data for multiple account IDs without having to manually create separate jobs for each one.

Great, and I need to create a new id from the parameter (name "merchant") and the "id" field from the response. This will be the unique key

A message from kapa.ai

To create a new unique key by combining the "merchant" parameter and the "id" field from the response, you can use Airbyte's record processing features, specifically the "Primary key" field. Here's how you can approach this: 1. First, ensure that your stream is set up with the parameterized request as discussed earlier, with "merchant" as the parameter name. 2. In the record processing section of the Connector Builder UI, you can define a custom primary key. This key will be used to uniquely identify each record and is crucial for deduplication in incremental syncs. 3. To create a composite primary key using both the "merchant" parameter and the "id" field from the response, you can set the "Primary key" field to include both of these values. According to the [Airbyte documentation on record processing](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#meta-data): > The "Primary key" field specifies how to uniquely identify a record. This is important for downstream de-duplication of records (e.g. by the incremental sync - Append + Deduped sync mode). > In some cases there is no such field but a combination of multiple fields is guaranteed to be unique, for example the shipping zone locations of the Woocommerce API do not have an id, but each combination of the

code

and

type

fields is guaranteed to be unique. In your case, you would set the "Primary key" to "`merchant`, `id`". This tells Airbyte to use both the "merchant" parameter (which comes from your input) and the "id" field from the response to create a unique identifier for each record. By setting up the primary key this way, you ensure that: 1. Each record is uniquely identified by the combination of the merchant and the id. 2. Downstream systems can use this composite key for deduplication and proper handling of the data. 3. If you're using an incremental sync mode like "Append + Deduped", Airbyte will be able to correctly identify and handle updates to existing records. Remember, this setup allows for automatic downstream deduplication of records based on the values of these two fields, as mentioned in the [Airbyte documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#meta-data).