@kapa.ai, I am interested in developing a connector with your connector builder interface. Here is the thing, I am interested in getting data on the Chift executions. I have multiple ways of getting this: • https://api.chift.eu/consumers/{consumer_id}/datastore/{datastoreid}/data (via the get executions for a specific datastore) • https://api.chift.eu/consumers/{consumer_id}/syncs/{syncid}/flows/{flowid}/executions (via get executions for a specific consumer sync and flow) Here is the catch though, I have these endpoints: • Get syncs (https://api.chift.eu/syncs) • Get consumers (https://api.chift.eu/consumers) • Get datastores (https://api.chift.eu/datastores) The syncs endpoint already gives me informaiton on the flows like so:

"flows": [
      {
        "name": "<string>",
        "id": "<string>",
        "description": "<string>",
        "execution": {
          "type": "code",
          "data": {
            "name": "<string>"
          }
        },
        "config": {
          "datastores": []
        },
        "triggers": [
          {
            "id": "<string>",
            "type": "timer",
            "cronschedules": [
              "<string>"
            ],
            "visible": true,
            "definitionFields": [
              {}
            ],
            "options": {},
            "priority": "NORMAL"
          }
        ]
      }
    ],

But Im not sure I can loop on this object and also loop on the consumer object, can I? How do you recommend I build this stream on your connector builder? Is it even possible? It does not seem to be as simple as the connections endpoint for example: https://api.chift.eu/consumers/{consumer_id}/connections. Where consumers is the parent id stream, and then I loop over each consumer to get the connections

This is a great question about building nested/dependent streams in Airbyte's Connector Builder. Let me break down what's possible based on the available documentation. What's Straightforward: Single Parent Stream You're right that the

connections

case is simple — one parent stream (

consumers

), one child stream. This works cleanly with a

SubstreamPartitionRouter

, where you loop over each

consumer_id

to fetch connections. [Nested streams] Your More Complex Case: Multiple Parent Streams For the executions endpoint

consumers/{consumer_id}/syncs/{syncid}/flows/{flowid}/executions

, you need three levels: 1.

consumers

→ gives

consumer_id

2.

syncs

→ gives

sync_id

3.

flows

(nested inside syncs) → gives

flow_id

The Problem with Multiple Parent Streams According to community discussions, defining multiple parent streams on one child stream results in every combination of each parent key being called — which is not what you want. [Nested Data Unwrapping] The Problem with Nested Objects (flows inside syncs) Your

flows

array is nested inside the

syncs

response. The Connector Builder currently does not offer a way to flatten arbitrarily-nested API responses into a list of records. The recommended workaround for this is implementing a

CustomRecordExtractor

, but this is currently only supported on local deployments. [Nested Data Unwrapping] My Recommendation Given these constraints, here are your practical options: Option 1: Use the Datastore-based endpoint

consumers/{consumer_id}/datastore/{datastoreid}/data

only requires two parent streams: •

consumers

→

consumer_id

•

datastores

→

datastoreid

This is a cleaner two-level nesting and more feasible in Connector Builder, though you'd still need to handle the two-parent combination issue by creating separate child streams, each pointing to a single parent. Option 2: Pre-flatten flows in an intermediate stream As suggested in the community forum, you could create an intermediate stream that fetches all syncs and extracts a flat list of

(sync_id, flow_id)

pairs, then use that as the parent stream. However, flattening the nested

flows

array from the syncs response is the blocker — this requires a

CustomRecordExtractor

. [Nested Data Unwrapping] Option 3: Low-Code CDK or Python CDK If the Connector Builder's limitations are too restrictive for your use case, Airbyte recommends falling back to the Low-Code CDK directly or the Python CDK for more complex scenarios. [Connector Development] In summary: The

consumers/{consumer_id}/syncs/{syncid}/flows/{flowid}/executions

path is likely not feasible purely within the Connector Builder UI due to the nested

flows

object inside

syncs

. The datastore-based path is more tractable but still has limitations. For full flexibility, consider the Low-Code CDK with a custom component. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.