@kapa.ai Is there a way to use Record Selector but keep the parent attributes?

Thinking completed The knowledge sources do not explicitly mention a built-in parameter on the

RecordSelector

itself to automatically retain parent-level or wrapper attributes when drilling down into nested records. When using

field_path

on a nested array, the extractor selects that array and discards surrounding/parent-level fields [Record Selection]. However, depending on your architecture, you can achieve this using the following approaches: 1. Select the parent object and use Stream Transformations Instead of pointing the

field_path

directly to the nested array or child fields, you can select the parent object (or element in an array) and then use stream

transformations

to pull nested fields into the top level [Issue with API Response Parsing using Record Selector; Forum Post: Extracting specific fields from a nested JSON response for a connector]. Using

AddFields

and

RemoveFields

, you can extract fields from child structures while keeping parent attributes [source-linear manifest.yaml; Forum Post: Extracting specific fields from a nested JSON response for a connector]:

transformations:
  - type: AddFields
    fields:
      - type: AddedFieldDefinition
        path: ["child_id"]
        value: "{{ record.child.id }}"
  - type: RemoveFields
    field_pointers:
      - ["child"]

2. Add Parent Stream values via SubstreamPartitionRouter If the "parent attributes" come from an actual parent stream via substream partitioning (

SubstreamPartitionRouter

), they are available in the partition context. You can inject them directly onto the child records using an

AddFields

transformation [source-ashby manifest.yaml]:

transformations:
  - type: AddFields
    fields:
      - path:
          - parent_id
        type: AddedFieldDefinition
        value: "{{ stream_partition.parent_id }}"

3. Use a CustomExtractor If you have nested response data where child objects must be extracted as separate records alongside metadata from their enclosing parent JSON object, standard declarative/Connector Builder components do not natively merge them [Forum Post: Nested Data Unwrapping in Connector Builder; Getting multiple values from a parent stream in Airbyte connector]. In this case, the recommended solution is to write a custom

RecordExtractor

(in low-code / Python CDK) to parse the response, inject the parent attributes into each extracted child item, and return the records [Forum Post: Nested Data Unwrapping in Connector Builder; source-outreach components.py]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai but the child is an array, will the stream transformation 1 for example automatically expand to more rows?

Thinking completed No, stream transformations (such as

AddFields

or

RemoveFields

) will not automatically expand or unnest an array of child items into multiple rows/records [Record Selector]. Transformations operate on a per-record basis: they take an individual record that has already been extracted, and add, modify, or remove fields on that specific record [YAML Reference Components>; Record Selector]. They cannot "explode" or unroll an array into separate records. If your child data is an array and you need each element to become its own record while retaining attributes from the parent object, the built-in declarative components/Connector Builder do not offer a native way to flatten or unnest arbitrarily structured responses while propagating parent attributes [Forum Post: Nested Data Unwrapping in Connector Builder]. For this scenario, your main options are: 1. Custom Record Extractor: Implement a custom component (e.g., a

CustomRecordExtractor

in Python) that parses the response, loops over the nested array, injects the parent fields into each array item, and yields them as individual records [Forum Post: Nested Data Unwrapping in Connector Builder]. 2. Substream Partitioning: Keep the parent objects in one stream, and create a separate child stream using a

SubstreamPartitionRouter

to fetch the child array and pass parent attributes via

stream_partition

[Github Issue (closed): [source-gmail] unable to addFields]. 3. Downstream Unnesting: Ingest the records with the child array intact, and then unnest the array into separate rows inside your destination or data warehouse (for example, using dbt and SQL functions like

UNNEST

) [Shopping Cart Analytics Stack With Shopify, Airbyte, dbt, Dagster and BigQuery]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai 2. soluctions seems fine, which substream I should use to not even make aditional http requests for this substream?

Thinking completed The knowledge sources do not explicitly describe a way to use a

SubstreamPartitionRouter

in the Connector Builder without making HTTP requests. A

SubstreamPartitionRouter

is designed specifically to route and execute outgoing HTTP requests for each parent partition [SubstreamPartitionRouter]. However, if your parent stream's response already contains the nested child array (for example,

Purchase Order

already includes its items) and you want to extract those items into separate records without making additional HTTP calls, you have two options depending on whether you use the declarative framework or custom code: Option 1: Use RecordExpander in the Declarative Manifest / YAML In the declarative CDK,

DpathExtractor

supports an optional component called

RecordExpander

[DpathExtractor].

RecordExpander

extracts items directly from a nested array inside the response and emits each item as a separate record, completely avoiding extra network calls [RecordExpander]. Crucially, it has a parameter: •

remain_original_record: true

: Embeds the entire parent record into each expanded child record under an

original_record

field [RecordExpander]. You can then use

AddFields

transformations to pull the parent attributes onto the child record and remove

original_record

[manifest.yaml]:

retriever:
  type: SimpleRetriever
  requester:
    path: "/purchase_orders"
  record_selector:
    type: RecordSelector
    extractor:
      type: DpathExtractor
      field_path:
        - data
      record_expander:
        type: RecordExpander
        expand_records_from_field:
          - items             # path to the child array inside each parent record
        remain_original_record: true
transformations:
  - type: AddFields
    fields:
      - path: ["order_id"]
        value: "{{ record.original_record.id }}"
  - type: RemoveFields
    field_pointers:
      - ["original_record"]

(Note: While fully supported in declarative YAML, this component is typically edited via the YAML editor rather than the standard Connector Builder GUI). Option 2: Custom Python Stream (IncrementalNestedStream pattern) In custom Python connectors, Airbyte achieves zero-request substreams by reading records from the parent stream, unpacking the nested array, and yielding the records from a buffer without sending HTTP requests [source_shopify base_streams.py>]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.