How to pass a field that s value is an array from a parent s Airbyte #ask-ai

How to pass a field that’s value is an array from ...

Earnest Veasley

05/28/2024, 9:57 PM

How to pass a field that’s value is an array from a parent stream to a child stream to use as a parameterized request. Parent stream returns a record field UID : [1, 3, 2] and I want the child stream to make a request for every value in the array returned from the field UID?

kapa.ai

05/28/2024, 9:57 PM

To pass a field that’s value is an array from a parent stream to a child stream and use it as a parameterized request, you can utilize the

SubstreamPartitionRouter

in Airbyte. This allows you to process each value in the array individually for the child stream requests. Here’s a step-by-step guide based on the provided knowledge sources: 1. Define the Parent Stream: Ensure your parent stream is correctly set up to return the array of UIDs. 2. Set Up the SubstreamPartitionRouter: Use the

SubstreamPartitionRouter

to handle the partitioning of the parent stream's array field. 3. Configure the Child Stream: Set up the child stream to use the partitioned values from the parent stream. Here is an example configuration snippet that demonstrates how to achieve this:

Copy code

yaml
partition_router:
  - type: SubstreamPartitionRouter
    parent_stream_configs:
      - type: ParentStreamConfig
        parent_key: UID
        partition_field: uid
        stream:
          type: DeclarativeStream
          name: ParentStream
          primary_key: []
          schema_loader:
            type: InlineSchemaLoader
            schema:
              $schema: <http://json-schema.org/schema#>
              properties:
                UID:
                  items:
                    type: number
                  type: array
          retriever:
            type: SimpleRetriever
            requester:
              type: HttpRequester
              url_base: <https://url.ofthe.service/>
              path: /parent/endpoint
              http_method: GET
              request_parameters: {}
              request_headers: {}
              authenticator:
                type: NoAuth
            record_selector:
              type: RecordSelector
              extractor:
                type: DpathExtractor
                field_path:
                  - UID
            paginator:
              type: NoPagination

- type: DeclarativeStream
  name: ChildStream
  primary_key: []
  schema_loader:
    type: InlineSchemaLoader
    schema:
      $schema: <http://json-schema.org/schema#>
      properties:
        data:
          type: object
  retriever:
    type: SimpleRetriever
    requester:
      type: HttpRequester
      url_base: <https://url.ofthe.service/>
      path: /child/endpoint/{{ uid }}
      http_method: GET
      request_parameters: {}
      request_headers: {}
      authenticator:
        type: NoAuth
    record_selector:
      type: RecordSelector
      extractor:
        type: DpathExtractor
        field_path:
          - data
    paginator:
      type: NoPagination

In this configuration: - The

SubstreamPartitionRouter

is used to partition the

UID

array from the parent stream. - The

ChildStream

is configured to make requests using each value in the

UID

array. This setup ensures that the child stream makes a request for every value in the array returned from the

UID

field of the parent stream.

kapa.ai

05/28/2024, 9:57 PM

To pass a field that’s value is an array from a parent stream to a child stream and use it as a parameterized request, you can utilize the

SubstreamPartitionRouter

SubstreamPartitionRouter

Copy code

yaml partition_router:   - type: SubstreamPartitionRouter     parent_stream_configs:       - type: ParentStreamConfig         parent_key: UID         partition_field: uid         stream:           type: DeclarativeStream           name: ParentStream           primary_key: []           schema_loader:             type: InlineSchemaLoader             schema:               $schema: <http://json-schema.org/schema#>               properties:                 UID:                   items:                     type: number                   type: array           retriever:             type: SimpleRetriever             requester:               type: HttpRequester               url_base: <https://url.ofthe.service/>               path: /parent/endpoint               http_method: GET               request_parameters: {}               request_headers: {}               authenticator:                 type: NoAuth             record_selector:               type: RecordSelector               extractor:                 type: DpathExtractor                 field_path:                   - UID             paginator:               type: NoPagination  - type: DeclarativeStream   name: ChildStream   primary_key: []   schema_loader:     type: InlineSchemaLoader     schema:       $schema: <http://json-schema.org/schema#>       properties:         data:           type: object   retriever:     type: SimpleRetriever     requester:       type: HttpRequester       url_base: <https://url.ofthe.service/>       path: /child/endpoint/{{ uid }}       http_method: GET       request_parameters: {}       request_headers: {}       authenticator:         type: NoAuth     record_selector:       type: RecordSelector       extractor:         type: DpathExtractor         field_path:           - data     paginator:       type: NoPagination

In this configuration: - The

SubstreamPartitionRouter

is used to partition the

UID

array from the parent stream. - The

ChildStream

is configured to make requests using each value in the

UID

array. This setup ensures that the child stream makes a request for every value in the array returned from the

UID

field of the parent stream. For more detailed information, you can refer to the relevant discussions and issues: - [Github Issue 36456](https://github.com/airbytehq/airbyte/issues/36456) - [Github Issue 34337](https://github.com/airbytehq/airbyte/issues/34337) If you need further assistance, you can also check the [Airbyte documentation](https://docs.airbyte.com/).

4 Views

Open in Slack

Previous Next