@kapa.ai how should an airbyte connector handle a

JSON:API

response where the additional data lives in

included

in the top level object parameter?

When dealing with a JSON:API response where related resources live in a top-level

included

array, the standard record selector alone is insufficient — you need custom extraction logic to resolve the relationships. How It Works In JSON:API, related resources are returned in a top-level

included

array rather than being nested inline within each record. The records in

data

contain

relationships

fields that reference items in

included

by

type

and

id

. [JSON:API Included Resolution] Recommended Approach: Custom Record Extractor The Klaviyo source connector provides a good reference implementation. It uses a custom component called

KlaviyoIncludedFieldExtractor

that: 1. Extracts the main records from the

data

path 2. Extracts all items from the

included

path 3. Matches them by

type

and

id

, then merges the included resource's attributes back into the relationship data Here's the core logic: [Klaviyo Extractor]

@dataclass
class KlaviyoIncludedFieldExtractor(DpathExtractor):
    def extract_records(self, response: requests.Response) -> Iterable[Mapping[str, Any]]:
        evaluated_field_paths = [field_path.eval(self.config) for field_path in self._field_path]
        target_records = self.extract_records_by_path(response, evaluated_field_paths)
        included_relations = list(self.extract_records_by_path(response, ["included"]))
        updated_records = self.update_target_records_with_included(target_records, included_relations)
        yield from updated_records

    @staticmethod
    def update_target_records_with_included(
        target_records, included_relations
    ) -> Iterable[Mapping[str, Any]]:
        for target_record in target_records:
            target_relationships = target_record.get("relationships", {})
            for relationship_type, relationship in target_relationships.items():
                relationship_data = relationship.get("data", {})
                if isinstance(relationship_data, dict):
                    # Single object relationship
                    included_type = relationship_data.get("type")
                    included_id = relationship_data.get("id")
                    for included_relation in included_relations:
                        if included_relation.get("type") == included_type and included_relation.get("id") == included_id:
                            relationship_data.update(included_relation.get("attributes", {}))
                            if "relationships" in included_relation:
                                relationship_data["relationships"] = included_relation["relationships"]
                elif isinstance(relationship_data, list):
                    # Array of objects relationship
                    for item in relationship_data:
                        included_type = item.get("type")
                        included_id = item.get("id")
                        for included_relation in included_relations:
                            if included_relation.get("type") == included_type and included_relation.get("id") == included_id:
                                item.update(included_relation.get("attributes", {}))
                                if "relationships" in included_relation:
                                    item["relationships"] = included_relation["relationships"]
            yield target_record

Key Points • No-code/low-code connector builder cannot handle this natively with just a record selector, as the selector only extracts records from a path but cannot merge data across sibling keys like

data

and

included

. [Record Selector Docs] • You need to use either the Low-Code CDK with custom components or the Python CDK to implement this merging logic. [Forum: Parsing API Response] • The custom extractor should extend

DpathExtractor

and override

extract_records

to handle the relationship resolution. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.