@kapa.ai I'm creating a connector using YAML to get data from an endpoint. However, I am having problems obtaining all the fields that are returned by the original API response. Example below RECORDS { "id": "28089818", "lead_id": "22364914", "owner_id": "37852", "owner_name": "Joãozinho", "cadence": "Wallet", "cadence_id": "45254", "created_date": "2024-10-25T180341.955Z", "start_date": "2025-01-13T030000.000Z", "end_date": "2024-10-25T181143.714Z", "status": "SWITCHED_CADENCE", "lead_origin_channel": "UNKNOWN", "conversion": "Industry" }, RESPONSE { "id": "28089818", "lead_id": "22364914", "owner_id": "37852", "owner_name": "Joãozinho", "cadence": "Wallet", "cadence_id": "45254", "lead_base": null, "lead_base_id": null, "rd_conversion_date": null, "created_date": "2024-10-25T180341.955Z", "excluded_date": null, "start_date": "2025-01-13T030000.000Z", "scheduled_date": null, "end_date": "2024-10-25T181143.714Z", "first_activity_date": null, "last_activity_date": null, "status": "SWITCHED_CADENCE", "lost_reason": null, "lost_reason_id": null, "lead_origin_channel": "UNKNOWN", "lead_origin_source": null, "lead_origin_campaign": null, "conversion": "Industry" }, Notice that in response, there are more fields than in records, and I need all response fields to be returned.

Based on the information you've provided, it seems you're experiencing an issue where some fields with null values are being omitted from the records in your Airbyte connector. This is a known behavior in some cases, and there are a few ways to address it. 1. First, it's important to note that Airbyte typically doesn't omit fields just because they're null. The issue you're experiencing might be related to how the record selector is configured in your YAML file. 2. To ensure all fields are included, even if they're null, you can try using the

AddFields

transformation. This allows you to explicitly add fields that might be getting omitted. Here's an example of how you could modify your YAML file to include these fields [Understanding the YAML file](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/record-selector#adding-fields):

yaml    stream:      <...>      transformations:        - type: AddFields          fields:            - path: ["lead_base"]              value: "{{ record['lead_base'] }}"            - path: ["lead_base_id"]              value: "{{ record['lead_base_id'] }}"            - path: ["rd_conversion_date"]              value: "{{ record['rd_conversion_date'] }}"            # Add other missing fields similarly

This transformation will ensure that these fields are included in the record, even if they're null. 3. Another approach is to check your record selector configuration. Make sure you're selecting the entire response object. If you're using a field path, it should typically be set to select the root of the response. For example [Record Selection](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#record-selection):

yaml    selector:      extractor:        field_path: []

This selects the entire response as a single record. 4. If you're using the Connector Builder UI, you can also check the "Field Path" setting to ensure it's correctly set to capture all fields from the response [Field Path](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#field-path). 5. Lastly, make sure your schema definition includes all the fields you expect to receive, including those that might be null. The schema should define all possible fields, even if they're not always present in every response. If you've tried these approaches and are still experiencing issues, it might be helpful to review your entire YAML configuration or consider opening an issue on the Airbyte GitHub repository for more specific assistance. Remember, when developing custom connectors, it's crucial to thoroughly test with various response scenarios to ensure all possible fields are correctly captured and processed.

Sorry @Nicholas Machado, I seem to be having a bit of trouble. Don't worry though, I've pinged my support folks and they will try to fix me up! 🧑‍💻 Perhaps try again in 5mins?