I use Stripe sourceDefinition version 5.6.0. No transformations are performed in my custom destination. When I run discovery schema for source, I see all the fields that I need. Some of them can be null and I handle it on receiving side. However, when I run sync, I receive data with some fields missing. It happens, when they are null in Stripe. But I already have logic for webhooks processing, which respects null - and I want to reuse it. How can I influence airbyte to send complete payload, even when fileds are explicitly null?

Airbyte's behavior with null fields is determined by the source connector and the JSON schema it provides. If a field can be null, it should be listed in the JSON schema as such, for example

[null, string]

. If a null property is found for a non-nullable schema, a validation error may occur in the platform or the destination may have trouble storing the record [source](https://docs.airbyte.com/understanding-airbyte/supported-data-types#nulls). However, if you want to ensure that all fields, even those that are null, are included in the payload sent to your destination, you might want to consider using the "Schemaless" option. When this option is selected, the schema will always be

{"data": object}

, regardless of the contents of the data. During the sync, each record is "wrapped" behind a key named

data

. This means that the destination receives the data with one top-level field only, and the value of the field is the entire record. This option avoids a time-consuming or inaccurate

discover

phase and guarantees that everything ends up in your destination, at the expense of Airbyte being able to structure the data into different columns [source](https://docs.airbyte.com/understanding-airbyte/schemaless-sources-and-destinations#schemaless-schema). Please note that this is a general approach and might not be applicable to all sources or destinations. Always test and verify the results in your specific setup.

I'm ok with schema. I just want to get all fields, including nulls.

In Airbyte, many sources cannot guarantee that all fields are present on all records. In these cases, sources should not list them as

required

fields, and add that the property can be

null

in the jsonSchema, e.g.

[null, string]

. If a null property is found for a non-nullable schema, a validation error may occur in the platform or the destination may have trouble storing the record [source](https://docs.airbyte.com/understanding-airbyte/supported-data-types#nulls). In the JSON schema, it's essential to include

$schema

,

type

, and

additionalProperties: true

fields in your schema. Typically, Airbyte schemas require null values for each field to make the stream more reliable if the field doesn't receive any data [source](https://docs.airbyte.com/connector-development/schema-reference). Here is an example of how you can define your schema to include null values:

json {     "$schema": "<http://json-schema.org/draft-07/schema#>",     "type": "object",     "additionalProperties": true,     "properties": {       "id": {         "type": ["null", "string"]       },       "date_created": {         "format": "date-time",         "type": ["null", "string"]       },       "date_updated": {         "format": "date-time",         "type": ["null", "string"]       },       "total": {         "type": ["null", "integer"]       },       "status": {         "type": ["string", "null"],         "enum": ["published", "draft"]       },       "example_obj": {         "type": ["null", "object"],         "additionalProperties": true,         "properties": {           "steps": {             "type": ["null", "string"]           },           "count_steps": {             "type": ["null", "integer"]           }         }       },       "example_string_array": {         "items": {           "type": ["null", "string"]         }       }     }   }

This schema will ensure that all fields, including those with null values, are included in the data extraction.