@kapa.ai How are null cursor values treated during an incremental load?

During an incremental sync in Airbyte, null cursor values are generally excluded or cause issues, depending on the connector: • Records with null cursor values are skipped: For source connectors like BigQuery and Snowflake, records where the cursor field is

NULL

are excluded from incremental sync. It is recommended to ensure your cursor field has a

NOT NULL

constraint or a default value. [BigQuery source; Snowflake source] • Null cursor values can cause sync failures at the destination: When using CDC (e.g., MySQL → ClickHouse), a null value in the

_ab_cdc_cursor

column can cause a hard error if the destination column is defined as non-nullable. For example, the ClickHouse destination connector raises:

An attempt to write null into not nullable column '_ab_cdc_cursor'

. [ClickHouse issue] • Cursor field must be properly set: According to the Airbyte protocol, if

source_defined_cursor

,

cursor_field

, and

default_cursor_field

are all unset, the configuration is considered invalid. [Airbyte Protocol] Best practice: Always choose a cursor field that is monotonically increasing and never null. If null cursor values are possible in your data, consider adding a

NOT NULL

constraint or a default value at the source, or switching to a CDC-based approach where appropriate. [Snowflake source] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai How about with a microsoft sql server source? I want Airbyte to continue loading all null cursor values, along with the current cursor value.

The knowledge sources do not contain enough information to answer your question about how null cursor values are specifically handled with the Microsoft SQL Server (MSSQL) source connector, or whether there is a configuration option to include null cursor values alongside current cursor values during an incremental load. What the knowledge sources do mention about the MSSQL source is the

scan_changes_with_user_defined_cursor

replication method, which has an

exclude_todays_data

option for temporal cursor fields [MSSQL Terraform resource], but there is no mention of null cursor value handling specifically. For a definitive answer on MSSQL null cursor behavior, I'd recommend: 1. Checking the Airbyte MSSQL source connector documentation directly. 2. Considering a CDC (Change Data Capture) approach for the MSSQL source, which uses Write-Ahead Logs rather than a cursor field and may handle edge cases like null values differently. [Incremental Append - Known Limitations] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.