<@U04S8PLJWMT> in Airbyte OSS I have a all connect...
# replication-ask-ai
a
@kapa.ai in Airbyte OSS I have a all connectors with snowflake destination (v4.0.35), that source multiple databases in a SQL Server and the incremental + dedupes stopped working and stopped replicating data. the only possible way of replicating is by changing the replication to full refresh and overwrite + dedupe
k
The knowledge sources only describe how sync modes behave conceptually; they do not mention any specific bug or known issue where Incremental | Append + Deduped stops working for SQL Server → Snowflake or requires switching to Full Refresh | Overwrite + Deduped to continue syncing. Because of that, I can’t confidently identify a root cause from the provided docs alone. From the docs, the key differences between the modes you mention are: - Incremental | Append + Deduped - Reads only new/updated records from the source (using a cursor such as
updated_at
). - Appends them to history tables in the destination. - Builds a deduped final table that is unique per primary key by sorting on the cursor and keeping the latest record. [[Incremental deduped](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] - Full Refresh | Overwrite + Deduped - Always reads all data from the source. - Replaces all data in the destination table on each sync. - Also ensures the final table is unique per primary key by deduplicating based on the cursor. [[Full refresh modes](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes); [FR overwrite+deduped](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes/full-refresh-overwrite-deduped)] Given that: - If Incremental has stopped replicating, the docs suggest checking the cursor field and primary key configuration, because Incremental relies on those to decide what is “new or modified” and what to dedupe. [[Incremental deduped](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] - Full Refresh working while Incremental does not is consistent with a problem in incremental logic (cursor, CDC, or state), since Full Refresh ignores previous state and always reloads everything. [[Sync modes overview](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes); [What is full refresh](https://airbyte.com/tutorials/full-data-synchronization#step-2)] However, the knowledge sources do not provide: - Any troubleshooting steps specific to Snowflake destination v4.0.35. - Any known regression or limitation for SQL Server sources with Incremental + Deduped. - Any logs, error patterns, or configuration examples for this exact scenario. Because of that, I can’t say from the docs whether you’re hitting: - A connector bug, - A misconfigured cursor/primary key, - Or a state issue that prevents Incremental from seeing new records. To move forward, you’ll likely need to: 1. Inspect the connection’s configured cursor field and primary key for each affected stream and confirm they still exist and are being updated as expected in SQL Server (this is implied by how Incremental + Deduped works, but not detailed for your exact case). [[Incremental deduped](https://docs.airbyte.com/platform/2.0/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] 2. Compare behavior with a Full Refresh run to confirm that new records are present in the source but not picked up by Incremental. 3. Check Airbyte logs and/or open an issue or discussion with those details (connector versions, logs, schema, cursor field) in the Airbyte community or GitHub, since the provided docs don’t cover this failure mode. If you can share logs or the exact stream configuration (cursor field, primary key, and sync mode per stream), I can help you interpret them, but I must stress that the current knowledge sources don’t contain enough information to definitively diagnose why Incremental + Deduped stopped working in your specific setup.
a
@kapa.ai incremental runs forever and multiple attempts are done
k
A message from kapa.ai
The knowledge sources explain how Incremental | Append + Deduped is supposed to work, but they do not describe a situation where an incremental sync “runs forever” or repeatedly retries, nor do they list any specific bug for SQL Server → Snowflake that would cause this behavior. Because of that, I can’t point to a documented root cause for your exact issue. From the docs, the main things that can block or stall incremental syncs are related to the cursor field and how deltas are queried: - Incremental sources use a query like:
Copy code
sql   SELECT *    FROM table    WHERE cursor_field > 'last_sync_max_cursor_field_value'
for Incremental + Deduped. [[Inc deduped limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations)] - If the cursor field is not updated correctly in the source, or if it stops increasing, the connector may not see any “new” rows and can behave unexpectedly. [[Inc deduped limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations)] - For Incremental (Append), the pattern is similar, using
>=
instead of `>`: [[Inc append limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append#known-limitations)]
Copy code
sql   SELECT *    FROM table    WHERE cursor_field >= 'last_sync_max_cursor_field_value'
The Snowflake source docs (as an example of incremental behavior) also emphasize that the cursor field must be: - Monotonically increasing, - Not updated after creation, - Without many duplicates or NULLs. [[Snowflake incremental](https://docs.airbyte.com/integrations/sources/snowflake#incremental-sync)] These same principles apply conceptually to other sources: if the cursor field is problematic, incremental sync can misbehave. However, the provided documentation does not say that a bad cursor or primary key configuration will cause the job to “run forever” or loop with multiple attempts; it only states that updated records might not be picked up or that intermediate changes can be missed. [[Inc deduped limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations); [Inc append limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append#known-limitations)] So, based solely on the docs: - I can confirm that Incremental | Append + Deduped depends heavily on a correct cursor field and primary key configuration, and misconfiguration can cause data not to be replicated. [[Incremental deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] - I cannot confirm from the knowledge sources why your incremental job is “running forever” or retrying; there is no explicit mention of this symptom or a fix for it. Given these limits, the only doc-based checks I can suggest are: 1. Verify that each affected stream has: - A valid cursor field that still exists in SQL Server and is being updated/monotonically increasing. - A correct primary key so deduplication can complete. [[Incremental deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] 2. Confirm that the source really has new/updated rows whose cursor value is greater than the last stored cursor value, otherwise the incremental query may not return anything. [[Inc deduped limits](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations)] 3. If Full Refresh | Overwrite + Deduped completes but Incremental keeps running, collect logs and configuration details and escalate via Airbyte’s support channels, because this “runs forever” behavior is not documented in the provided sources. If you can share the configured cursor field and primary key for one problematic stream, I can map that against the documented expectations, but I still won’t be able to confirm any undocumented bugs or performance issues beyond what the docs state.
a
@kapa.ai could it be that the sql server source connector has been upgraded and doesn't allow Incremental | Append + Deduped
k
A message from kapa.ai
The knowledge sources do not state that the SQL Server (MSSQL) source connector was upgraded in a way that removes or disallows Incremental | Append + Deduped. What the docs do say is: - Sync modes are a combination of source and destination capabilities. Incremental vs Full Refresh is about how the source reads; Append / Append + Deduped / Overwrite are about how the destination writes. [[Sync modes overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes)] - Append + Deduped is described as a supported mode in general, when both source and destination support the required capabilities (incremental + primary key). [[Incremental deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped); [Sync modes overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes)] There is no mention in the provided MSSQL migration guide of removing incremental support or changing which sync modes are allowed; it only documents changes like data type mappings and CDC cursor behavior. [[MSSQL migration guide](https://docs.airbyte.com/integrations/sources/mssql-migrations)] So, based on the available docs: - I cannot find any evidence that a recent SQL Server source upgrade removed support for Incremental | Append + Deduped. - Whether that mode appears for a given stream still depends on the combination of: - The source supporting Incremental for that stream, and - The destination (Snowflake) supporting Append + Deduped for that stream. [[Sync modes overview](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes)] If the mode has disappeared or behaves differently after an upgrade, that behavior is not documented in the sources you provided, so I can’t confirm it from documentation alone.