@kapa.ai Incremental syncs stopped working, and instead of performing incremental syncs, it just syncs everything from the integration, why?

There are several known reasons why incremental syncs in Airbyte can fall back to syncing all records instead of only new ones. Here are the most common causes: 1. Missing or Empty Connection State If the cursor/state is not being saved between syncs, Airbyte has no reference point for the next incremental sync and will re-fetch everything. This can happen if the cursor field is not properly configured or if the state is not being generated correctly. [Incremental Sync Issue] 2. Cursor Field Not Properly Defined For some connectors (e.g., Postgres), the user must manually define the cursor field. If it's missing or set to

null

, the sync will behave like a full refresh. [Fetching Updated Data] 3. Records Updated Without Updating the Cursor Field If records in the source are modified but the cursor field (e.g.,

updated_at

) is not updated, those records won't be picked up by incremental sync. The underlying query used is:

SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_field_value'

If the cursor value doesn't change, the record is invisible to the next sync. [Incremental Sync - Append] 4. Connector-Specific Bugs Several connectors have had known bugs where incremental sync was broken: • Klaviyo: Incremental streams were behaving as full refresh. [Klaviyo Incremental Issue] • HubSpot Contacts: Moving to a low-code approach broke incremental syncing, causing all contacts to be pulled instead of only updated ones. [HubSpot Issue] • Gong: Streams like

extensiveCalls

and

callTranscripts

had hardcoded start dates, causing full re-fetches on every sync. [Gong Issue] • Postgres (CTID cursor): When a sync returns 0 records for a stream, the cursor resets to

(0,0)

, causing a full table re-read on the next sync. [Postgres CTID Issue] 5. Source Schema Changes If a column is added, renamed, or deleted in the source, incremental sync cannot handle this automatically. A Full Refresh - Overwrite is recommended in this case. [Incremental Sync - Append] Recommended troubleshooting steps: 1. Check the Connection State in the Airbyte UI to confirm a cursor value is being saved after each sync. 2. Verify the cursor field is correctly configured and is being updated in the source data. 3. Check your connector version and update it, as many incremental sync bugs have been fixed in newer versions. 4. Review the sync logs for clues such as a

null

cursor or missing state. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai What if nothing has changed in terms of the source schema?

If the source schema hasn't changed, the most likely causes based on the knowledge sources are: 1. Empty or Missing Connection State The most common culprit is that the cursor/state is simply not being saved after each sync. If the connection state is empty, Airbyte has no reference point and will re-fetch all records. You can verify this by checking the Connection State in the Airbyte UI after a sync completes. [Incremental Sync Issue] 2. Cursor Field Returning Null If the cursor field is

null

in the logs, the incremental sync won't work correctly and will behave like a full refresh. [Incremental Sync Issue] 3. Connector-Specific Bug Some connectors have known bugs where the state is not properly preserved between syncs. For example, with Postgres CTID-based incremental syncs, when a sync returns 0 new records, the cursor was resetting to

(0,0)

, causing a full table re-read on the next sync. [Postgres CTID Fix] For custom connectors, a "doubled up" or corrupted state object can cause subsequent syncs to fail or behave unexpectedly. [Connector State Issue] Recommended Next Steps 1. Check the Connection State in the UI — is a cursor value actually being saved after each sync? 2. Review the sync logs for any

null

cursor values or state-related errors. 3. Update your connector to the latest version, as incremental sync bugs are frequently patched. 4. If using a custom connector, verify that

get_updated_state()

is returning the correct state structure and that the cursor field value is a plain string/timestamp, not a nested dictionary. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.