@kapa.ai why does my Postgresql to Snowflake connection launch a full sync when I try to only refresh one stream?

When you trigger a refresh on a single stream in Airbyte, only that stream's cursor is reset and re-synced — other streams in the connection are not affected. However, there are a few known reasons why a full sync might appear to trigger unexpectedly in a Postgres → Snowflake connection: 1. CDC WAL (Write-Ahead Log) Has Been Purged If you're using CDC incremental mode, Postgres may have removed the WAL files needed for the incremental sync. When this happens, Airbyte is forced to fall back to a full refresh sync. You'll see this log message: > _Saved offset is before Replication slot's confirmed_flush_lsn, Airbyte will trigger sync from scratch_ This can happen when: • There are many database updates that cause Postgres to purge WAL files. • The connector acknowledged the WAL to Postgres, but the destination failed to consume the data. Possible mitigations include syncing more frequently or increasing

wal_keep_size

. [Postgres Troubleshooting] 2. CTID Cursor Bug (Incremental Append, Non-CDC) If you're using standard cursor / CTID-based incremental sync (not CDC), there is a known open bug: when a sync returns 0 records for a stream, the CTID cursor is incorrectly reset to

(0,0)

instead of preserving the previous position. This causes the next sync for that stream to perform a full table re-read. [CTID cursor bug] 3. Refresh Sync Behavior by Design A Refresh Sync intentionally re-imports all records for the refreshed stream(s) from the beginning (it resets the cursor). This is expected behavior — it will be slower and more expensive than a normal incremental sync. Importantly, only the stream(s) you select for refresh are affected; other streams in the connection sync normally. [Refreshing your data] In summary: If you're seeing an unexpected full sync on streams you did not select for refresh, the most likely culprits are the CDC WAL purge issue or the CTID cursor reset bug. If the full sync is on the stream you did refresh, that is expected behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Oh, so when you refresh one stream, it is expected that the other streams will sync normally at the same time? Is there a way to only sync one stream?

New message text here