@kapa.ai, i have updated airbyte to 1.7.0 i have a connection that is doing to postgres to s3 replication it is cdc and incremental append but due to some reason i am not getting incremental records what can be the reason

Based on the knowledge sources, there are several known reasons why CDC incremental syncs from Postgres to S3 might not be picking up incremental records: Common Causes 1. WAL Resume Position Timeout The connector searches for the WAL resume position, and if it times out (default wait time may be too short), it closes the connection thinking there are no more messages. You may see logs like:

INFO i.d.c.p.PostgresStreamingChangeEventSource(searchWalPosition):323 Searching for WAL resume position
INFO ... Source has no more messages, closing connection.

Fix: Increase the Initial Waiting Time in Seconds (Advanced setting) in the connector settings. One user resolved this by increasing it to 2400 seconds. [CDC not working issue] 2. WAL Logs Being Purged If the WAL logs needed for incremental sync have been removed by Postgres, the connector will fall back to a full refresh. You may see this log: >

Saved offset is before Replication slot's confirmed_flush_lsn, Airbyte will trigger sync from scratch

Fixes: • Sync more frequently so WAL logs aren't purged before being consumed. • Increase

wal_keep_size

on your Postgres instance. [Postgres troubleshooting] 3. Tables Without Primary Keys CDC incremental mode only works for tables with primary keys. Tables without primary keys won't have the Incremental option available. [Postgres CDC tables] 4. Schema Changes Not Handled If your source schema changed (columns added, renamed, or deleted), the Incremental sync cannot handle this automatically. A Full Refresh - Overwrite is recommended to resync with the new schema. [Incremental append limitations] 5. New Schemas/Columns Added Without Snapshot When using CDC, any newly added schema or column must first be snapshotted before CDC can track changes on it. You need to: 1. Approve the schema changes in Airbyte. 2. Enable the new schema/column. 3. Trigger a Refresh Stream from the Status page. [CDC limitations] 6. Changes Made via TRUNCATE/ALTER CDC only captures changes from `DELETE`/`INSERT`/`UPDATE` statements. Changes made via

TRUNCATE

or

ALTER

won't appear in the WAL logs and therefore won't be synced. [CDC limitations] ------------------------- I'd recommend checking your sync logs for the WAL resume position timeout message first, as that appears to be a common culprit in similar setups. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.