Hello, We are using the following: Azure Managed Postgres (version 16.10) with WAL Parameters: max_wal_senders=5 max_wal_size=4096 MB min_wal_size=2048 MB wal_keep_size=400 MB checkpoint_timeout = 10 minutes Airbyte OSS Installation on Azure AKS: Postgres Connector: v3.7.0 Google BigQuery Connector: v3.0.10 UI Platform: 1.5.1 We are using incremental append mode. We are sync'ing Posgtres tables to Google BigQuery every 15 mins using Postgres replication with CDC. The first full refresh works OK. Any subsequent incremental sync fails as follows: Failure in source: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future. See https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs for more details. Any suggestions or known fixes for this please ?

Your issue—where the first full refresh works but subsequent incremental syncs fail with "Saved offset is before replication slot's confirmed lsn"—is a known problem with Airbyte's Postgres CDC integration, especially in environments with low transaction volumes or certain connector versions. ### Root Cause This error typically means that the WAL (Write-Ahead Log) segments required for incremental sync have been removed by Postgres before Airbyte could process them. This can happen even if WAL retention settings seem sufficient, due to: - Low transaction volume on the tables in your publication (WAL segments are not advanced/cleaned up as expected). - A bug in recent Airbyte Postgres connector versions, where the saved LSN (Log Sequence Number) is not correctly managed, causing the connector to think it needs to re-sync from scratch. - The use of PgJDBC versions incompatible with Debezium, which can cause the confirmed_flush_lsn to advance incorrectly, breaking the CDC state tracking. ### Known Issues and Workarounds 1. Connector Version Bug There is a confirmed bug in Airbyte Postgres connector versions >= 3.6.19 (including 3.7.0) related to the PgJDBC driver, which causes the exact error you are seeing. The recommended workaround is to downgrade the Postgres source connector to version 3.6.18, which does not have this issue. See the detailed explanation and workaround here: [CDC Not Working With Connector Version 3.6.22 (PgJDBC Issue)](https://github.com/airbytehq/airbyte/issues/49802) 2. WAL Retention and Heartbeat Even with high WAL retention settings, if your tables have low write activity, the WAL may not advance as needed. Airbyte recommends: - Adding a high-transaction table to your publication (even if you don't sync it) to keep the WAL moving. - Alternatively, create a heartbeat table and configure Airbyte to periodically write to it, which will generate WAL activity and prevent this issue. See the official troubleshooting guide for details: [Advanced WAL Disk Consumption and Heartbeat Action Query](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query) 3. State Reset Issues Some users have reported that "clearing" or "resetting" the connection in Airbyte UI does not always fully reset the CDC state, especially if you change the set of tables being synced. In such cases, deleting and recreating the connection may be necessary to fully clear the state and start fresh. [Clearing Connection does not reset state](https://github.com/airbytehq/airbyte/issues/66091) ### Recommendations - Downgrade the Postgres source connector to v3.6.18 and test if the issue persists. - Add a heartbeat table and configure the Debezium heartbeat query in Airbyte to ensure regular WAL advancement. - Increase sync frequency if possible, or further increase WAL retention settings, though this is often not sufficient alone due to the connector bug. - If you change the set of tables or encounter persistent state issues, delete and recreate the Airbyte connection. If you follow these steps and the issue persists, it is likely due to the connector version bug, and downgrading is the most reliable fix at this time. For more details and ongoing discussion, see: [CDC Not Working With Connector Version 3.6.22 (PgJDBC Issue)](https://github.com/airbytehq/airbyte/issues/49802) [Postgres Troubleshooting: Under CDC incremental mode, there are still full refresh syncs](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)

What is your

max_slot_wal_keep_size

?

On our database: wal_keep_size=400MB This is managed by Azure and cannot be change. I've monitoring the wal size between sync's and it does not get about 80MB. The above issue was resolved by following the advise above, ie downgrading: - Downgrade the Postgres source connector to v3.6.18