@kapa.ai i keep having recurring issue with Airbyte CDC

Failure in source: Saved offset is before replication slot's confirmed lsn. Please reset the connection, and then increase WAL retention and/or increase sync frequency to prevent this from happening in the future. See <https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs> for more details.

We run the sync every two hours and still keeps failing after resetting the connection and we end creating a new connection each time

This error is a known CDC issue with the Postgres source; in some connector versions it can recur even when WAL retention and sync frequency look reasonable. From the linked docs and GitHub issues, there are three main angles to check: 1. Confirm it’s not just WAL retention / frequency The official explanation of this error is that the WAL records Airbyte needs have been removed, so the saved offset is older than the replication slot’s

confirmed_flush_lsn

. In that case, a full resync is expected. [Postgres troubleshooting] You’ve already: • Reset the connection • Run every 2 hours • Increased WAL retention Since it still recurs, you’re likely hitting one of the connector bugs below rather than just retention. 2. Check your Postgres connector version There is a regression starting in source-postgres 3.6.19 where

confirmed_flush_lsn

can advance even when there are no new events for the publication tables, due to a PgJDBC change. This causes exactly your symptom: Airbyte’s saved offset ends up behind the slot’s

confirmed_flush_lsn

, and subsequent syncs fail with this error. [CDC LSN bug] Key points from that issue: • The bug affects many versions: 3.6.19 up to at least 3.7.0. [CDC LSN bug] • Workaround: downgrade to 3.6.18 (or 3.6.15) – multiple users confirm this stops the recurring error. [CDC LSN bug] • A fix has been released in *3.7.2*; contributors report that 3.7.2 resolves the issue. [CDC LSN bug] Action: Check the version of your Postgres source connector: • If it’s between 3.6.19 and 3.7.1, either: ◦ Downgrade to 3.6.18 as a stable workaround, or ◦ Upgrade to 3.7.2 (or later) where the bug is reported fixed. [CDC LSN bug] 3. Add a heartbeat table (helps especially with low-traffic DBs) On low-traffic databases, WAL behavior plus the PgJDBC regression can make this worse. A documented mitigation is to add a heartbeat table and configure a Debezium heartbeat query so there are always CDC events to advance the LSN in a controlled way. [Postgres WAL troubleshooting; CDC LSN bug] Steps (summarized from docs and issue): 1. In your source DB, create a heartbeat table:CREATE TABLE public.airbyte_heartbeat ( id SERIAL PRIMARY KEY, timestamp timestamptz DEFAULT CURRENT_TIMESTAMP NOT NULL, text TEXT ); 2. Add it to the publication used by Airbyte:ALTER PUBLICATION your_publication_name ADD TABLE public.airbyte_heartbeat; 3. In the Postgres source in Airbyte, under Advanced → Update Method → Debezium heartbeat query, set:INSERT INTO public.airbyte_heartbeat (text) VALUES ('heartbeat'); This keeps CDC active even when your business tables are quiet and has helped users avoid the LSN mismatch. [Postgres WAL troubleshooting; CDC LSN bug] 4. “Invalid CDC position behavior” setting For affected versions, some users temporarily switched Invalid CDC position behavior (Advanced) from “Fail sync” to “Re-sync data” so that when the offset is invalid, Airbyte automatically does a full reload instead of failing. [PgJDBC CDC issue; CDC LSN bug] • This can reduce manual intervention, but: ◦ It will silently trigger full refreshes when the error happens. ◦ That can be expensive on large tables. Given you’re already recreating connections, this may still be preferable until you’re on a fixed connector version. 5. When resetting / recreating connections There is also a report that clearing a connection does not fully reset CDC state, and the same error persists. [Connection reset issue] If you’re repeatedly creating new connections to work around this, that aligns with this behavior and again points to the version/LSN bug rather than your WAL settings. Recommended concrete path 1. Check and change connector version: ◦ If on 3.6.19–3.7.1 → move to 3.7.2+ or downgrade to 3.6.18. 1. Add the heartbeat table + Debezium heartbeat query as above, especially if your DB has low write volume. 2. Optionally set Invalid CDC position behavior = “Re-sync data” to avoid hard failures while you validate the new version. 3. After changing version/heartbeat, do one clean reset (or new connection) and let the initial full sync complete; subsequent runs should stay incremental. If you can share your current Postgres connector version and whether your DB is high- or low-traffic, I can narrow down which of these steps is most critical for your case based on the docs and issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@Amal Mokhtar I just asked the same thing yesterday, upgrading the connector to 3.7.2 seems to have fixed it for me

Thanks a lot Joe. I just performed an upgrade from 3.7.0 to 3.7.2 and i hope it doesn't happen again.