What suggestions do you have for syncs to consume more wal for postgres cdc? My cdc set up can't catup with the amount of wal being generated. I added the heartbeat, and tweaked the initial wait time seconds, but it doesn't seem to have worked. The max wal keep size limit is 2TB but once the full refresh is done is at 1TB+ in 4-5 hrs.

Thinking completed Based on the available knowledge sources, when a Postgres CDC connection generates WAL much faster than the connector can consume it (or cannot flush LSNs fast enough), the bottleneck is usually either sync frequency, connector throughput limitations, or destination backpressure [Github Issue (open): [source-postgresql] CDC sync is only allowing WAL to be emptied every two syncs; Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. Here are the primary recommendations to help the connector consume WAL faster and keep up with high-volume generation: 1. Increase Sync Frequency • Sync more frequently: For high-velocity data, Airbyte recommends running syncs multiple times per day (e.g., every 1–4 hours) [Documentation Platform > Understanding-airbyte > CDC best practices>]. • Running syncs frequently prevents the accumulation of millions of records and massive WAL bloat between runs, which can overwhelm the connector and replication slot [Documentation Platform > Understanding-airbyte > CDC best practices>; Documentation Integrations > Sources > Postgres > Troubleshooting Postgres Sources>]. 2. Relieve Destination Backpressure A frequent cause of slow WAL ingestion is destination bottlenecks—if the destination takes too long to flush data, it slows down the entire pipeline [Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. • Use staging: If your destination supports staging (e.g., S3/GCS staging before loading), ensure it is enabled [Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. • Tune destination buffer parameters: Adjusting the buffer parameters on the destination (e.g., increasing concurrent streams in buffer) has been shown to resolve severe throughput drops [Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. 3. Split Tables Across Multiple Connections • Limit tables per connection: Airbyte support recommends limiting CDC connections to a maximum of 20–30 tables per connection [Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. • If you are syncing a large number of tables under a single CDC publication/connection, split them into multiple connections to parallelize throughput and reduce destination flush congestion [Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. 4. Ensure Publications Only Include Required Tables • Check your Postgres publication: if high-write tables exist in the publication that are not included in your Airbyte catalog, Debezium must still process and filter them out row-by-row [Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. • If thousands of events are filtered out, offset commits and LSN flushes can be delayed, causing a large WAL backlog [Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. Make sure your publication strictly contains only the tables you intend to sync. 5. Upgrade Connectors • Ensure you are running the latest versions of both the Postgres source connector and your destination connector. Older versions had issues where LSN flushing was delayed or incremental state was only emitted once at the end of a sync rather than checkpointed frequently [Forum Post: The CDC method does not consuming replication slot (Postgres); Forum Post: Postgres CDC connector stuck , never stops reading from source and staging files while never copying them]. (Note: The provided knowledge sources do not contain specific internal tuning knobs to increase the raw WAL read chunk size directly within Debezium/Airbyte.) 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai When the replication slot is lost, do you need to recreate it first and then do a full refresh? I wonder if there is a way for me to run a full refresh without the replication slot cuz while doing a full refresh takes a lot of the replication slot's wal size keep limit

Thinking completed Based on the provided knowledge sources: 1. Do you need to recreate the replication slot when it is lost? Yes. If a replication slot's status in Postgres becomes

lost

, the connector cannot recover on its own [Github Issue (closed): [source-postgres] Invalid CDC Position Behavior not working]. In fact, if the slot is invalid or lost, the connector will fail the sync with a configuration error [Github Issue (closed): [source-postgres] Invalid CDC Position Behavior not working; application.yml]. The slot must be dropped and recreated in Postgres, and the connection state in Airbyte must be reset [Github Issue (closed): [source-postgres] Invalid CDC Position Behavior not working]. 2. Can you run a full refresh without the replication slot? The knowledge sources do not explicitly describe an official workflow for running an initial snapshot/full refresh without the replication slot present while using the CDC connector. However, they do note the following important behavior: • Slot requirement under CDC: When configured to use CDC, the connector performs an explicit check querying

pg_replication_slots

for the specified slot. If the slot does not exist or has an invalid/lost WAL status, it throws an error and aborts [ReplicationSlotManager.kt; Github Issue (closed): [source-postgres] Invalid CDC Position Behavior not working]. • WAL accumulation during initial snapshot: When CDC is configured, the replication slot begins holding WAL changes from the moment it is created so it can track changes that happen while the initial snapshot is being read. The documentation warns specifically against creating the replication slot too early (before data/tables are populated) because subsequent syncs will have to process all accumulated WAL records [Explore Airbyte's Change Data Capture (CDC) synchronization]. If you run a connection configured strictly with standard Full Refresh sync modes (e.g., Full Refresh - Overwrite or Full Refresh - Overwrite + Deduped) rather than CDC incremental replication, Airbyte syncs data directly via database queries without using logical replication slots or tracking WAL logs [Full Refresh - Overwrite + Deduped; Full Refresh - Overwrite]. But if you intend to transition back to CDC afterward, a new replication slot must be created and Airbyte's CDC process will still need to perform its initial load/snapshot to align its offset with the slot [ReplicationSlotManager.kt; Explore Airbyte's Change Data Capture (CDC) synchronization]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Is it reasonable to evaluate

flush.lsn.source=true

so

restart_lsn

advances continuously instead of only at

START_REPLICATION

. This collapses peak retention from

rate × sync_duration

to

rate × flush_interval

.

Thinking completed Based on the provided knowledge sources, setting

flush.lsn.source = true

is not recommended in Airbyte's architecture because it creates a significant risk of data loss [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. In Airbyte's Postgres CDC connector,

flush.lsn.source

is intentionally hardcoded/configured to

false

[Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. According to Airbyte contributors: • Preventing Data Loss: Setting

flush.lsn.source = false

is necessary to prevent scenarios where the replication slot advances before records are successfully written and committed to the destination, or before the sync state is persisted [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. • At-Least-Once Delivery: If

flush.lsn.source

were set to

true

, the replication slot could advance continuously during the sync. If the job fails mid-sync (or if the destination fails to persist the data/state), the WAL on the Postgres server would already have been cleared past those records. Airbyte would not be able to re-read that WAL data on a subsequent retry, leading to lost data [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. • How Airbyte Handles Flushes: By design under the default commit behavior, the LSN is advanced on the startup of the subsequent sync. This counterintuitive design ensures that the prior sync was verified successful and the state was recorded and read back properly before allowing Postgres to purge the WAL [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. While setting it to

true

would theoretically lower peak WAL retention by collapsing it to

rate × flush_interval

, the knowledge sources indicate this breaks Airbyte's consistency guarantees and is therefore not supported [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I'm thinking what I could do is to set the wal size keep max to something like 4TB, and play with some params like the initial wait time etc. Does using the heartbeat table help advance restart_lsn at all? What are other settings I can play with

Thinking completed Does using the heartbeat table help advance restart_lsn? The heartbeat table is designed primarily to advance the slot’s LSN (

confirmed_flush_lsn

/ replication position) on low-activity databases or when there are long gaps between changes to publication tables [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. • In scenarios where other database activity is high but the specific tables in the publication are idle, Postgres keeps WAL logs because no relevant events are passing through the slot [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. • By adding a heartbeat table to the publication and running a periodic query (e.g., via Debezium heartbeat query or

pg_cron

), you generate artificial events that allow the connector to acknowledge and advance the replication slot offset [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. • However, in a high-write database generating massive WAL, changes are already constantly occurring. The bottleneck is the connector or destination not finishing/flushing fast enough, so adding a heartbeat query will generally not help

restart_lsn

advance any faster during an active snapshot or high-throughput stream. Other parameters and settings you can adjust Based on the connector configuration specifications and CDC best practices, here are the settings you can configure: 1. initial_load_timeout_hours (Initial Load Timeout) • What it does: Sets the maximum time (in hours) an initial load/snapshot is allowed to run before catching up on accumulated CDC events [PostgresSourceConfigurationSpecification.kt; CDC best practices]. • Range & Default: Defaults to

8

hours (valid range: 4 to 24 hours) [PostgresSourceConfigurationSpecification.kt]. • Usage: For very large databases (> 50 GB) or slow snapshots, setting this to the maximum of 24 hours prevents premature timeouts while snapshotting, allowing Airbyte to finish taking the baseline before switching to CDC streaming [CDC best practices]. 2. initial_waiting_seconds (Initial Waiting Time) • What it does: The amount of time the connector waits when it launches to determine if there is new data to sync or not [PostgresSourceConfigurationSpecification.kt]. During snapshots, it sets the time limit for building schema structure and baseline data; during CDC incremental runs, it determines how long to wait before timing out [CDC best practices]. • Range & Default: Defaults to

1200

seconds (20 minutes). Valid range is

120

to

2400

seconds [PostgresSourceConfigurationSpecification.kt; Setting up initial CDC waiting time]. • Usage: For high-activity databases with simple schemas, keep this at default or lower. Note that decreasing this only helps speed up zero-record syncs; it will not help if WAL volume is huge [Setting up initial CDC waiting time]. 3. lsn_commit_behaviour (LSN Commit Behavior) • What it does: Determines when Airbyte flushes the LSN of processed WAL logs back to Postgres [PostgresSourceConfigurationSpecification.kt]. • Options: ◦

After loading Data in the destination

(Default) ◦

While reading Data

[PostgresSourceConfigurationSpecification.kt] • Trade-off: If set to

While reading Data

, LSN commits happen earlier, but if a downstream failure occurs while loading into the destination, the next sync will result in a full sync [PostgresSourceConfigurationSpecification.kt]. 4. Queue Size (Advanced CDC Buffer) • What it does: Controls the internal buffer size in memory for change events [CDC best practices]. • Usage: Larger queues can handle bursts of change events more efficiently but consume more memory [CDC best practices]. Airbyte recommends keeping this at the default of

10,000

unless guided by support, as improper sizing can cause system instability [CDC best practices]. 5. Postgres WAL Retention Settings • Raising

max_slot_wal_keep_size

(or

wal_keep_size

/

wal_keep_segments

) provides headroom to prevent the replication slot from transitioning to

lost

status during extended full snapshot runs [Github Issue: Invalid CDC Position Behavior not working; Github Issue: CDC sync is making the confirmed flush LSN]. In AWS RDS/Aurora environments, setting

max_slot_wal_keep_size = -1

allows unlimited retention (bounded only by disk capacity) [Github Issue: CDC sync is making the confirmed flush LSN]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai We have no records updates during off hours, but during working hours we have about ~1 million records updates every hour or so. In that case, what settings should I modify in order to keep advancing restart_lsn to keep consuming wal?

Thinking completed Based on the provided knowledge sources, managing your workload requires balancing two distinct periods: handling the ~1 million updates/hour burst during business hours, and preventing WAL stalls during the zero-update off-hours. Here are the specific settings and configurations to adjust to keep

restart_lsn

moving and WAL being pruned: 1. Match Sync Frequency to Change Patterns • High-velocity (working hours): Airbyte recommends setting the sync schedule to run every 1 to 4 hours (or more frequently) for high-velocity replication [CDC best practices]. ◦ Running hourly or multi-hour syncs prevents the connector from accumulating millions of records between syncs, which risks hitting your WAL keep-size ceiling or overwhelming the destination [CDC best practices]. • Off-hours: Airbyte CDC guidelines caution to balance data volume with sync overhead to avoid running repeated empty syncs if there are genuinely zero changes [CDC best practices]. 2. Configure a Heartbeat for the Off-Hours During off-hours when there are no updates to publication tables, Postgres can stop advancing the replication slot's LSN, leading to accumulated WAL if background or other system transactions still occur [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. To fix this: 1. Create a dedicated heartbeat table:CREATE TABLE airbyte_heartbeat ( id SERIAL PRIMARY KEY, timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp, text TEXT );[Troubleshooting Postgres Sources] 2. Add it to your publication:ALTER PUBLICATION publicationName ADD TABLE airbyte_heartbeat;[Troubleshooting Postgres Sources] 3. Under the Postgres source connector's advanced settings in Airbyte, configure the Debezium heartbeat query property:INSERT INTO airbyte_heartbeat (text) VALUES ('heartbeat');[Troubleshooting Postgres Sources] This ensures small, periodic CDC events are captured during off-hours, allowing the replication slot to acknowledge offsets and advance

confirmed_flush_lsn

even without user data updates [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. 3. Source Connector Settings to Tune In the Postgres CDC source connector configuration, consider adjusting: • LSN Commit Behavior (

lsn_commit_behaviour

): ◦ Default:

After loading Data in the destination

[PostgresSourceConfigurationSpecification.kt]. ◦ Alternative:

While reading Data

[PostgresSourceConfigurationSpecification.kt]. ◦ Trade-off: Setting this to

While reading Data

flushes the LSN earlier to free WAL segments sooner. However, if a failure occurs while loading the destination, the subsequent sync will trigger a full resync [PostgresSourceConfigurationSpecification.kt]. • Initial Waiting Time (

initial_waiting_seconds

): ◦ Range:

120

to

2400

seconds (default is

1200

seconds) [PostgresSourceConfigurationSpecification.kt]. ◦ During low/no-activity syncs, lowering this setting prevents the connector from idling for up to 20 minutes before concluding there are no new records to sync. 4. Database-Level Retention Buffer • The database's WAL retention period (

max_slot_wal_keep_size

/

wal_keep_size

) must be longer than your sync frequency [CDC best practices; CDC best practices]. • While 7 days is optimal to handle maintenance windows and weekend buffers, the retention setting must be sized sufficiently so that 1 million records per hour do not breach your

max_slot_wal_keep_size

threshold before the next scheduled sync completes and flushes [CDC best practices; CDC best practices]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai What are the ones that would help consume retained wal?

Thinking completed Based on the knowledge sources, the specific settings and actions that help consume and clear retained WAL are: 1. lsn_commit_behaviour (Source Connector Setting) • Setting: Change from

After loading Data in the destination

(the default) to

While reading Data

[PostgresSourceConfigurationSpecification.kt]. • How it helps consume retained WAL: It instructs Airbyte to flush the processed LSN back to Postgres earlier while reading data, rather than waiting for the entire batch to load into the destination [PostgresSourceConfigurationSpecification.kt]. • Trade-off: If a downstream failure happens while loading into the destination, the next run will be forced into a full refresh [PostgresSourceConfigurationSpecification.kt]. 2. Debezium Heartbeat Action Query (Source Connector Setting) • Setting: Configure the

Debezium heartbeat query

property alongside an

airbyte_heartbeat

table in your publication [Troubleshooting Postgres Sources]. • How it helps consume retained WAL: When there are periods with few or zero updates to the published tables (such as your off-hours), Debezium filters out events from other database activity, preventing offsets from being committed and causing a growing backlog of non-removable WAL segments [Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. The heartbeat periodically writes an artificial event to the publication table, enabling the replication slot to acknowledge processed WAL offsets and advance

confirmed_flush_lsn

[Forum Post: Stack Syncs and Automated Full Resyncs on CDC; Troubleshooting Postgres Sources]. 3. Sync Frequency (Connection Schedule) • Setting: Increase the schedule to sync every 1–4 hours [CDC best practices]. • How it helps consume retained WAL: In Airbyte, the replication slot does not acknowledge/purge WAL until syncs run and commit [Forum Post: Postgres Source replication slot safe_wal_size only reset when a change occurs; Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. Syncing frequently prevents the 1 million updates/hour from piling up and keeps WAL moving through the pipeline in manageable intervals [CDC best practices]. 4. Database-Level Parameters (Postgres Configuration) •

max_slot_wal_keep_size

/ `wal_keep_size`: ◦ While this does not directly "consume" WAL, increasing these parameters or setting

max_slot_wal_keep_size

to

-1

(unlimited, disk-bound) provides the database with sufficient headroom so the replication slot is not marked as

lost

before the connector has a chance to catch up and consume the backlogged WAL [Troubleshooting Postgres Sources; Github Issue: CDC sync is making the confirmed flush LSN]. _(Note: Settings such as

initial_waiting_seconds

and

initial_load_timeout_hours

control sync timeouts and schema wait windows, but they do not increase the rate at which retained WAL is consumed.)_ 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Deos

While reading Data

work during full refresh? Usually the pointer stays with

After loading Data in the destination

Thinking completed Based on the provided knowledge sources, there is no explicit mention of whether the

While reading Data

setting actively flushes the LSN during the initial snapshot (full refresh) phase. However, the knowledge sources provide the following relevant technical context about how the connector behaves during initial snapshots: 1. Absence of streaming offset positions during snapshots: During the initial snapshot phase, streaming LSN fields like

lsn_proc

and

lsn_commit

are not present in Debezium's offset context—they are only written during the incremental CDC streaming phase [fix(source-postgres): fix CDC replication slot LSN not advancing]. 2. Replication slot advancement timing: The replication slot's LSN is generally advanced at the start of subsequent syncs rather than mid-snapshot, ensuring that baseline data and state messages have been completely processed and verified [fix(source-postgres): fix CDC replication slot LSN not advancing; Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. 3. Commit Behavior Definition: The specification documentation notes only that

While reading Data

flushes the LSN of processed WAL logs while reading, but warns that if a failure occurs while loading data downstream into the destination, the subsequent sync will be forced into a full sync [PostgresSourceConfigurationSpecification.kt]. Because the replication slot must hold WAL retention from the moment the snapshot begins in order to capture changes that occur concurrently with table reads, the pointer is typically held until the snapshot phase concludes and incremental CDC processing begins [Explore Airbyte's Change Data Capture (CDC) synchronization; fix(source-postgres): fix CDC replication slot LSN not advancing]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai "The replication slot's LSN is generally advanced at the start of subsequent syncs rather than mid-snapshot" does this mean I need to set initial wait time longer for airbyte to consume enough amount of wal?

Thinking completed No, setting the initial wait time longer will not help Airbyte consume more WAL. According to the documentation,

initial_waiting_seconds

(Initial Waiting Time) does not determine how much WAL is processed or speed up consumption [Documentation Integrations > Sources > Postgres > Troubleshooting Postgres Sources>; CDC best practices]. Its purpose is strictly: 1. During the snapshot phase: Setting the time limit for building schema structure and capturing baseline data [CDC best practices]. 2. During the CDC incremental phase: Setting how long the connector waits when it launches to detect whether there are new change events/records to sync before it determines there is no data [PostgresSourceConfigurationSpecification.kt; CDC best practices]. The documentation notes that setting this parameter to a longer duration simply results in slower syncs [Documentation Integrations > Sources > Postgres > Troubleshooting Postgres Sources>]. How the LSN advancement actually works The fact that the replication slot LSN advances at the start of subsequent syncs means: • Airbyte needs at least two successful syncs to see the replication slot advance [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. • Sync 1 reads the snapshot/changes, writes the records to the destination, and records the final state [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. • When Sync 2 starts, the connector verifies that the previous state was successfully recorded and commits/flushes the slot's LSN to Postgres, allowing Postgres to finally purge the retained WAL up to that point [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. To clear retained WAL, what you need is for the first sync to finish and for a subsequent sync to run successfully, not a longer initial wait time [Github Issue (closed): [source-postgres] not flushing lsn back to Postgres after sync]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Give me troubleshooting steps when syncs are not advancing LSM at all

Thinking completed Based on the provided knowledge sources, when Postgres CDC syncs are not advancing the Log Sequence Number (LSN) or

confirmed_flush_lsn

, follow these troubleshooting steps: 1. Run at Least Two Consecutive Successful Syncs By design under the default

After loading Data in the destination

setting, Airbyte advances the replication slot at the start of the subsequent sync, not at the end of the current one [Github Issue: not flushing lsn back to Postgres after sync]. • If you have only run the initial sync, the slot will not yet show an advanced LSN. • Any failed or cancelled syncs in between will also prevent the LSN from advancing. Ensure that at least two syncs complete successfully without errors [Github Issue: not flushing lsn back to Postgres after sync]. 2. Verify Events Are Being Generated in Publication Tables (Heartbeat Setup) If your database has low transaction volume or if transactions occur on tables not included in your Airbyte publication, Debezium may filter out all messages [Troubleshooting Postgres Sources; Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. When all events are filtered out, offset commits are prevented, causing the replication slot to stall [Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. To fix this: 1. Create a heartbeat table in your database schema:CREATE TABLE airbyte_heartbeat ( id SERIAL PRIMARY KEY, timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp, text TEXT );[Troubleshooting Postgres Sources] 2. Add it to your Airbyte publication:ALTER PUBLICATION publicationName ADD TABLE airbyte_heartbeat;[Troubleshooting Postgres Sources] 3. In your Postgres source settings in Airbyte, expand optional settings and configure the Debezium heartbeat query:INSERT INTO airbyte_heartbeat (text) VALUES ('heartbeat');[Troubleshooting Postgres Sources] (Alternatively, you can temporarily add a high-volume table to your publication to force LSN progression [Troubleshooting Postgres Sources].) 3. Check Replication Slot Status and Lag in Postgres Inspect the replication slot in PostgreSQL to see if the connector is falling behind or if the slot has been invalidated:

SELECT slot_name, active, restart_lsn, confirmed_flush_lsn,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots
WHERE slot_type = 'logical';

[Github Issue: not flushing lsn back to Postgres after sync; Github Issue: CDC sync is making the confirmed flush LSN] • If the replication slot status has become

lost

or invalid, the connector cannot recover on its own; you will need to drop and recreate the slot and reset the connection state [Github Issue: Invalid CDC Position Behavior not working]. • Compare the database's

confirmed_flush_lsn

with the saved state in Airbyte. If Airbyte's saved LSN is behind the slot's confirmed LSN, syncs can fail with

Saved offset is before replication slot's confirmed lsn

[Github Issue: CDC sync is making the confirmed flush LSN]. 4. Inspect Connector Logs for LSN Commit Activity Check the orchestrator/source logs at the start of a subsequent sync for: • Messages containing

"Attempting to commit up to"

or

"Successfully committed"

to confirm whether the connector is trying to advance the slot [Github Issue: not flushing lsn back to Postgres after sync]. • Warnings such as:Received 10001 events which were all filtered out, so no offset could be committed.This indicates that WAL is accumulating because non-replicated tables are generating WAL that Debezium must discard [Forum Post: Stack Syncs and Automated Full Resyncs on CDC]. • Note that

flush.lsn.source = false

is expected in the logs—Airbyte disables Debezium's native auto-flush to ensure at-least-once delivery, managing the LSN flush independently [Github Issue: not flushing lsn back to Postgres after sync]. 5. Check Connector Version for Known Bugs Multiple versions of

source-postgres

had known issues where

confirmed_flush_lsn

failed to advance or state was improperly emitted: • A bug existed after the initial snapshot where

lsn_proc

and

lsn_commit

were absent from Debezium's offset, causing the slot advance method to silently exit without advancing [fix(source-postgres): fix CDC replication slot LSN not advancing]. • Multiple intermediate versions (e.g.,

3.6.28

through

3.7.0

) suffered from state LSN mismatches where the sync failed to track the slot properly [Github Issue: CDC sync is making the confirmed flush LSN]. • Ensure you upgrade your Postgres connector to version

3.7.2

or later (or `3.8.x`+), which includes fixes for the missing LSN fallback and socket flush race condition [Github Issue: CDC sync is making the confirmed flush LSN; fix(source-postgres): fix CDC replication slot LSN not advancing]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.