@kapa.ai I'm having a problem with

source-postgres:3.8.1

. Following a successful sync, apparently the

lsn

is not being flushed to Postgres. I'm using the following query in Postgres to keep track of the lag in the given replication slot:

source=# SELECT slot_name, active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots;
  slot_name   | active |  lag
--------------+--------+-------
 airbyte_slot | f      | 65 kB
(1 row)

The lag increases as I add more data in the source. After I do one (or more) syncs, it never goes down. Would that make Postgres keep WAL files forever?

The knowledge sources don't explicitly mention version 3.8.1, but this issue is well-documented for versions >= 3.6.19 and is closely related to known bugs in the Postgres CDC connector. What's Happening The lag not decreasing after syncs is a known issue where the

confirmed_flush_lsn

is not being properly advanced (or is being advanced incorrectly), meaning Postgres cannot safely remove WAL files. Yes, this can cause Postgres to accumulate WAL files indefinitely, which risks filling your disk. [CDC LSN Issue] The root causes identified are: 1. A regression in PgJDBC 42.7.0+ (used by the connector since v3.6.19): PgJDBC automatically advances

confirmed_flush_lsn

when there are no new events for publication tables, even if the WAL contains changes for tables outside the publication. This misaligns the LSN state. [CDC LSN Issue] 2. Low transaction volume on tracked tables: If your publication tables have few changes, the replication slot's

confirmed_flush_lsn

may not advance as expected. [WAL Disk Consumption] Recommended Workarounds 1. Set up a heartbeat table (most effective) This is the recommended fix for low-traffic databases:

CREATE TABLE public.airbyte_heartbeat (
    id SERIAL PRIMARY KEY,
    timestamp TIMESTAMP NOT NULL DEFAULT current_timestamp,
    text TEXT
);

ALTER PUBLICATION <your_publication_name> ADD TABLE public.airbyte_heartbeat;

Then in the Airbyte Postgres source connector UI, under Advanced → Update Method → Optional fields → Debezium heartbeat query, enter:

INSERT INTO public.airbyte_heartbeat (text) VALUES ('heartbeat');

This keeps the LSN advancing even when your tracked tables have little activity. [WAL Disk Consumption] [CDC LSN Issue] 2. Add a high-volume table to the publication (read-only option) If you don't want to grant write access, add a high-traffic table to the publication (you don't need to sync it):

ALTER PUBLICATION <publicationName> ADD TABLE <high_volume_table>;

This advances the LSN naturally. [WAL Disk Consumption] 3. Check if a connector downgrade helps Community reports indicate that v3.6.18 does not have this issue, while versions >= 3.6.19 do. A fix was reportedly included in v3.7.2. Since you're on 3.8.1, it's possible the fix is present but the heartbeat configuration is still needed for low-traffic scenarios. [CDC LSN Issue] Preventing WAL Accumulation In the meantime, you can set

max_slot_wal_keep_size

to limit how much WAL Postgres will retain for the replication slot (at the risk of the slot being invalidated if it falls too far behind):

-- Example: limit to 2GB
ALTER SYSTEM SET max_slot_wal_keep_size = '2048';

The heartbeat table approach is the most robust long-term solution for your situation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai on subsequent tests, sometimes following a sync the lsn was flushed to Postgres, making the lag decrease, sometimes not. What is the logic that defines when it's flushed back to the database?

The knowledge sources do not contain a complete, definitive explanation of the exact logic governing when the LSN is flushed, but they do provide enough detail to piece together the key factors: Why the LSN Flush Is Inconsistent 1. "After loading data in the destination" mode requires two syncs With the default

LSN commit behavior

set to After loading data in the destination, the replication slot is advanced at the start of the next sync, not at the end of the current one. So you need at least two successful syncs to observe the slot advancing. [LSN Not Flushing] 2. PgJDBC race condition (Bug 1 in v3.8.1) A race condition was identified in `ReplicationSlotManager.advanceLsn()`:

forceUpdateStatus()

only flushes the status update to the TCP socket, but the PostgreSQL server needs time to process it. The previous code closed the replication stream immediately, which could discard the in-flight update before the server applied it — explaining why the flush sometimes works and sometimes doesn't. [LSN Fix PR] 3. Missing LSN fallback after initial snapshot (Bug 2) After the initial snapshot phase, the fields

lsn_proc

and

lsn_commit

are absent from the Debezium offset. When both are null, the

advanceReplicationSlot()

method silently returns without advancing the slot at all. This is why post-snapshot syncs are particularly unreliable. [LSN Fix PR] 4. PgJDBC 42.7.0+ auto-advancing the LSN incorrectly PgJDBC 42.7.0+ (used since connector v3.6.19) automatically advances

confirmed_flush_lsn

when there are no new events for the publication tables — even if the WAL contains changes for tables outside the publication. This can cause the slot's LSN to jump ahead of Airbyte's saved state, making subsequent syncs inconsistent or failing. [CDC LSN Issue] Summary The inconsistency you're seeing is essentially the combination of these factors: A fix for the race condition and missing fallback was submitted in PR #79671, targeting the exact version (3.8.1) you're using. In the meantime, the heartbeat table workaround remains the most reliable way to keep the LSN advancing consistently. [LSN Fix PR] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements