Hi all, running into a persistent CDC issue with

source-postgres:3.7.2

and hoping someone has seen this. Problem: After a full refresh completes successfully, the very next incremental CDC sync loads 0 records, and this repeats every time. Root cause (probably): The full refresh does the ctid snapshot correctly but saves the initial WAL LSN as its final state (the LSN at snapshot start, not at snapshot end). So on the next incremental run,

confirmed_flush_lsn == saved_offset_lsn

, Debezium sees itself as caught up, skips the snapshot, streams only heartbeats, and exits with 0 records. All WAL changes between snapshot start and the next sync run are lost. This seems related to

flush.lsn.source = false

, the slot LSN never advances during the ctid load so the saved state never moves forward. Question: Is this a known bug in 3.7.x? Is there a patched version or a recommended workaround short of doing a manual refresh every day? @kapa.ai

Sorry @Amal Mokhtar, I'm not enabled for this channel yet. Please contact your workspace admins to help get me set up. 👋

That is not a known issue. The initial WAL LSN is the correct one to resume from, as we don't lock all the tables during snapshot, and data can be written while snapshotting. In our connector, neither Debezium nor the replication slot are used as the source of truth for when to terminate the incremental sync. We consume the data emitted by Debezium, and when the target LSN is reached, we terminate it.

@Matt Bayley Thanks a lot for the clarification. But then what could be causing Debezium to only emitting heartbeat records and no actual data records during the incremental sync? The WAL gap exists but no row changes come through. As a workaround, i had to change the loading method to Full Refresh | Overwrite + Deduped which is not ideal. Please check screenshot attached. CC: @Maya Zaher

If every incremental sync is producing 0 records, I suspect a configuration or permissions issue.