Hi , Can you please explain more about "WAL-based CDC using logical decoding (via replication slots) to stream changes going forward" what's really happening here i want to know the internal working is it possible? @kapa.ai

Airbyte’s docs describe this at a conceptual level, but they don’t go into deep internal implementation details. I’ll summarize what is documented and clearly flag where the information stops. ### 1. What “WAL‑based CDC using logical decoding” means Many databases (including Postgres) write every change to a transaction log / WAL (Write‑Ahead Log). This log records `INSERT`/`UPDATE`/`DELETE` operations so they can be replicated or recovered later.[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc); [CDC sync tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)] Logical decoding is a Postgres feature that can read the WAL and turn its internal binary records into a logical stream of row‑level changes. Airbyte uses this to implement CDC.[[Postgres CDC tutorial](https://airbyte.com/tutorials/postgresql-change-data-capture-elt); [Logical decoding plugins](https://airbyte.com/tutorials/postgresql-change-data-capture-elt)] ### 2. Role of replication slots and plugins For Postgres sources, Airbyte requires: - A logical replication slot created with a plugin such as `pgoutput`: ```sql SELECT pg_create_logical_replication_slot('airbyte_slot', 'pgoutput'); ```[[Cloud SQL Postgres CDC](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#advanced-setup-using-cdc); [CDC sync tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)] - A publication listing the tables whose changes should be streamed: ```sql CREATE PUBLICATION cdc_tut_pub FOR TABLE cdc_tut; ```[[CDC sync tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)] The docs explain the purpose of these pieces: - The replication slot “holds the state of a replication stream” so Postgres knows how far a consumer (Airbyte/Debezium) has read and which WAL segments must be retained until they are consumed.[[CDC sync tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)] - The logical decoding plugin (e.g.

pgoutput

,

wal2json

) converts the WAL’s internal representation into a format that external tools can understand.[[Logical decoding plugins](https://airbyte.com/tutorials/postgresql-change-data-capture-elt)] ### 3. How Airbyte uses this for CDC From the Airbyte side, the documented behavior is: 1. Initial sync (snapshot) - Airbyte does not use WAL for the initial load. - It runs

SELECT

queries to take a snapshot of the current table contents; this is “effectively a Full Refresh”.[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] - Changes that happen during this snapshot are not read from WAL. 2. Subsequent syncs (log‑based incremental) - Airbyte “reads data from the previously synced position in the logs up to the start time of the sync.”[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] - It keeps track of the current log position (e.g. LSN) between syncs.[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] - Only changes made via `DELETE`/`INSERT`/`UPDATE` are captured; operations like `TRUNCATE`/`ALTER` are not.[[CDC limitations](https://docs.airbyte.com/platform/1.6/understanding-airbyte/cdc)] 3. What the records look like - Airbyte outputs records for `INSERT`/`UPDATE`/`DELETE` in a unified format and adds CDC metadata columns such as: -

_ab_cdc_lsn

/ `_ab_cdc_cursor`: log position where the record was retrieved. - `_ab_cdc_updated_at`: transaction timestamp. - `_ab_cdc_deleted_at`: present only for deletes.[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] 4. Orchestration model - CDC sources are not treated as infinite streams; Airbyte runs discrete syncs on a schedule and, in each sync, reads from the last stored log position up to “now”.[[CDC overview](https://docs.airbyte.com/platform/understanding-airbyte/cdc)] ### 4. What’s not fully documented The knowledge sources do not provide: - Low‑level details of how Debezium is invoked, how exactly it maps WAL messages to Airbyte records, or how internal offsets are stored. - A step‑by‑step description of every internal component in the Airbyte Postgres connector. They only state that: - Airbyte uses Debezium internally for CDC.[[CDC sync tutorial](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication)] - It relies on logical replication of WAL via a replication slot and logical decoding plugin (e.g.

pgoutput

).[[Cloud SQL Postgres CDC](https://docs.airbyte.com/integrations/sources/postgres/cloud-sql-postgres#advanced-setup-using-cdc); [Logical decoding plugins](https://airbyte.com/tutorials/postgresql-change-data-capture-elt)] So if you’re looking for a full internal architecture diagram of the connector, that level of detail is not present in these docs. For that, you’d need to inspect the connector’s source code and Debezium’s own documentation, which go beyond what’s included in the provided knowledge sources.