Hi all - Our team is looking into getting CDC data...
# help-connector-development
a
Hi all - Our team is looking into getting CDC data from Postgres via AirByte. We notice here: https://github.com/airbytehq/airbyte/blob/e8ddda970905ae338377d2b2a7ed5fa9ffa57ed4[…]airbyte/integrations/debezium/internals/DebeziumEventUtils.java, that in case of
UPDATE
operations, AirByte strips out the old values even if they are received from Debezium (assuming
REPLICA IDENTITY
is set to
FULL
). I am curious why we do that. We are happy to contribute a patch that inserts the old values into a new field (let’s say
_ab_cdc_before
) in the payload, but would that be welcomed? If not, I’m curious to understand the rationale. Thanks!
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. The extracts provided do not contain specific information about why Airbyte strips out the old values in case of
UPDATE
operations, even when they are received from Debezium. However, I can provide some general information about how Airbyte handles CDC data from Postgres. Airbyte uses [Debezium](https://debezium.io/) internally to support CDC and uses [logical replication](https://www.postgresql.org/docs/10/logical-replication.html) of the Postgres write-ahead log (WAL) to incrementally capture deletes using a replication plugin. For each table you want to replicate with CDC, you need to add the replication identity first, then create a publication for the table. The Airbyte UI currently allows selecting any tables for CDC, but if a table is selected that is not part of the publication, it will not be replicated even though it is selected. If a table is part of the publication but does not have a replication identity, that replication identity will be created automatically on the first run if the Airbyte user has the necessary permissions [[source](https://docs.airbyte.com/integrations/sources/postgres)]. For more specific information about your question, I would recommend checking out the [Airbyte documentation](https://docs.airbyte.com/) or asking in the [Airbyte community](https://discuss.airbyte.io).