Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

*Hello, team! TLDR: Replication constantly growing*
Our setup: Airbyte Open Source deployed on an EC2 cluster
We have a main datasource - Postgres in RDS and a main destination - Redshift. We have already set this pipeline up in Fivetran, and now we are trying to connect Airbyte also. Fivetran connector has been set up using a replication slot with the ‘test-decoding’ plugin.
Now for Airbyte, we have created a new replication slot using **pgoutput** plugin connected with a publication slot that has registered for all the tables, that are being synced to Fivetran. It’s recommended on Airbyte’s Github to set up frequent syncs so that the replication slot is cleared constantly, but however frequent the syncs are, the replication slot just keeps on growing constantly, and is not being cleared. From our investigation, the replication slot seems that has been stuck on the same LSN pointer (see screenshot) for the last several days, and as a workaround we have to clear the slot manually by recreating it :upside_down_face: How can we avoid this workaround , maybe there is something we have to change in the connector setup?
One of the suggestions that we have found is setting up the heartbeat ping from debezium to Postgres, so that was files are marked as processed in the database. How can we check/manipulate debezium parameters? Which file contains the setup? Please tag me for more info - happy to provide more details if necessary.

Version info: Airbyte (4.0.17), Postgres connector (1.0.19)
Tagging my colleague <@U047T3JDZS7> :octavia-wave:

image.png

When I was writing <https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication|this tutorial>, I had issues getting CDC to work with earlier versions of Posgres. I ended up running with PostgreSQL 14.5. However,  I did not dig into the underlying issue as the newer version solved it.

Noted, Thank you! <@U03BB17Q7K2> !
We will try to upgrade PostgreSQL first and will come back with results.
Also small adds: we run it under AWS RDS.

Thank you <@U03BB17Q7K2>! I’ll report back after the update

<@U03BB17Q7K2>, the update helped in a way: we updated to 14.3 on Friday, and since then replication slot growth has been slower than before (about 3x), and for a souple days the LSN pointer was moving, BUT now the problem has returned: the LSN pointer has been stuck on the same LSN pointer for a whole day now, and the replication slot has been growing. I am curious if this is caused by our DB settings: we have set wal sender timeout to 0 as recommended by Fivetran, which we are using alongside Airbyte

I’m not sure what could cause that - I’ve re-sent this thread to the channel to see if anyone else may know what is going on with Postgres

<@U03BB17Q7K2> , marking this as solved. Our main issue was that we tried running Fivetran and Airbyte together in the same db, so the set of required parameters for Fivetran was affecting Airbyte. I’m talking about wal_sender_timeout setting which is recommended by Fivetran to be set to 0. The solution to this for us is to set this back to default and set hourly syncs :relieved: