Hi everyone, Please help to understand how the Pos...
# replication-troubleshooting
d
Hi everyone, Please help to understand how the Postgres Source with CDC suppose to work I have done the following (according to connector readme): • Created identity for all tables I’m going to use with Incremental | Dedeped + History sync strategy • Created publication and included all tables I want to sync (those which for Incremental as well as some for Full Refresh) • Created publication slot • Set connector details in the Airbyte • On the other end I use Amazon Redshift with S3 Staging option Initial sync took 25h
85.24 GB | 271,975,505 emitted records | 271,975,505 committed records | 25h 2m 5s | Sync
For the subsequent sync operation I expected to take less, because it should use publication slot to get only new entries for tables where Incremental | Deduped + History is selected, and rewrite completely those entries where Full refresh | Overwrite is chosen, But I don’t understand (missed the doc?) why the subsequent sync is taking even longer (still in progress since yesterday) and re-scanning all data from the source. UPDATE: The issue seems to be in another dimensions We hit Airbyte performance limit, so we have increased to more powerful CPU and increased Network throughput Also the Redhisft initial single-node has 100% storage full, so we are switching to multi-node setup with automatic storage scaling
Screenshot 2022-08-21 at 20.57.50.png
All the tables here are part of the publication
The Postgres is and AWS RDS Instance
The paramter_group was set to enable replication and instance rebooted prior to starting the sync
The reason I mix Incremental and Full refresh is I’ve chosen to use incremental for tables with many rows, but for utility tables (usually <2k rows) I want simply refresh data in destination