Hi, I'm building a postgres2postgres sync. If I synced some of our very large tables (tens of GB), even if I used incremental mode, the initialization would take several days, end every time we changed the underlying structure, the sync refresh would take several days again. If I have an already existing data on destination Postgres, is it possible to use them to "pre-fill" the target structures to avoid complete refresh of sync? In other words, can I initiate the incremental sync manually to let the Airbyte do just the future increments? Thanks!

Hey @Svatopluk Chalupa, thanks for the question. I don’t know if this is possible but have you tried looking to CDC? https://docs.airbyte.com/understanding-airbyte/cdc/ Perhaps it may be a good choice for your use case. I’m not entirely sure if you can avoid full refresh for incremental syncs but I could be wrong.

Hi, thanks for suggestions, I'm aware of the CDC, we used Logical Replication of Postgres before. It's not this case however. It wouldn't help in case of source table change anyway, I suppose. Which is the main point of my question - how to handle the typical change management issue, when source table changes. Do I need to update catalog, reset sync and refresh ALL the data? Or is there a way how to reuse previous data on destination?

Hey @Svatopluk Chalupa, thanks for the clarification. I see that CDC wouldn’t help in this case. Unfortunately, with schema changes, you’ll have to reset the data as we don’t support schema evolution for the Postgres connector (at the moment).

OK, thanks for confirmation.