Hi, I wanna ask about incremental+dedup loading ca...
# replication-troubleshooting
y
Hi, I wanna ask about incremental+dedup loading case. I have upgraded Airbyte from 0.39-alpha to 0.40.28 for log retention feature. This case is started when using the newest version. Sync incremental+dedup always scan/emit huge data size (almost same with reset-sync operation or full-refresh mode) every running from source (Postgres). The destination (BigQuery) is seems good (contains expected data). Sadly, this big load made our VM’s CPU is on spike with more time-consuming process every time. So far, I found that huge emitting data case happens on multi tables/streams (in the same job/connection). Otherwise, single table/stream (in one job/connection) has expected pattern (smaller number of data emittance). *NB, I have tried refreshing schema, resetting, deleting all staging table manually, but the multi-stream connection still emit/scan huge data on incremental+dedup mode. The single stream connection is working as expected probaly because there is schema change in the source when resetting or refreshing the connection.
n
Hi, Yogic! The initial sync should always be a full refresh one, so it would cause high sync times/data usage with multiple streams. Let me look into this and I'll have some more info/ideas for you soon!
y
Hi @Nataly Merezhuk (Airbyte), we have been through initial sync and reset/refresh the connection but the data reading and emittance is quite big every running (we trigger the connection every 2 hours via Dagster, the size almost same with initial sync or non-incremental mode).
n
I'm looking at the logs and see that the connectors are out of date. Could you update them as well? That might clear up the issue, and if it does not - we'll investigate further!
y
Ok, will try it.