Hi, I wanna ask about incremental+dedup loading case. I have upgraded Airbyte from 0.39-alpha to 0.40.28 for log retention feature. This case is started when using the newest version. Sync incremental+dedup always scan/emit huge data size (almost same with reset-sync operation or full-refresh mode) every running from source (Postgres). The destination (BigQuery) is seems good (contains expected data). Sadly, this big load made our VM’s CPU is on spike with more time-consuming process every time.
So far, I found that huge emitting data case happens on multi tables/streams (in the same job/connection). Otherwise, single table/stream (in one job/connection) has expected pattern (smaller number of data emittance).
*NB, I have tried refreshing schema, resetting, deleting all staging table manually, but the multi-stream connection still emit/scan huge data on incremental+dedup mode. The single stream connection is working as expected probaly because there is schema change in the source when resetting or refreshing the connection.