Hello. I am currently using incremental updates with the postgres source to s3 destination. I have noticed that even though I am using an

update

column on a

pg

table, it seems to read through all data of the table, but only writes the

delta

changes. On some table syncs however, I can see it only reads very small chunks. Is there a particular reason for this behaviour?

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2819 to ensure timely resolution!

A few examples via print screen. An expected sync behaviour and another also incremental but seems to sync all data. I can see that on both, it only writes

delta

changes, so the one which seem to sync all data, is still only appending changes, I am a bit baffled why this is and there must be a reason?

What incremental sync mode are you using for this sync? Can you share a log of one of the syncs that reads all of the data and another log of one of the syncs that only reads a few pieces of data?

Hi, I think I have same case with you. In my case, I have upgraded Airbyte from 0.39-alpha to 0.40.28 for log retention feature. This case is started when using the newest version. Sync incremental+dedup always scan/emit huge data size (almost same with reset-sync operation or full-refresh mode) every running from source (Postgres). The destination (BigQuery) is seems good (contains expected data). Sadly, this big load made our VM’s CPU is on spike with more time-consuming process every time. So far, I found that huge emitting data case happens on multi tables/streams in the same job/connection. Otherwise, one table/stream in one job/connection has expected pattern (smaller number of data emittance). *NB, I have tried to refresh schema, reset, delete all staging table manually, but the multi-stream connection still emit/scan huge data on incremental+dedup mode.

hai @Sajarin Dider any updates?

it's been a while without an update from us. Are you still having problems or did you find a solution?

We have set all connection to be single-stream to prevent this and get neat job separation. So far, everything is running as expected, @Saj Dider (Airbyte). Also can you help us in different topic? We have raised thread here, https://airbytehq.slack.com/archives/C021JANJ6TY/p1679550833414249