Hi all. I’m running a postgres-to-postgres connection. I have a lot of tables, but one in particular is rather large (12mil+ records). For this reason I have set it up for incremental sync using a cursor field. What I’ve noticed however, is that the second sync was much slower than the first. This time is spent in normalization. It seems that although reading from source is only the new records, the normalization is done for the whole 12mil anyways, which causes the target instance that runs in AWS to run out of IOPS credits and slow to a crawl. Is this normal behavior? There is also no JSON to unpack or anything in this table.

2022-10-15 00:08:27 normalization > 128 of 132 OK created incremental model <table name removed> ....................................... [INSERT 0 12446501 in 17788.85s]

Hi Steven, my understanding is that the records from the current sync and the previous sync will be normalized. This is discussed in this tutorial: https://airbyte.com/tutorials/incremental-data-synchronization

This code selects rows from table_two_scd which are considered active as indicated by  _airbyte_active_row value of 1 only if the value of _airbyte_emitted_at is greater than or equal to the highest _airbyte_emitted_at value that has already been written to table_two. Again, this means that records from the current sync and the previous sync will be normalized.

Therefore, I would expect the next sync after the 2nd sync to be less heavy. CC: @Chris Duong [Airbyte].

Thanks for the reply @Alex Marquardt, when you say previous sync, do you mean all previous syncs or just the preceding one?

I am referring to the very last (previous) sync run. Not all previous sync runs - see: https://airbyte.com/blog/understanding-data-replication-modes for a definition of a sync run

Will run it soon!

I think you can look in the logs presented in the UI to see which “cursor” value is currently being executed upon. This would give some indication if only documents from the current and previous sync run are being executed on. If it is still running across the entire data set, then you should confirm that the field that you have chosen for the cursor satisfies requirements for the cursor (i.e. new records always have a higher value in this field than previously synced records)

@Alex Marquardt It took 10 minutes now 🙂

Awesome! Great news!

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2862 to ensure timely resolution!