Hey! I have a question about fields that are synced as Incremental | Deduped + History and are normalized on some of the fields: • the top level, say

table

is indeed deduped, and that’s nice • there is a

table_scd

for the history, and that works fine as well, every entry has a unique

_airbyte_ab_id

as expected • The normalized table (say normalizing a

properties

field), ends up as

table_properties

as expected for this example, but the content is different from what I’ve expected to some respect: The differences and questions are: • the contents of

table_properties

are normalized from

table_scd

(as confirmed by looking at the generated dbt code) rather than as expected from

table

, so end up with a loads more entries than expected. I would have expected

table_properties

to be deduped after the normalized just as

table

was before. Am I expecting something wrong? • in

table_properties

there’s actually a lot more entries even than

table_scd

because a single entry seem to potentially (but not always) end up normalized multiple times, only the

_airbyte_normalized_at

field being different, while all the content fields are the same (as are the hashid fields to confirm). Looking at duplicates, I see some some entries normalized multiple times for us up to 8x even (most of the duplicates are 2-3x) on our 1/h schedule. What can cause multiple normalization like this, if the underlying table does have only a single entry?

Have you seen: https://airbyte.com/tutorials/incremental-data-synchronization ?

@[DEPRECATED] Marcos Marx turned this thread into Zendesk ticket 2858 to ensure timely resolution!

Hey @Alex Marquardt (Airbyte) thanks, this is a very useful source, and there’s a lot to learn indeed. As much as I’ve been digging into the “incremental deduped + history synchronization”, it doesn’t answer my actual questions: • the chart just describes a single raw JSON -> normalized conversion, and that makes sense why is it like that • what is unclear, and the example doesn’t seem to touch on is when not just the top level table, fields are normalized as well: as mentioned, they don’t seem to have 2 tables like the top level (SCD / deduped), rather they have a single table that is all the changes + duplicate normalized values are there. This is what I’m not sure why they work like that. That docs page doesn’t seem to show any scenario compared to what I’m seeing, but maybe I’ve missed it? Thanks, I’ll keep digging in anyways, and will be looking closer at the generated DBT in the logs.