Source: Shopify Destination: Postgres Hello! I have a question regarding duplicate rows. I am using deduped history sync mode, however there are still duplicate rows in orders_line_items. The parent table "orders" doesn't contain duplicate rows, the child table "orders_line_items" contains duplicates. What would be a good approach to filter out duplicate rows? I have some ideas in mind using dbt models: 1. Use

id

as a filter

SELECT DISTINCT ON (id)

2. Use

_airbyte_line_items_hashid

as a filter ? Is it safe to use this approach?

SELECT DISTINCT ON (__airbyte_line_items_hashid_)

Hey can you help with the airbyte version, source version and destination version you are using. Also is the duplication only happening line_items or other tables also?

@Harshith (Airbyte) Airbyte Version: 0.35.42 - alpha Source: Shopify 0.1.31 Destination: Postgres 0.3.13 I have checked orders_fulfillments (child table of orders) there are also exact same duplicate behaviour.

Hi @Andrei Batomunkuev is this source offering incremental dedup mode? What's your current sync mode?

@Augustin Lafanechere (Airbyte) I have set incremental dedup mode

@Andrei Batomunkuev https://discuss.airbyte.io/t/incremental-sync-successful-but-rows-in-nested-table-are-duplicated/248/2

@Marcos Marx (Airbyte) Thank you. I have looked at the shared link. I have gone through AirByte documentation and found information about extracting AirByte dbt models (https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt#exporting-dbt-normaliza[…]project-outside-airbyte) I will update you how it goes.

Sure get back to us if you need some help