Sharath Chandra
02/08/2023, 7:01 PMPierre Zaplet-Brouillard
02/08/2023, 7:44 PMIncremental | Deduped + history (with primary key defined as distinct_id, event, time - unfortunately insert_id is missing for some reason) created an export table which doesn't have duplicates.
I'll have to run it again in the following days to confirm that the data is indeed loaded incrementally, and not fully refreshed on each run.
Also, I noted that there's an intermediate table, _airbyte_raw_export which seems to have twice more records, and therefore worries me about the long-term impact of this strategy.
(I'm working with BigQuery)Sharath Chandra
02/08/2023, 8:06 PMPierre Zaplet-Brouillard
02/08/2023, 8:08 PMinsert_id, which is the Mixpanel defined unique id for events.Sharath Chandra
02/08/2023, 9:29 PMexport and export_scd .
export is doing incremental load correctly.
When you too try, please let me know if it's the same to you as well.Pierre Zaplet-Brouillard
02/09/2023, 1:25 PMexport is indeed loaded incrementally
My second run today shows that some records emitted in yesterday's run were not overwritten in today's run (341 on the screenshot attached)
But I also get a lot of artefacts which I'm not sure I can get rid of. And they're a bit concerning in terms of storage costs:
For instance, _airbyte_raw_export has now 4.7 times more rows than export, and there is no table expiry date set.
So I imagine it is just going to keep on building up.
In a few days, I will try to delete some of those tables and see how it affects the incremental load.Pierre Zaplet-Brouillard
02/09/2023, 1:26 PMdistinct_id, event, time is robust enough.
What do you think @Sharath Chandra?Sharath Chandra
02/09/2023, 1:29 PMPierre Zaplet-Brouillard
02/10/2023, 10:21 AMuser
02/13/2023, 4:39 PM