Hi all, looking for a bit of advice regarding data ingestion.
(edit: not native english so hope it makes some sense)
I correctly set up a shopify to bigquery connection. I usually prefer to use the bigquery dts destination to have a single table, however I cant find incremental + dedup option, so, to avoid to dedup data later I could do a full ovewrite at each refresh. OR I could use the bigquery destination, set up incremental + dedup, but I would find myself with dozens of shopify tables in my bigquery table.
My question is:
• should I am for a clean bigquery table and go for full overwrite?
• should I aim for incremental + dedup, and nevermind if I have to scroll on dozens of tables in bigquery.
Sync is once every 24 hours, shop is not massive (live 40MB of data from the sync data).