Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Hi folks, new on airbyte and tried tweaking some configurations within the connection settings, but how do I make sure that only one normalized table that matches with the original table format, gets dumped in bigquery instead of the airbyte raw data table coming along with it?

Hi, the raw table is part of the sync today and there isn't a way to configure it to not be there. What issues is the raw table causing?

Ah got it, its just that if there are a lot of tables that need to be connected, having two tables with the same data just seems redundant

Its also taking more storage, so this isn't something that can be scaled

if you are using full refresh, the table should be able to be deleted after, otherwise we require information in that table to properly denormalise

we are aware of this and it's something we are looking at

That's weird, I'm running a full refresh - append but the raw table is still there

does it only get deleted if you run an overwrite job?

Ah I should be clearer, I mean that the raw table can be manually deleted

the table will always be there for all syncs

FYI, the `destination-bigquery-denormalized` does not produce a raw table btw but it’s pretty new, freshly released.
So we still have some things to stabilize though: <https://airbytehq.slack.com/archives/C01MFR03D5W/p1625200298367900>

&gt; Ah I should be clearer, I mean that the raw table can be manually deleted
This can also be configured with a custom dbt transformations that cleans the destination up at the end of the sync by deleting these raw tables

You could also take advantage of this to also define your custom partition key/clustering key etc if you wanted/needed to as well for example

Thank you, that was helpful <@U01BV1SDQMA>