Heya! I have a question regarding Airbyte + dbt We have an important table in Google sheet with product information that get's updated once every week where new rows get added and others deleted/changed ( and the schema changes sometimes). And the goal is to use airbyte + dbt on a posgres database to save all historic data for downstream aggregation and analysis My first thought was to sync tables by

full refresh - append

but this already broke because someone added a new column and the sync schema didn't stay the same so we need to stick with the

overwrite

option. Does anyone have a suggestion on how to properly archive/store all data with dbt? My idea was to use the

snapshot

feature of dbt as it can also add new columns that don't fit with the created schema. Thank you!

The new column is important to the project? Maybe create a new sheet to only replicate the columns are necessary to the project. So the original sheet you can add new columns without any problem. Maybe using the snapshot in postgres is a way to store the historical data too. 🙂

Mhm the thing is that new columns might become important down the line so a full backup might be a good idea. Sadly the communication about that is lacking. But yes I will try it with the snapshot feature, thank you!