https://linen.dev logo
#ask-ai
Title
# ask-ai
h

Hicham Rahj

08/25/2021, 8:08 AM
Hello, i have a question, i've seen that airbyte always create a destination with name "_airbyte_raw" is there any scenario where is doesn't create it ? and is it possible to delete them after the jobs are run or will this create a problem during the futur syncs ?
m

mauro

08/25/2021, 8:46 AM
from what I understand,
_airbyte_raw
is where incoming data gets inserted. if you use incremental it keeps the old data + new data. then the normalization recreates the final table(s) based on all the data present (it's not incremental). I think if you delete the _raw data you will then lose it on the next sync.
h

Hicham Rahj

08/25/2021, 9:06 AM
so that's a problem if we have a huge amount of data we'll twice the necessary size all the time
g

George Claireaux (Airbyte)

08/25/2021, 1:18 PM
Hey @Hicham Rahj, if you post these kind of Qs in the
troubleshooting
channel, it's more likely to get picked up by community members and Airbyte team members 😄
👍 1
In answer to your Q, it depends. For example normalization currently requires all the history of _raw tables so if you're using normalization with incremental then you'll need to keep them there. I think if you're doing full refresh then you could delete them but @Chris (deprecated profile) correct me if I'm wrong?
👍 1
h

Hicham Rahj

08/25/2021, 1:28 PM
will keep that in mind, i didn't post it there because i thought it's just a general question to understand airbyte not really a problem i'm facing. I want to use incremental so i guess i'll have to keep it. Are you thinking about changing this behavior? because if some data is already transformed, I don't see why it should be kept in _raw format unless it's for recovery purposes but i don't think that's why it's kept in this case.
Or for incremental syncs but in that case keeping only the cursor field would be more efficient
3 Views