Hello folks, I am using Airbyte open source. What...
# replication-troubleshooting
a
Hello folks, I am using Airbyte open source. What I’m trying to do: 1. Ingest data from some sources ( full refresh, I want to check for updates as well) 2. Do some minor transformations on the data 3. UPSERT data into destination What I think would work: • I use DBT module to write queries that would UPSERT the data at destination instead of overwriting What I need help with: 1. I am confused when and where does the SQL generated by DBT execute a. Does it run on the data from the source b. Does it run on the raw data copied to the destination c. Does it run after the data is copied to destination (after normalization) Please help. thank you
a
please refer this doc, it might help you to answer your questions
a
• I’ve gone through those docs before, they mention that it only transforms the data coming form the source. • But here, Fredehagelund92 mentions that it is possible to do incremental syncs without a cursor field using DBT • I’m using the sheets connector which does not support incremental syncs • Hence I’m trying to do that with DBT • Without accessing the destination, how do I compare to old data
u
If you're using full refresh normalization will run in the
raw
tables generated by Airbyte after the sync finished.
a
After the normalization is run on the
raw
tables does airbyte use the
dbt
generated SQL to write to the destination table? Can this SQL be modified to do UPSERT operation?
a
Yes, you can use custom DBT for this for more info
a
Thank you very much for the explanation. Although I now have another question, What happens to this process when I’m using a custom connector to call APIs instead of writing to a DB? Is there a place where I could read this in-depth?
u
Ayushman please read Airbyte docs about connectors: https://docs.airbyte.com/connector-development/