For those of you who run Airbyte together with dbt: how do you run dbt? on dbt cloud or on e.g. GCP Cloud Run, or it's own GCE instance, or same instance as Airbyte?

have you seen the docs to run your custom dbt in airbyte? https://docs.airbyte.io/operator-guides/transformation-and-normalization/transformations-with-airbyte

tl;dr you import your dbt package in the normalisation phase (irrespective of how your actual dbt project is deployed)

thanks guys. what I don't really grasp atm is since I will have one git repo with my dbt transformations. Do I add this repo to every single connection? E.g. Let's say I have these source, all ending up in Bigquery tables: • Google Ads • Google Analytics • Bing • Facebook • Sheets And my transformation is to do some data cleaning and union them. Will this dbt git repo be added to every single connection?

yes you can add it to every connection you need it run from

This assumes a 1 to 1 relationship no? e.g. load google ads, run transformations for google ads. I am guessing Jonas may be trying to do something more complex like loading a few sources and running transformations that depend on multiple sources e.g. google ads + bing and then dbt. I think in that case you would need to orchestrate with something like Airflow.

in the tutorial i linked first, there is also an example of how to run dbt partially on his private repo if that helps

I am using a single dbt repo separate from my analytics dbt repo to handle normalization. I only call models for the source tables when the source runs, in the 1:1 manner Noel described.

Have both 1:1 transformations and more complex ones like you said @Noel Gomez @Jeff Crooks so have repo with multiple models and use commands like these to call on specific models

run --models google_ads

? where do you host your DBT docker container that Airbyte calls on?

yeah + path argument you can select folders, or tags, whichever you prefer