Hi, I'm trying to use dbt custom transformation and I don't understand why dbt doesnt trigger "dbt deps" before trying to run my commands. I've tried to add 2 custom transformations : first with "deps" and second one with "run" but it looks like the second one is unaware of the first one

You’re a lifesaver! Yep, it looks like each transformation step runs in its own container so without this the subsequent dbt steps won’t see any installed packages. Putting this in my dbt_project.yml fixed it:

packages-install-path: '../dbt_packages'

There really should be something in the Airbyte docs to explains how to use dbt packages…

Hi @Matt Webster imho, the best command is to use dbt build instead of "dbt deps + dbt run" (this is not the same: https://docs.getdbt.com/reference/commands/build), but I'm not sure yet what would be the "best practice"

Reviewed! thanks

@Mickaël Andrieu thanks for pointing out the dbt build option. i’m currently doing deps, seed, run, and test every time with the thought that maybe this is overkill and I could trim some staps out once my system gets more mature. but i’m going to make a note to benchmark this against

dbt build

and see which works better. and glad to see the docs updated too!

Hello, I have followed the tutorial here , but packages installed during first transfo (dbt deps) still aren't located in the next one (dbt run --select model) Our dbt_project.yml already has :

packages-install-path: 'dbt_packages'

therefore, I've tried to override its value at runtime using: 1.

deps --vars '{"packages-install-path":"../dbt_packages"}'

2.

run --select models --vars '{"packages-install-path":"../dbt_packages"}'

when parsing the error log, I can see the following arguments get added to the two runs

--profiles-dir=/data/35/0/transform --project-dir=/data/35/0/transform/git_repo

based on the custom path added for packages-install-paths, the package should be located in

/data/35/0/transform/dbt_packages

, but perhaps this directory get wiped after each transformation, which leads to the error during step 2?

You gave me an idea: what if you set the path as a variable in your configuration ? Something like (the name of the var is shitty, and there may be a smarter way):

packages-install-path: "{{ var('AIRBYTE_PACKAGES_DIR', 'dbt_packages') }}"

And then you can try this:

dbt deps --vars '{"AIRBYTE_PACKAGES_DIR": "../dbt_packages"}'

@Mickaël Andrieu thanks for coming back to me ! unfortunately

dbt deps --vars "{key:value}"

doesn't seem to work,

--vars

seem to be for

dbt run

only. I guess we will simply set the default value in dbt_project.yml to '`../dbt_packages`' and adapt our docker image accordingly 😎