Hey I would like to add a custom dbt transformation for my r Airbyte #advice-data-transformation

Hey, I would like to add a custom dbt transformati...

Olivier AGUDO PEREZ

09/30/2022, 1:10 PM

Hey, I would like to add a custom dbt transformation for my replication to Bigquery. I also need to specify custom destination dataset in Bigquery (default one writes to

airbyte_raw

). To do that I had to put my file

profiles.yml

into my dbt project github repo in order to specify my dataset. Following Airbyte doc, I have now to specify the

profiles.yml

path into Airbyte GUI

Entrypoint arguments for dbt cli

: that field is as follow

run --profiles-dir .

And this is where problems begin. When I run my Airbyte sync, I have the following error during dbt phase :

Could not find profile named 'normalize'

Problem is my profile name is not "normalize", neither in

profiles.yml

or in

dbt_project.yml

. What is the way to specify that

profiles.yml

path and why is it trying to use normalize project ? Also when I try to run the sync + transformation without specifying custom dataset, it works great.

Chris Duong [Airbyte]

09/30/2022, 1:46 PM

This is what is forcing the custom dbt transformation to use the ‘normalize’ profile: https://github.com/airbytehq/airbyte/blob/d485d04b4dd81b582d3caf98473e0282b35475a6[…]yte-workers/src/main/resources/dbt_transformation_entrypoint.sh

Chris Duong [Airbyte]

09/30/2022, 1:47 PM

This is because when you don’t check-in your

profiles.yml

in your git repo, airbyte is using normalization docker image and scripts to generate a

profiles.yml

from the credentials in your destination connector. And that

profiles.yml

is always going to be named “normalize”

Chris Duong [Airbyte]

09/30/2022, 1:49 PM

if you rename your

profiles.yml

file in your git repo to be “normalize”, then it should work?

Olivier AGUDO PEREZ

09/30/2022, 2:40 PM

Hey, even if I rename the profile "normalize" the error is still here

Olivier AGUDO PEREZ

09/30/2022, 2:41 PM

I tried with

run --profiles-dir .

telling him the profiles file is in the current dir (which, looking at the code, should be the cloned repo directory) but still nothing (Note that that command works on my local DBT)

Chris Duong [Airbyte]

09/30/2022, 2:42 PM

What if you configured

--profile <my real profile>

? in your dbt cli options? (yes i’m just suggeting hacks here)

Olivier AGUDO PEREZ

09/30/2022, 2:51 PM

I also tried to rename my profile to "normalize" (which is basically what you suggested), but again it seems that airbyte is not able to use the provided

profiles.yml

😕

Olivier AGUDO PEREZ

09/30/2022, 2:54 PM

Also, "hard coding" the path won't work, since the git repo path will change through the runs (i.e.

/data/127/1/transform/git_repo

and then

/data/127/2/transform/git_repo

)

16 Views

Open in Slack