Has anyone come across this issue with MySQL destination and using custom transformation? using the normal transformation fails because it converts a json to string instead of text so I tried to set up the custom dbt but it fails to install the deps for some reason. I have followed the guides with setting the

packages-install-path: ../dbt

but it still fails. Is there a config somewhere where I need to define the profile My transformations doesn't contain any special arguments just deps followed by another transformation to run Any help would be much appreciated

Hi Benjamin! I haven't seen this come up before but I'll look into it and get back to you!

Thank you @Nataly Merezhuk (Airbyte) Looking forward to hear what you find out

Could you please include the full logs?

@Nataly Merezhuk (Airbyte) here's the log. I got the profile part to work but the dbt is still failing because of the adapter for mysql is still missing

Could you give me a link to the docs you've been following? Also, could you run this to verify that dbt is properly configured?

# find automatically latest workspace where normalization was run
NORMALIZE_WORKSPACE=`docker run --rm -i -v airbyte_workspace:/data  busybox find /data -path "*normalize/models*" | sed -E "s;/data/([0-9]+/[0-9]+/)normalize/.*;\1;g" | sort | uniq | tail -n 1`

#!/usr/bin/env bash
docker run --rm -i -v airbyte_workspace:/data -w /data/$NORMALIZE_WORKSPACE/normalize --network host --entrypoint /usr/local/bin/dbt airbyte/normalization debug --profiles-dir=. --project-dir=.

More here: https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-dbt#validate-dbt-project-settings

Hi @Nataly Merezhuk (Airbyte) If I run it without selecting normalization but just raw data and then add the custom transformation as explained from here https://docs.airbyte.com/operator-guides/transformation-and-normalization/transformations-with-airbyte I get this

02:55:30  Running with dbt=1.0.0
dbt version: 1.0.0
python version: 3.9.9
python path: /usr/local/bin/python
os info: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Using profiles.yml file at /data/normalize/profiles.yml
Using dbt_project.yml file at /data/normalize/dbt_project.yml

Configuration:
  profiles.yml file [ERROR not found]
  dbt_project.yml file [ERROR not found]

Required dependencies:
 - git [OK found]

1 check failed:
dbt looked for a profiles.yml file in /data/normalize/profiles.yml, but did
not find one. For more information on configuring your profile, consult the
documentation:

<https://docs.getdbt.com/docs/configure-your-profile>

If I run it with Airbyte normalization first instead of selecting Raw Data I get this error

02:56:55  Running with dbt=1.0.0
dbt version: 1.0.0
python version: 3.9.9
python path: /usr/local/bin/python
os info: Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.31
Using profiles.yml file at /data/2/0/normalize/profiles.yml
Using dbt_project.yml file at /data/2/0/normalize/dbt_project.yml

02:56:55  Error importing adapter: No module named 'dbt.adapters.mysql'
Configuration:
  profiles.yml file [ERROR invalid]
  dbt_project.yml file [OK found and valid]

Required dependencies:
 - git [OK found]

1 check failed:
Profile loading failed for the following reason:
Runtime Error
  Credentials in profile "normalize", target "prod" invalid: Runtime Error
    Could not find adapter type mysql!

Looking at the specific workspace where it was run, I do however see both the profile and the dbt_project file as you can see below

/data/1/0/transform # ls
entrypoint.sh    git_repo         profiles.yml     sshtunneling.sh
/data/1/0/transform # cd git_repo
/data/1/0/transform/git_repo # ls
README.md        dbt_project.yml  models           seeds            tests
analyses         macros           packages.yml     snapshots
/data/1/0/transform/git_repo #

Could you check that the profile name is set to 'normalize' in

profiles.yml

? Found this issue: https://github.com/airbytehq/airbyte/issues/12465

@Nataly Merezhuk (Airbyte) the profile name is set to normalize. It was first generated from airbyte's normalization, exported, slightly modified in the models and uploaded to a private github repo

I see! Let me look into this further, I'm not quite sure what else might be going wrong at this point unfortunately!

@Nataly Merezhuk (Airbyte) Have you been able to find any solution to this possibly?

Hi, Benjamin! My apologies for the wait! Are you using multi-step custom transformations by any chance and are you running on Docker? I've found an issue that might have a workaround for you: https://github.com/airbytehq/airbyte/issues/5590

Hi @Nataly Merezhuk (Airbyte) Yes we are using Docker, do I understand the workaround correctly that I should compile it locally, push it to the github repo and then only do dbt run? How will this solve the issue with not being able to find the MySQL module?

So in my case I was able to make it work by using the

airbyte/normalization-mysql:0.4.3

in the transformation