Ok, having trouble since I’m very new to data engi...
# ask-community-for-troubleshootingj
Jordan Scott
03/17/2022, 2:25 AMOk, having trouble since I’m very new to data engineering… I tested fivetran and got dbt set up no problem, but that's because it was just enough plug-n-play for my level of development knowledge.
Could somebody reeaaally break it down for me as if i were 5 yrs old?
HOW can I make my own dbt models to use with airbyte open-sourced? Where can I access the pre-built dbt project to manipulate and connect through github?
I have it running on a EC2 instance with an ssh tunnel from my Mac and testing with an Instagram>BigQuery connection up and running.
Please don't share the airbyte-dbt help doc, as it doesn't seem to dumb it down enough for me 😂_… or the demo hour youtube video_ 🙏
👀 1
o
Octavia Squidington III
03/17/2022, 11:12 AMloading...
n
Noel Gomez
03/17/2022, 2:25 PMyou dont have to use dbt as part of the airbyte load process. You can just load with Airbyte and then separately run dbt on the loaded data. I do this and just work off the RAW data from Airbyte. that might be simpler 🙂
j
Jordan Scott
03/17/2022, 2:31 PMI appreciate the advice @Noel Gomez! I was thinking that was my next/first step also. I’m still looking for better understanding of running dbt through airbyte as I want to scale up to that for bigger projects coming my way this year, but I’m not sure if I can run dbt Cloud or if I need to get better with CLI in general, ha.
n
Noel Gomez
03/17/2022, 2:33 PMI tend to prefer not running dbt as part of ingestion because transformations are not going to be one to one. I mean you will probably load 2+ sources, then transform across them.
a
Augustin Lafanechere (Airbyte)
03/17/2022, 2:44 PMHi @Jordan Scott, let us know where you are blocked in our documentation; it'll be valuable feedback to improve it. I suggest that you first locally build your DBT project to make the transformation you want on your raw data. Then, once it's ready, you can add a custom transformation in Airbyte after pushing your dbt code to a Github repo. If you plan to make transformations on different sources, as @Noel Gomezsuggests, you're better off running transformations using an orchestrator such as Airflow.
✅ 1
Augustin Lafanechere (Airbyte)
03/17/2022, 2:46 PMTo retrieve the pre-built normalization model you need to execute these commands from your docker machine.
Augustin Lafanechere (Airbyte)
03/17/2022, 2:49 PMBreaking it down:
1. First run a sync with normalization on.
2. After a first successful sync, export the dbt normalization project from the airbyte server using these instructions.
3. Make the changes you need locally on the exported dbt project
4. Push your dbt project to github repo that your Airbyte instance is able to clone
5. Set a custom transformation on your sync, as described here
Let me know which step is difficult for you.
🙏 1
j
Jordan Scott
03/17/2022, 3:05 PMThat all makes sense in theory 😅
I believe I get stuck with writing CLI in general. So I just need to take a step back to understand what each command means so I can make sure I know WHAT I’m doing instead of blindly typing in commands and hoping for the best.
Jordan Scott
03/18/2022, 4:47 PMOk, so I was able to get the SQL files copy/pasted over to my own personal dbt project - awesome.
I’ve changed things in the models and ran dbt in dbt cloud and it works great - perfect.
But when I try to connect my dbt project in airbyte, I keep getting an error - bummer.
Could someone tell me what may be wrong by looking at this screenshot of the logged error(s)? Or what else would I need to screenshot to identify where the error might be?
Jordan Scott
03/18/2022, 4:50 PMI’m running two transformations —
depsrunrunJordan Scott
03/18/2022, 4:52 PM…and my dbt_project.yml file:
a
Augustin Lafanechere (Airbyte)
03/22/2022, 6:14 PMHey @Jordan Scott, could you share which version of DBT you are using in DBT cloud? Our image is using 0.19.1, maybe this is why some properties are not supported, triggering your error. Maybe @Chris Duong [Airbyte] has a better insight.
j
Jordan Scott
03/22/2022, 6:27 PMScreen Shot 2022-03-22 at 1.27.15 PM.png
a
Augustin Lafanechere (Airbyte)
03/22/2022, 6:33 PMI rather asking about the dbt project version not the dbt cloud version. I think you can get this from your environment setting
j
Jordan Scott
03/22/2022, 6:46 PMah that’s right - sorry. I’m using v1.0
Jordan Scott
03/22/2022, 6:46 PMShould I bring it down to 0.19.1?
a
Augustin Lafanechere (Airbyte)
03/22/2022, 6:47 PMYes please
j
Jordan Scott
03/22/2022, 6:47 PMok on it
Jordan Scott
03/22/2022, 7:07 PMIt worked! Ok so apparently those alternative path options in the dbt_project.yml file weren’t supported yet in 0.19.1.
When I downgraded in dbt cloud, it gave me the same warning that Airbyte was throwing at me.
Just ran Airbyte with the dbt transformations and it worked perfectly! WOOHOO thank you for the help!
c
Chris Duong [Airbyte]
03/22/2022, 7:21 PMyou can also upgrade/use a docker image that is using dbt v1.0 with airbyte too.
But if you use a version shipped with v.0.19, then yes, your dbt_project.yml should follow that version properties
👍 1
Chris Duong [Airbyte]
03/22/2022, 7:24 PM2 Views