https://datahubproject.io logo
Channels
Powered by
#ingestion
Title
# ingestion
g

green-lion-58215

09/23/2022, 10:47 PM
Quick qn on DBt ingestion. I am ingesting dbt test results with the run_results.json. I can also see the test cases in the datahub UI as well. But it does not show any pass/failures for the test cases. It is all coming up as “no evaluations found”. I am using datahub 0.8.41 for context. dbt version 0.21.1 any help is appreciated.
@mammoth-bear-12532
m

mammoth-bear-12532

09/26/2022, 4:25 PM
Hi @green-lion-58215 can you route the results to a file sink?
then you can inspect whether 
assertionRunEvent
aspects are being generated
g

green-lion-58215

09/26/2022, 4:56 PM
I ran the ingestion with debug mode enabled. But I do not see this assertionRunEvent
is there any setting or config I need to enable?
m

mammoth-bear-12532

09/26/2022, 5:15 PM
what does your run_results.json look like?
I'm assuming you are using this config:
Copy code
test_results_path: "${DBT_PROJECT_ROOT}/target/run_results.json"
g

green-lion-58215

09/26/2022, 5:29 PM
yes I am
m

mammoth-bear-12532

09/26/2022, 5:44 PM
how are you validating that no assertionRunEvents are being emitted?
did you configure a file sink and inspect the json?
g

green-lion-58215

09/26/2022, 6:06 PM
for DBT I am not able to proivide a file sink. it throws an error while trying to do so. I ran the dbt ingestion using --debug flag and I checked the output generated to see if there is any assertion run events
m

mammoth-bear-12532

09/26/2022, 6:20 PM
ah I think I know the failure you ran into
add this to your recipe
Copy code
datahub_api:
  config:
    server: "<http://localhost:8080>"
sink:
  type: file
  config:
    filename: events.json
g

green-lion-58215

09/26/2022, 7:37 PM
Ok. I am able to write to file now. But still I am not able to see the assertionRunEvents either in file nor in the log output
m

mammoth-bear-12532

09/26/2022, 8:59 PM
how about the run_results.json ... does it have test results for sure?
g

green-lion-58215

09/26/2022, 8:59 PM
yes it does
@mammoth-bear-12532 is there any way to troubleshoot this?
m

mammoth-bear-12532

09/29/2022, 6:22 AM
I’m surprised. Let’s debug this after townhall tomorrow.
g

green-lion-58215

09/29/2022, 3:18 PM
sure
if a call is useful let me know I can hop on a call
@mammoth-bear-12532 when would you have time for a short call?
m

mammoth-bear-12532

09/29/2022, 5:39 PM
Hey @green-lion-58215 at 11am PST would work.
g

green-lion-58215

09/29/2022, 5:40 PM
any slots for after 12 PST? I have a meeting at 11
m

mammoth-bear-12532

09/29/2022, 5:45 PM
how about 3:30pm?
g

green-lion-58215

09/29/2022, 5:45 PM
sure. sounds good
a

able-library-93578

07/19/2023, 8:07 PM
@mammoth-bear-12532, @green-lion-58215 was there a resolution to this. I am having the same issue. I will try about recommendation on adding to ingestion yaml, but wanted to ask. Thanks.
versions: Datahub: 0.10.4, dbt core 1.5.0
g

green-lion-58215

07/20/2023, 12:45 AM
for me, the reason if I recall was that the test results file generated was not the proper one. dbt will create test results file for multiple commands. try running the dbt test command and take the test results file immediately. that should resolve it
m

mammoth-bear-12532

07/20/2023, 1:18 AM
wow, good memory @green-lion-58215! I had forgotten 🙂
cc @delightful-ram-75848: flagging this to add to our dbt docs FAQ
a

able-library-93578

07/20/2023, 3:46 PM
@green-lion-58215 Thanks, I will try that. I believe the run_results.json I have is a proper one. I will give it a try though. Thanks for your reply. Much appreciated.
Powered by