Hi everyone!
Until yesterday, we had our datahub ...
# troubleshooth
hallowed-dog-79615
07/05/2022, 11:28 AMHi everyone!
Until yesterday, we had our datahub crossplatform lineages properly connected. We set some "outlets" in our Airflow tasks, so then Datahub linked the task urns with downstream dataset urns. After reingesting some snowflake and dbt metadata, this connection has disappeared. I mean:
If you go to the Runs details of an airflow task, the output is properly stated (snowflake table as shown in the image). If you then click on that snowflake table, and go to the lineage, there is not a single trace of the airflow task or DAG. And it was there a couple of days ago.
At the same time, some snowflake and dbt urns referring to the same models/tables have been combined in one sibling association, which is nice, but... could this be related with the absence of link between airflow and dbt/snowflake in the lineages? Should we configure our outlets in a different way to be properly recognized?
Thanks in advance!
Dani
hallowed-dog-79615
07/06/2022, 7:27 AMIt seems that if I point to dbt urns in the Airflow outlets. instead of pointing to snowflake urns, the connections are there.
I understand that this is a bug, as the pointed objects are dual: dbt + snowflake, and pointing any of the two should result in them being connected in the lineage. Is that so?
Thanks!
b
big-carpet-38439
07/06/2022, 6:26 PMThat is correct- this is indeed a bug. We are actively looking into it. Thank you for reporting!
big-carpet-38439
07/06/2022, 6:26 PMcc @green-football-43791
h
hallowed-dog-79615
07/07/2022, 12:55 PMThanks for reading and confirming!
g
green-football-43791
07/07/2022, 3:34 PMlooking now!
green-football-43791
07/07/2022, 3:57 PMHey Daniel, I wasn’t able to easily reproduce this on my own
green-football-43791
07/07/2022, 3:57 PMI added an airflow task to some sibling connected lineage and it didn’t have any trouble rendering
green-football-43791
07/07/2022, 3:59 PMThere’s a few things I’d like to check on your side to try and isolate what the issue might be. Would you have time to jump on a short call some time today?
green-football-43791
07/07/2022, 4:06 PMAs some pre-work, it would be great if you could check the graphql response from the
getEntitygreen-football-43791
07/07/2022, 4:06 PMand share that here (or in a dm to me)
green-football-43791
07/07/2022, 4:12 PMI’d also be curious if the task’s entity page has any lineage on it
h
hallowed-dog-79615
07/08/2022, 7:37 AMDMing you!
3 Views