https://datahubproject.io logo
#ingestion
Title
# ingestion
b

busy-furniture-10879

06/03/2022, 4:42 PM
I'm running into a situation where DBT isn't picking up the full lineage for situations like:
Copy code
{Snowflake Table A} -> {DBT Job B} -> {Snowflake Table C}
It picks up B->C but not A->B I saw in the docs that this was resolved in 0.8.16.2, and I'm on 0.8.34.1. Any ideas as to what I'm doing wrong?
l

loud-island-88694

06/03/2022, 6:30 PM
@green-football-43791 can you please take a look here?
@busy-furniture-10879 can you please share logs from your dbt ingestion?
is A is a dbt source?
b

busy-furniture-10879

06/03/2022, 6:32 PM
A is not sourced from DBT, its source is a custom program that dumps from a Rabbit Queue
g

green-football-43791

06/03/2022, 6:33 PM
Did you create a dbt
source
to reference the table?
b

busy-furniture-10879

06/03/2022, 6:34 PM
We did not create a source to reference the table, is that a requirement?
g

green-football-43791

06/03/2022, 6:35 PM
this allows us to track the lineage from B to A
b

busy-furniture-10879

06/03/2022, 6:35 PM
We'll get to updating our models to use sources then.
g

green-football-43791

06/03/2022, 6:35 PM
are you just referencing the table directly from the B query?
b

busy-furniture-10879

06/03/2022, 6:35 PM
Yes
g

green-football-43791

06/03/2022, 6:35 PM
I see. I believe we do not support that.
from the sources docs, they also specify a reason to use sources is
Copy code
select from source tables in your models using the {{ source() }} function, helping define the lineage of your data
i believe this is necessary for dbt to understand the lineage.
b

busy-furniture-10879

06/03/2022, 6:40 PM
Makes sense. I'll start updating our models
4 Views