https://datahubproject.io logo
Join Slack
Powered by
Hello everyone，I just ingest mysql metadata into d...
# getting-started
b
Hello everyone，I just ingest mysql metadata into datahub, but the lineage don't show up, what else do I need to do that I can check the lineage?
g
I think you need to setup it manually.
b
how to set up, is there any Docs?
g
here is the docs, https://datahubproject.io/docs/lineage/sample_code/ you need to run a Python script.
b
thanks! let me check
Hi, I found only 
lineage_dataset_job_dataset.py
has something to do with mysql, so I download this script and use 
python3 lineage_dataset_job_dataset.py
,but there is no output, then I use 
python lineage_dataset_job_dataset.py
there is an error like 
SyntaxError: invalid syntax
g
@billions-football-16062 you need to change to your sink server here https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/lineage_dataset_job_dataset.py#L46 (line 46) and configure (downstream, source, upstream) here based on your dataset
b
is this a self-definition lineage?
Can datahub detect lineage automaticly like Atlas
g
Yes can. But for some database, it is not supported to detect lineage then need to self add.
b
I define 
lineage_emitter_rest.py
like below, then the lineage truly show up, however, this is not what I want. I think datahub should automaticlly detect the lineage. For example, I use Hive database, when I create table3 based on table1 and table2, the lineage should show that table3 comes from table1 and table2 and so on.
g
ah, it based on what platform/db we're using. on my side, i am using 
postgres
and yes i need to manually do it.
b
🤣that souds like a huge job
what if the table's relationship changed, you need to rerun the scripts?
g
yeah. maybe need to setup Airflow to schedule
or any cron to rerun if needed
b
you write the scripts first and use ariflow to execute it right?
g
when I ingest Athena metadata, the lineage is auto detected
yes, i am not automate the script yet
b
got it
have you ever try Hive before?
g
nope.
b
ok, here is a little question in my mind
I never use Athena before, but i assume it is a RDBMS, how does datahub analyse his lineage?
g
ah very technical question, need someone from DataHub to let us know!
b
haha
I mean, like Atlas ,you need to initializing the metastore first, then if you make a change ,the Atlas can analyse your sql and show the lineage
but I am not sure how does dataHub analyse the lineage
Just now you said DataHub can analyse lineage when you using Athena, is it show the lineage when you first instropect the metadata?
g
Yes, right.
l
@billions-football-16062 datahub extracts lineage automatically for cloud warehouses like snowflake, bigquery, redshift based on native capabilities these systems provide
For other databases, sql parsing is necessary and it is on our roadmap but not for q3
g
happy to hear this @loud-island-88694
b
@loud-island-88694 copy that with many thanks. And I have a question there, what should I do if I want to analyse hive's lineage?
2 Views
Open in Slack
PreviousNext
Powered by