Question on Upstream lineage (UpstreamLineage):
I...
# ingestionw
witty-keyboard-20400
10/05/2021, 3:32 PMQuestion on Upstream lineage (UpstreamLineage):
In the bootstrap_mce.json, I see that UpstreamLineage is defined at DatasetSnapshot level.
Copy code
{
"com.linkedin.pegasus2avro.dataset.UpstreamLineage": {
"upstreams": [
{
"auditStamp": {
"time": 1581407189000,
"actor": "urn:li:corpuser:jdoe",
"impersonator": null
},
"dataset": "urn:li:dataset:(urn:li:dataPlatform:kafka,SampleKafkaDataset,PROD)",
"type": "TRANSFORMED"
}
]
}
}g
green-football-43791
10/05/2021, 3:34 PMfield level lineage is on our roadmap: https://datahubproject.io/docs/roadmap
green-football-43791
10/05/2021, 3:34 PMlooks like we need to update those timelines a bit, but this is something on our near term radar that we aim to start tackling in the next few weeks.
😐 1
✅ 1
green-football-43791
10/05/2021, 3:34 PMdo you have specific column level lineage use cases in mind?
green-football-43791
10/05/2021, 3:35 PMare there specific sources or goals you have in mind for column level lineage?
w
witty-keyboard-20400
10/05/2021, 3:35 PMYes, definitely. I'm working on EDD and Metadata lineage at our org (embibe.com)
witty-keyboard-20400
10/05/2021, 3:36 PMI started with Neo4J, but felt that the metadata Type system is already thought after in DataHub, so proceeded with the metadata modeling.
g
green-football-43791
10/05/2021, 3:37 PMgotcha- we'd love your input during the design process.
w
witty-keyboard-20400
10/05/2021, 3:37 PMField level lineage is very important to us and I think for most of the org.
1. E.g. as a product owner I want to see from the dashboard which all fields (standard and derived) are contributing to my dashboard and from where are they originated.
g
green-football-43791
10/05/2021, 3:37 PMwe have a channel #column-level-lineage for discussing this feature specifically
green-football-43791
10/05/2021, 3:37 PMso feel free to join and give your input there 🙂 it would be great to hear your thoughts.
w
witty-keyboard-20400
10/05/2021, 3:37 PMalright, I'd follow
witty-keyboard-20400
10/05/2021, 4:12 PMLooks like I've to fall back to Neo4J and use my own custom relationships between nodes to build a metadata model for my org.
h
helpful-optician-78938
10/06/2021, 2:22 AMHi @witty-keyboard-20400, the API for column-level lineage will be coming soon, followed by the v1 UI a little later. We recommend using the APIs rather than the custom relation approach if possible.
👀 1
3 Views