and another doubt regarding uploading downstreams ...
# troubleshoota
acceptable-terabyte-34789
11/15/2022, 9:49 AMand another doubt regarding uploading downstreams and upstreams...
we have developed a custom spark listener using the spline library and we can emit downstreams at column-field level but downstreams are not present in datahub...
there isn't any error. Where can I see logs to help me in the tool?
d
dazzling-judge-80093
11/15/2022, 9:52 AMCan you see these edges on the Lineage UI or you can’t see there as well?
a
acceptable-terabyte-34789
11/15/2022, 9:53 AMI can't see them, just the upstreams ones
d
dazzling-judge-80093
11/15/2022, 9:53 AMAre you sure you are sending it in?
Can you generate some sample mcps and send them to me?
a
acceptable-terabyte-34789
11/15/2022, 9:56 AMOK, wait a moment please. Thanks for your help
acceptable-terabyte-34789
11/15/2022, 10:39 AM Copy code
MetadataChangeProposalWrapper(entityType=dataset, entityUrn=urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD), changeType=UPSERT, aspect={fineGrainedLineages=[{downstreamType=FIELD, transformOperation=substring(productgroup,5,length(productgroup)), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),productgroup_transf)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(productgroup))]}, {downstreamType=FIELD, transformOperation=year(cast(startdate)), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),startyear)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD),Set(startdate))]}, {downstreamType=FIELD, transformOperation=recid, downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),key)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(recid))]}, {downstreamType=FIELD, transformOperation=concat(productline,arrstatus), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),concatenation)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD),Set(arrstatus)), urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(productline))]}], upstreams=[{type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD)}, {type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD)}]}, aspectName=upstreamLineage)acceptable-terabyte-34789
11/15/2022, 10:39 AM@dazzling-judge-80093
acceptable-terabyte-34789
11/15/2022, 11:23 AMFineGrainedLineage are not inserted but I can see them at graphql..
acceptable-terabyte-34789
11/15/2022, 11:26 AMmessage has been deleted
d
dazzling-judge-80093
11/15/2022, 12:31 PMFinegrainedLineage should be sent in with -> https://datahubproject.io/docs/generated/metamodel/entities/dataset#upstreamlineage
Similar as you can see here in python -> https://github.com/datahub-project/datahub/blob/d87dcfae758d73aaef2e27dd68a812f611[…]metadata-ingestion/src/datahub/ingestion/source/unity/source.py
a
acceptable-terabyte-34789
11/16/2022, 5:22 PM@astonishing-answer-96712
a
astonishing-answer-96712
11/16/2022, 5:27 PMThank you @acceptable-terabyte-34789, could you check to make sure you access control policies are set correctly to view the lineage in the UI, as per this blog https://blog.datahubproject.io/recent-improvements-to-user-group-management-and-fine-grained-access-control-in-datahub-8d78a4ad2388
a
acceptable-terabyte-34789
11/16/2022, 10:15 PMwe logged as admin user..and nothing changed
acceptable-terabyte-34789
11/16/2022, 10:15 PMI post the mcpw clearer:
acceptable-terabyte-34789
11/16/2022, 10:15 PM Copy code
MetadataChangeProposalWrapper(entityType=dataset, entityUrn=urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD), changeType=UPSERT,
=======================================================================================
aspect={fineGrainedLineages=[{downstreamType=FIELD, transformOperation=substring(productgroup,5,length(productgroup)), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),productgroup_transf)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(productgroup))]}, {downstreamType=FIELD, transformOperation=year(cast(startdate)), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),startyear)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD),Set(startdate))]}, {downstreamType=FIELD, transformOperation=recid, downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),key)], upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(recid))]}, {downstreamType=FIELD, transformOperation=concat(productline,arrstatus), downstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,datahub_test.lineage,PROD),concatenation)],
=========================================================================================
upstreamType=FIELD_SET, upstreams=[urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD),Set(arrstatus)), urn:li:schemaField:(urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD),Set(productline))]}],upstreams=[{type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement_sim,PROD)}, {type=TRANSFORMED, dataset=urn:li:dataset:(urn:li:dataPlatform:athena,t24_snapshots.aa_arrangement,PROD)}]}, aspectName=upstreamLineage)a
astonishing-answer-96712
11/16/2022, 10:23 PM@big-carpet-38439 any idea why this UI visibility issue might be happening with fine grained lineage?
2 Views