I was playing around with lineage. Now I ended up ...
# troubleshoots
square-activity-64562
08/03/2021, 8:35 AMI was playing around with lineage. Now I ended up with this graph. I would like to remove the dataset to dataset lineage. I am not sure how to do that. UpstreamLineageClass does not seem to have a removed status. https://github.com/linkedin/datahub/blob/352a0abf8d7e4dd5d5664a8c7cdf3d77bf6f1c51/metadata-ingestion/src/datahub/metadata/schema_classes.py#L3274
square-activity-64562
08/03/2021, 10:06 AMI tried doing
Copy code
lineage_mce = MetadataChangeEventClass(
proposedSnapshot=DatasetSnapshotClass(
urn=_get_dataset_urn(downstream_urns),
aspects=[
UpstreamLineageClass(
upstreams=None,
),
],
)
)
emitter = DatahubRestEmitter("<http://datahub-datahub-gms.apps.svc.cluster.local:8080>")
emitter.emit_mce(lineage_mce)b
big-carpet-38439
08/04/2021, 3:55 AM@green-football-43791 What's the easiest way to remove dataset -> dataset lineage?
s
square-activity-64562
08/04/2021, 5:54 AMI am also stuck with this because I accidentally sent one event incorrectly. I tried sending remove event but it did not get removed
Copy code
MetadataChangeEventClass({'auditHeader': None, 'proposedSnapshot': DataJobSnapshotClass({'urn': 'urn:li:dataJob:(urn:li:dataFlow:(airflow,replace_cta,prod),cta_global_okr_pct_change_id_lineage)', 'aspects': [DataJobInfoClass({'customProperties': {}, 'externalUrl': '<https://DOMAIN_NAME/admin/taskinstance/?flt0_dag_id_equals=replace_cta&flt3_task_id_equals=cta_global_okr_pct_change_id_lineage>', 'name': 'cta_global_okr_pct_change_id_lineage', 'description': None, 'type': 'COMMAND', 'flowUrn': None, 'status': None}), DataJobInputOutputClass({'inputDatasets': ['urn:li:dataset:(urn:li:dataPlatform:athena,analytics.cta_global_okr,PROD)'], 'outputDatasets': ['urn:li:dataset:(urn:li:dataPlatform:athena,analytics.cta_global_okr_mom_pct_change,PROD)'], 'inputDatajobs': []}), StatusClass({'removed': True})]}), 'proposedDelta': None})square-activity-64562
08/04/2021, 5:58 AMWe really need a delete button on the lineage tab to mark something as removed. Sending events does not work
s
some-microphone-33485
08/04/2021, 7:03 AM+ 1 for this feature .
s
square-activity-64562
08/04/2021, 12:57 PMI checked
datahub_datajobindex_v2/_search?pretty=true&size=100"removed" : truedatahub_graph_service_v1b
big-carpet-38439
08/04/2021, 2:30 PM@green-football-43791 @early-lamp-41924 incorrect deletes behavior ^ and also good feature requests
g
green-football-43791
08/04/2021, 2:53 PMThanks for raising this Aseem- I created an issue to track the problem:
https://github.com/linkedin/datahub/issues/3028
green-football-43791
08/04/2021, 2:53 PMwe'll get to this ASAP
d
dazzling-appointment-34954
01/19/2022, 1:26 PMI also stumbled across this problem today, is there a way to remove a single lineage graph? I ingested lineage through an emitter and now I want to get rid of 1 connection. I actually deleted the target dataset but still it is shown in the lineage graph (I guess due to the aspect value in the dataset). Anyway I am not able to delete the aspect at the downstream dataset, could you please help?
dazzling-appointment-34954
01/19/2022, 1:35 PMUpdate: Made it work through a hard delete of the upstream lineage Dataset. Soft delete did not cut it for the UI but with the hard delete it now works 🙂 Anyway feels like a bug that lineage remains after soft deleting the target, what do you think?
b
big-carpet-38439
01/19/2022, 10:52 PMThis does feel like a bug... Soft deleted entities should be filtered out of the result set, but I think are not ... @green-football-43791 This may come up in the work you and dexter are doing right now
e
early-lamp-41924
01/19/2022, 11:23 PMI do remember we discussed this issue before. We should filter these nodes out or at least have a UI indication that the node was deleted
6 Views