Hi, I am ingesting hive table lists, columns from ...
# ingestion
s
Hi, I am ingesting hive table lists, columns from Hive Metastore RDS (MySQL). However, I would like to set owners and tags for the ingested hive tables. Can I set owner, tags for dataset entities after ingestion?
b
s
Thanks @better-orange-49102. I was able to find a way to set OwnershipTypeClass, DatasetSnapshotClass However, Is it possible to customize description of dataset in the same way?
b
inside a transformer you have access to the entire ingested dataset, prior to it being processed by the sink. so you could customize the description, although i think you may have to write your own transformer
s
Thanks for the answer. Some information are on the google sheet, I think it's better to create custom transformer which fetching the additional informations (owner, tags, description, ...) Here is the google sheet sample I would like to use.
Basically, it's lineage information (source -> target), but want to set owner, tags, description while emitting lineage MCE.
b
im not certain about lineage because i've never used that aspect for my own implementation; but inside the transformer you have access to the MCE of the dataset, so you can modify it
s
Okay. will try it. Thanks 🙂
I might use these functions which are recently added. https://github.com/linkedin/datahub/blame/master/metadata-ingestion/src/datahub/emitter/mce_builder.py • make_global_tag_aspect_with_tag_list • make_ownership_aspect_from_urn_list • get_or_add_aspect Seems it's possible to add apsects to mce
b
yup, mce.proposedSnapshot.aspects takes a list, shd be able to append/remove aspects from it
❤️ 1
thank you 1