Hi everyone, i am newbie I have to question: 1. Ho...
# ingestion
g
Hi everyone, i am newbie I have to question: 1. How i can change the description of the specific trable. Can you instruct me how to edit description in detail, please? Where is document about push-API ? I don't want to change description of all tables in database 2. When i delete table at my own database in local and i ingest this changed database again with the hope that data hub will update too. But it does not. 3. How metadata are stored beneath Data hub ? What is database used for saving, if there's any ?
g
Hey there @glamorous-kite-95510- 1. right now you can edit column descriptions, but not table descriptions through the UI. There is an issue to track table description edits here: https://github.com/linkedin/datahub/issues/2114 and we are working to prioritize it asap
👍 1
2. In this case, are you saying you see stale data that has been deleted? To remove an entity, you will need to mark it as deprecated. See https://datahubproject.io/docs/what/entity/#how-to-delete-an-entity for more details
3. metadata is stored primarily in mysql. We store search indexes in elasticsearch and graph relationships in neo4j
g
1. I am interested in ingesting metadata into Data Hub to change description of one specific table . However, I'm confronting 2 problems. Firstly, I intend to use Transformations . But in all URNs available in Data Hub, I can't see description of table field at all. If there were, which was the URN and i wonder if I ingested metadata with that description of table into Data Hub, whether all datasets in database had the same description or not ? Is there any way i can ingest metadata with the description of a specific table without affecting one's others. Can you give me an example of Transformation including description of one table. Secondly, if Transformation doesn't work, is there anything else? Can you give me an example that fits me, please? Supposed I have a database in Postgresql named "commercial" that includes two tables: accounts and bills. How i can ingest this database with the description, 'this is accounts table', that should belong to accounts table without storing the this description at level of database. And then, how about adding the description for bills table. Is this possible ?
g
Hey @glamorous-kite-95510 - one option is to create your own transformer that inspects the MetadataChangeEvent's URN and only applies a description if the urn is a match
using the add_dataset_ownership transformer as an example,
at line 51 they check if the entity is of type dataset
you could do a similar check but instead look at the urn of the snapshot
g
1. I don't see dataset description urn field in all URNs available on DataHub. May i have to create something like DescriptionUrn.class and all the things involved 2. If i do the thing like add_dataset_description transformer as you suggested. Do all the datasets in database have the same description , right ? The main thing is i just want to ingest the description of only one specific table. Can I use the DatasetUrn.class to get the name of dataset in order to clarify the expecting dataset or it is something else?
b
"I don't see description urn field in all URNs available on DataHub" are you referring to demo json that shows dataset descriptions?
g
Can you give me that link about demo json that shows dataset descriptions
g
No, I am referring to description urn field in URNs class just like tag field belongs to TagUrn class