If I want to store new types of entities in Datahu...
# ingestion
o
If I want to store new types of entities in Datahub, should I be extending the metadata model and creating new Entity types? Let's say I want to store schema information for the inventory in my shop (books, gadgets etc). The full list of existing Snapshot types is quite small: https://github.com/linkedin/datahub/blob/master/metadata-models/src/main/pegasus/com/linkedin/metadata/snapshot/Snapshot.pdl I'm also using https://datahubproject.io/docs/metadata-modeling/extending-the-metadata-model as reference If so, are there any examples where people have created their own Entity types and defined corresponding Aspects, Ownership etc?
g
Hey @orange-flag-48535 - you are correct in your general approach. The snapshot type is limited because the Datahub project is trying to be very intentional with its modeling decisions. When an entity type is added, the project needs to be committed to supporting it. Can you share more about your use case? Could your inventory schema information be modeled as a dataset?
o
Is Dataset some kind of generic type? In general, are you saying that Datahub is focussed on cataloging certain kinds of entities only, and not any schema a company would like to store and evolve? Regarding my example, let's say I have books and phones. For books I'll have one set of fields (which will evolve) and similarly for phones. Would that fit into Dataset? And would I lose type safety by going for Dataset?
g
have you played around with the datahub demo site? That may answer some of your questions
I would open up some dataset pages & take a look at how they are structured
o
Thanks. Will do. I got Datahub running, but didn't look into Dataset part.