hey, whats the max cardinality of a custom Entity ...
# advice-metadata-modeling
c
hey, whats the max cardinality of a custom Entity that DataHub should be able to comfortably handle?
m
The cardinality shouldn't be a function of whether it is a custom entity or not
c
hmm, how do you mean?
m
Meaning that if you create a new entity type : let’s say “Data Product” and then you want to store 1M such entities in DataHub, it would have the same cost as storing 1M entities of type “Dataset” in DataHub
The main factor would be : how big is one entity record, how many fields are indexed etc.
c
Right.. Entities themselves would be rather small, we'd want to use them to represent physical resources which dataset are tied to. the important thing would be the relationships. so the Entities might have a single string identifier field, and then each on will be associated with some subset of our datasets. I guess the question I'm getting after is should the number of entities that datahub can handle be able to scale up linearly with resources, or are there known bottlenecks at application level that impose some order-of-magnitude limit on the cardinality of entity sets?
m
We scale pretty linearly with resources. We're not aware of any entity cardinality related bottlenecks.