https://datahubproject.io logo
Join Slack
Powered by
Calling all folks interested in no-code metadata m...
# announcements
b
Calling all folks interested in no-code metadata modeling! Going through the process of spec-ing it out. Intending to get clarity on • Authoring: How should no-code model extensions be defined? What data structures should be supported? • Visualization: How should no-code model extensions be displayed on the UI? • Ingestion: What type of writes should no-code model extensions support? UI-based? API-based? Both? I intend to start to answer these questions based on feedback from the community. If it were available, how would YOUR organization leverage the ability to extend DataHub models without code?
👀 4
cc. @incalculable-ocean-74010 cause I know you've got thoughts on this one 😛
i
😬 let’s talk
m
are you planning on writing like a gradle plugin to auto generate the code from the pegasus models, maybe with a config file where you can customise certain aspects of the model (like eg: this entity can be edited from the UI, this one cant be)
w
Which graph formalism are you aiming* to support ? If OWL is involved, I got a prototype for metadata / lineage which could help you
i
b
So I don't think we are in the market to fundamentally change how we model entities + relationships
In terms of approaches we could either do • codegen to keep everything strongly typed through the entirety of the stack (but the assumption here is that you'd still need to define a PDL model which means forking which we want to avoid) • have some sort of "extensions" aspect attached to every entity that allows one to dynamically specify a schema and provide values using the API alone (That is, at runtime)
i'm leaning toward #2
i
Option #2 losses strong types as feature doesn't it?
What do you mean provide values using the API alone?
m
could you have a hybrid of both #1 and #2 whereby the the pdl models are stored in an entirely different repository and the models generate their code as a package which is published to something like artifactory or nexus and the main datahub repo is simply importing the generated code? im not that familiar yet with the codebase but this is a common pattern when dealing with protobuf and avro codegen
✅ 1
basic idea is to separate the model with the generic base code somehow
👌 1
m
MxE v5 will allow for model federation, which means no splitting and no "extensions"; at least for the backend. Frontend obviously needs to depend on everything transitively
I'm happy to talk more about that, zoom might be best
fwiw from a backend perspective even without UI a way to quickly build a service to CRUD the metadata would be nice 🙂
i
What is MxE @microscopic-receptionist-23548?
b
@miniature-ram-76637: Currently the model is tightly coupled to a bunch of code in GMS layer (The API endpoints) and datahub-frontend layer (the GQL API used by the client). It'd be interesting to think about whether we could extract the domain-specific portions and keep them inside artifactory in isolation but itd be a big lift architecturally from where we are @incalculable-ocean-74010: Option 2 doesn't have compile time type safety thats right. But it would have runtime validation similar to a classical database
@microscopic-receptionist-23548: Not sure what you mean by 'No splitting and no extensions'
i
+1 on Dean's solution. An ideal solution for me would also separate the models to from a base datahub engine that knows how to generate the backend CRUD based on the input models.
b
So build-time codegen?
As opposed to runtime registration of extensions
@incalculable-ocean-74010 Can you elaborate on why this is ideal for your use case?
m
s/splitting/forking/ , whoops
an ideal state for me is models can go into whatever git repo you want, today they all need to go into the DH repo to build the MCE/MAE definitions. Granted, again, I guess the UI also depends on them, so "extensions" may work for the UI. But point is the backend / models don't need to work that way
i
It's ideal because it seems more flexible & hence more powerful. Suppose there is a use-case where the entities you want to model aren't the base ones. Instead of datasets you have something else, say offices for an HR-oriented DataHub, or Features for ML, or some payment use-case. I'm thinking of datahub as a metadata search engine, the metadata model itself is not tied to a data-mindset per se.
b
Are there examples of your-company-specific entities that you have in mind? DataHub is inherently domain-specific, it's not really built to be a general content mgmt system
m
basically I'm on the same wavelength as dean and pedro, at least for models. UI is more complicated for sure...
b
That is, why would you have a scenario in which the entity you seek to model shouldn't just be contributed back?
m
even in a model federated world you can still contribute it; you just publish a new git repo 🙂 but it makes development easier since you wouldn't need to fork DH?
b
Hmm so you are saying Pedro would own his own models repo?
m
also LI has a lot of internal stuff. sadly a lot of it is stuff that should've been open sourced but it is too late now since it uses internal naming (e.g. for charts we did incharts first internally...)
b
And it would be like a plugin for datahub?
The UI is definitely trickier. CodeGen is probably our best bet in this world ^ (if we don't use existing model extensions)
m
something like that yes, though if it makes sense to contribute it back maybe "pedro's models" is not a great name but "airflow models" or something specific 🙂
b
Model extensions is more loosely typed but can be done at runtime and we can have some ability to configure how the extensions are rendered on the UI
IE a limited surface area for server-driven modules
i
Are there examples of your-company-specific entities that you have in mind?
Yes, Streams (unbounded data assets), Tables (bounded strongly typed data assets), Translations, Speeches, Videos amongst others. Some of them translate very well to other use-cases for sure, but perhaps some only make sense for a couple of companies world-wide. Do we want DataHub's metadata model to grow to encompass all cases? 
That is, why would you have a scenario in which the entity you seek to model shouldn't just be contributed back?
Perhaps it is not a entity that makes sense outside of the small & specialized scope that some one is working on.
b
@incalculable-ocean-74010 @miniature-ram-76637 Is there any use case you would have for extending a model with simple properties that can be shown in the UI in configurable ways? Trying to rule things out
@incalculable-ocean-74010 In the case that you added all of these things (Some of which I'd argue should be in native dh), would you be willing to implement support on the frontend?
Or would you expect them to somehow magically appear with some default display format
i
If we don't care about semantics I might be able to extend the DataSet entity for the notion of Translations, properties in the UI should be enough for a first run, though that depends on what users want to see.
I would fully expect to have to add "plugins" to the UI to support custom metadata models, if they aren't extended from base ones. If they extend a default metadata model, having a default display would be nice
✅ 1
m
LI uses 
Dataset
for both "streams" (kafka) and "tables" (hdfs, hive, mysql, etc)
fyi
b
Pedro has mentioned his company uses further specification as Stream and Table. Today, Pegasus is unable to model inheritance anyways... So not sure how we could further specify without just writing new entity models altogether
Unless you have seen other approaches @microscopic-receptionist-23548
m
I do think there is value in reusing as many models as possible here, even if you don't like `Dataset`; it is at least "free"
b
Is the issue that Streams + Tables cannot be individually searched + browsed?
I can see that as a reason to split
m
yeah im not familiar with Pedro's use case
b
Once you get to the Dataset page we can do whatever makes sense from UI pov to show streams and tables differently.. but getting there may be the critical thing
i
In my mind Streams + Tables have different metadata, whether they are an incomplete subset of what 
Dataset
already has is another question.
m
just curious, but what is different?
at least for LI it's worked well enough for a few years now lol
i
Pegasus allows encapsulation right? Perhaps we don't inheritance at the pdl-level
b
Yes composition
Ie Stream includes Dataset
But that doesn't help much since you want to actually extend the List of aspects
(assuming)
i
On the one hand, streams have "permissible" schemas at my company, it constantly changes while tables are fixed. This means metrics related to a stream are "estimates" and continuously changing, not so much over a table. On the other, internally our stakeholders know not to touch streams but tables are ok. They don't know that kafka is a "stream" storage. My target users don't know the difference between Hive or Kafka, nor should they. Having specialized, pluggable metadata models would allow me to make this distinction very clear.
If I have to copy parts of the Dataset aspect to model a Stream I would be ok. It would at least be possible, which is already great 🙂
@microscopic-receptionist-23548 I don't know how it is at Linkedin but I wouldn't be surprised if I got a support ticket asking why a stakeholder can't use SQL to query a Kafka-based dataset (we don't give them access to KSQL, that is used internally by the team to join streams). Users see results from DataHub showing Hive + Kafka Datasets, they don't know the difference and think it is all queryable.
m
I think our users are a little more aware of our ecosystem and the different platforms within it. Even if you can use some SQL language, accessing datasets is usually different per platform (e.g. mysql vs hive vs hdfs)
but maybe this is very confusing to people, idk 😛
i
I think it is for my non-tech colleagues. Internally we are actually thinking of using something like Trino to abstract all the different database techs.
👍 1
I may not have the most representative use-case @big-carpet-38439 😅
m
So our use case is we want to model things like data providers, their services and subscriptions and usage terms for governance tracking purposes
b
I see.... Is that a foreign key from Dataset, Dashboard, Chart, etc relationship to some provider?
m
Also let our internal users comment on entities and create reviews which would flag things for data owners to review
👍 1
b
Can you model this using DataPlatform entity type
m
I'll try post a domain meta model prototype tomorrow that we did before we discovered datahub
👍 1
b
What I'm trying to understand: Would it require both a) extension of existing models and/or b) new models?
m
Would it be worth creating a whole separate slack channel on this topic as it's a fairly big and complicated topic but also forms a pretty big core component to the whole platform and future architecture direction? Just a thought
Because there will definitely be cases where my company and probably a lot of others will want to not only extend the existing models entities and aspects but add new ones with relationships that are specific to that business
b
Mmm yeah sure
Let’s take it #datahub-modeling as there are already many channels
2 Views
Open in Slack
PreviousNext
Powered by