Would like to ask the people who already deployed Datahub and are ingesting Glossary terms, is there a single party who maintains the glossary terms or are there multiple parties? If multiple, how do you deconflict and make sure they do not overwrite each other? I'm thinking a single glossary repo where everyone's edits are tracked before committing to Datahub. There will be some workflow logic to check for new terms and create those terms (but ignore existing terms because I intend for users to edit descriptions via UI. And if terms are removed, to delete them.) This is because once i write the information into datahub via REST, I do not know who is the person who edits any information. (Referring to the "createdby" column in the RDBMS store, which currently does not record identity of users if it is not a UI edit)

For us, we have a git repo with a glossary.yaml file. Only the Datahub admins have write access to the

main

branch and our users create PRs. The glossary.yaml file is ingested as part of a CI/CD job.

Ah thanks for sharing @modern-monitor-81461, so glossary is not edited via the frontend UI?

@better-orange-49102 We try not to since we consider the glossary as part of the "infrastructure" and we have a philosophy of infrastructure-as-code. Same thing goes for datasets descriptions, we try as much as possible to modify those at the source (i.e. in the platform) and we let datahub ingestion bring them to the UI. To be honest, we are still early stage in Datahub adoption at my organization, so this might change at some point (we might start using the UI more and more for modifying metadata). To me, it seems you have to play one of the two extremes: • You do everything in platforms and you ingest in Datahub (so you store descriptions in SQL tables and fields and ingest in the UI) • You do everything in the UI Otherwise, how do you deconflict? Which one (UI or platform) holds the truth? Curious to hear what is your approach.

My org don't have a practice of keeping description in tables, and I have a significant amount of self created datasets that don't come from databases. I'm just thinking of restricting descriptions to only dataset owners for now since our engagements with stakeholders seem to indicate they're wary of giving the masses editorial rights. It's still very nascent and the considerations to actually roll out a catalog is giving me a headache 😅

We also have quite a few datasets that we consider "raw data" and those are csv/log/parquet/zip files stored in a datalake. We consider those "bronze" datasets and they show up in Datahub. We have 2 ways of ingesting those: 1. The data provider emits directly to Datahub 2. We store a "companion" metadata file in the datalake and we use a custom ingestion source to read that file and ingest metadata into Datahub. That companion file is a YAML file that a human can edit at will and every night, it is being crawled and ingested All in all, we are probably reluctant to move to UI-only since our Datahub deployment is not fully production yet and we fear to lose institutional knowledge if things go bad. We will need to setup DB backups and the like.

Wow! That would be a super long yaml file to edit 😂. But that's an interesting idea. Our considerations to use UI is due to a sizeable group of non technical users who could potentially share their knowledge from using the datasets, and my data publisher may not be the best placed person to advise on everything. And we need to give them a UI for them to input knowledge. I've a daily backup job for MySQL using percona xtrabackup, works rather well for my relatively small knowledge base.