Hi i am just getting started with datahubproject a...
# getting-started
f
Hi i am just getting started with datahubproject and was just wondering about workflow. I am part of a platform team that is going to enable datahub as a metadata/data discovery tool for all teams using our platform. It will be as a platform capability. Now different teams use different types of data technology (mysql, postgres, elasticsearch and kafka). How can we as a platform team ensure that the metadata is continuously updated and aligned with the data services? How would the workflow be? Does each team submit a request to add their metadata to the datahub?
d
Hi, I think schedule ingestion (each team can run their own ingestion using their own scheduling tools - etc. airflow) would work best in this case!
If the team wants to push metadata out on every change, they can utilize our SDKs to do this.
f
but will this mean that we still run a centralized datahub instance where all metadata is displayed?
Okay so here is an example, correct me if i am wrong: We run a centralized datahub instance maintained by the platform team:
team_x
: deploys a cron job featuring a schedule ingestion for ingesting metadata from mysql db.
team_y
: deploys a cron job fdeaturing a schedule ingestion for ingesting metadata from an elasticsearch database. Both these cronjobs can be run as a kubernetes cron job inside a kubernetes cluster. @delightful-ram-75848 is this correct way of thinking?