Hi guys
we are using airflow for ingestion and its...
# ingestionh
hallowed-article-64840
12/14/2021, 8:09 AMHi guys
we are using airflow for ingestion and its working fine
but we need a mechanism for detecting changes to inform other teams. for newly added table and metadata change for current tables we need to be informed.
I was looking kafka topics and thought MetadataChangeEvent_v4 may can help, but its filled in each ingestion interval for each tables regardless of its changed or not.
how we can do that in datahub ?
e
early-lamp-41924
12/14/2021, 9:01 AMHi. MetadataAuditEvent_v4 and MetadataChangeLog_Versioned_v1 would be the best topics. Note, we are currently migrating away from the MAE topic to the MCL topic, so in the near future, you can just listen to the second topic.
early-lamp-41924
12/14/2021, 9:01 AMSo ingesting new metadata can happen through 2 sources
early-lamp-41924
12/14/2021, 9:02 AMFirst, is by pushing events to the MetadataChangeEvent_v4. the other is to use the rest ingest api. Bc of this MetadataChangeEvent_v4 may not contain all the changes. Also, once an ingest event comes through the above 2 sources, it compares it with the latest version stored in MySQL to see if there was actually a change in the aspect.
early-lamp-41924
12/14/2021, 9:03 AMIf there is a change, it updates MySQL with a new version and emits to the above 2 topics. So in turn, the above two topics will only get events if there is a change making it the perfect candidate to detect change (or addition of new) in datasets
h
hallowed-article-64840
12/15/2021, 9:40 AMthanks @early-lamp-41924, its was so clear and helpful