DataHub cli - ingesting business glossaries, how t...
# troubleshoot
f
DataHub cli - ingesting business glossaries, how to determine how many of them actually got changed (not overwritten)? When ingesting my business glossaries via cli by
datahub ingest -c recipes/glossaries/glossary_recipe.yml
I'd like to learn from the output how many definitions got actually changed. But I am afraid that whenever I run this recipe all the terms get overwritten and I am always getting in the
total_records_written
the number of records in the recipe. Question: how I can determine if any glossary got modified? I need this to configure notifications for the business team (they would like to be informed whenever any glossary changes). Thanks!
✅ 1
b
hey! looking into this and will be back with an answer soon
hey @fierce-garage-74290! sorry for the delay on this one, however I think I do have a solution for you. This would actually be a great use-case for the datahub actions framework. This framework allows you to listen in for any metadata change events that we process. You can specify which events to listen to and fire off some event when those events happen. So for example, if I ingest a business glossary file that I've already ingested, but the documentation for one term has changed, only one MCE (metadata change event) will fire off with the new aspect that we create for that term. the event would look something like this:
Copy code
{
  "event_type": "MetadataChangeLogEvent_v1",
  "event": {
    "entityType": "glossaryTerm",
    "entityUrn": "urn:li:glossaryTerm:ClientsAndAccounts.Testing123",
    "changeType": "UPSERT",
    "aspectName": "glossaryTermInfo",
    "aspect": {
      "value": "{\"sourceUrl\":\"<https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Balance>\",\"customProperties\":{},\"name\":\"Testing123\",\"definition\":\"Amount of money available or owed - EDITING\",\"parentNode\":\"urn:li:glossaryNode:ClientsAndAccounts\",\"sourceRef\":\"FIBO\",\"termSource\":\"EXTERNAL\"}",
      "contentType": "application/json"
    },
    "systemMetadata": {
      "lastObserved": 1675357730805,
      "runId": "datahub-business-glossary-2023_02_02-12_08_50"
    },
    "previousAspectValue": {
      "value": "{\"sourceUrl\":\"<https://spec.edmcouncil.org/fibo/ontology/FBC/ProductsAndServices/ClientsAndAccounts/Balance>\",\"customProperties\":{},\"name\":\"Testing123\",\"definition\":\"Amount of money available or owed\",\"parentNode\":\"urn:li:glossaryNode:ClientsAndAccounts\",\"sourceRef\":\"FIBO\",\"termSource\":\"EXTERNAL\"}",
      "contentType": "application/json"
    },
    "previousSystemMetadata": {
      "lastObserved": 1675357630649,
      "runId": "datahub-business-glossary-2023_02_02-12_07_10"
    },
    "created": {
      "time": 1675357730999,
      "actor": "urn:li:corpuser:__datahub_system"
    }
  },
  "meta": {
    "kafka": {
      "topic": "MetadataChangeLog_Versioned_v1",
      "offset": 8410,
      "partition": 0
    }
  }
}
so you could easily subscribe to specific change events and for specific entities. so if you want to see whenever a glossary term has aspects change or created. check out the docs! https://datahubproject.io/docs/act-on-metadata https://datahubproject.io/docs/actions
f
Wow, that's what I was looking for. Thanks a lot!
b
you betcha!