Hey @mammoth-bear-12532, We’ve upgrade DataHub version to v0.8.18 and we would like to test metadata-models-custom (e.g. “DQ Rules”) into our setup. Our DataHub deployment is running on the top of Kubernetes. Due to that reason, we don’t know where we need to upload the generated

metadata-models-custom-0.0.1-dev.zip

file? What’s the corresponding path of

~/.datahub/plugins/models/<registry-name>/<registry-version>/

in Kubernetes setup?

is it

/etc/datahub/plugins

folder in GMS pod?

The rest of the subtree needs to match. (models/registry-name/registry-version)

I found the culprit

19:07:05 [main] WARN  c.l.m.m.r.PluginEntityRegistryLoader - /home/datahub/plugins/models directory does not exist or is not a directory. Plugin scanning will be disabled.

I need to create the directory in advance. I think it should be created automatically. I’m not sure why it requires me to create that beforehand

here: https://github.com/linkedin/datahub/blob/master/entity-registry/src/main/java/com/[…]nkedin/metadata/models/registry/PluginEntityRegistryLoader.java

🔥 🙂

I’ve a couple of questions @mammoth-bear-12532 • After registering the custom model, is it safe to delete them from the particular folder? EntityRegistryLoader class tries to load that

20:47:00 [pool-3-thread-1] DEBUG c.l.m.m.r.PluginEntityRegistryLoader - Will be loading paths in this order /home/datahub/seref/0.0.1
20:47:00 [pool-3-thread-1] DEBUG c.l.m.m.r.PluginEntityRegistryLoader - com.linkedin.metadata.models.registry.PluginEntityRegistryLoader@182f27ac: Found registry version 0.0.1
20:47:00 [pool-3-thread-1] DEBUG c.l.m.m.r.PluginEntityRegistryLoader - Registry seref:0.0.1 already exists. Skipping loading...

• Is it easy to delete the model if their registration is failed ? When I look at the

/config

endpoint, I see the following errors and I would like to delete them.

{
  "models": {
    "udemy_dq": {
      "0.0.1": {
        "loadResult": "FAILURE",
        "registryLocation": "/home/datahub/udemy_dq/0.0.1",
        "failureReason": "java.lang.IllegalArgumentException: Aspect customDataQualityRules does not exist\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.getAspectSpec(PatchEntityRegistry.java:157)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.lambda$new$4(PatchEntityRegistry.java:135)\n\tat java.util.ArrayList.forEach(ArrayList.java:1259)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:134)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:102)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:63)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:69)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.loadOneRegistry(PluginEntityRegistryLoader.java:146)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$null$5(PluginEntityRegistryLoader.java:100)\n\tat java.util.ArrayList.forEach(ArrayList.java:1259)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$start$6(PluginEntityRegistryLoader.java:100)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",
        "failureCount": 5
      },
      "0.0.2": {
        "loadResult": "FAILURE",
        "registryLocation": "/home/datahub/udemy_dq/0.0.2",
        "failureReason": "com.linkedin.metadata.models.registry.EntityRegistryException: Did not find an entity registry (entity-registry.yaml/yml) under /home/datahub/udemy_dq/0.0.2\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.getFileAndClassPath(PatchEntityRegistry.java:83)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:69)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.loadOneRegistry(PluginEntityRegistryLoader.java:146)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$null$5(PluginEntityRegistryLoader.java:100)\n\tat java.util.ArrayList.forEach(ArrayList.java:1259)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$start$6(PluginEntityRegistryLoader.java:100)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",
        "failureCount": 5
      }
    },
    "udemy_data_quality": {
      "0.0.1": {
        "loadResult": "FAILURE",
        "registryLocation": "/home/datahub/udemy_data_quality/0.0.1",
        "failureReason": "java.lang.IllegalArgumentException: Aspect customDataQualityRules does not exist\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.getAspectSpec(PatchEntityRegistry.java:157)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.lambda$new$4(PatchEntityRegistry.java:135)\n\tat java.util.ArrayList.forEach(ArrayList.java:1259)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:134)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:102)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:63)\n\tat com.linkedin.metadata.models.registry.PatchEntityRegistry.<init>(PatchEntityRegistry.java:69)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.loadOneRegistry(PluginEntityRegistryLoader.java:146)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$null$5(PluginEntityRegistryLoader.java:100)\n\tat java.util.ArrayList.forEach(ArrayList.java:1259)\n\tat com.linkedin.metadata.models.registry.PluginEntityRegistryLoader.lambda$start$6(PluginEntityRegistryLoader.java:100)\n\tat java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n",
        "failureCount": 5
      }
    },
    "seref": {
      "0.0.1": {
        "loadResult": "SUCCESS",
        "registryLocation": "/home/datahub/seref/0.0.1",
        "failureCount": 0
      }
    },
    "udemy_dm": {
      "0.0.1": {
        "loadResult": "SUCCESS",
        "registryLocation": "/home/datahub/udemy_dm/0.0.1",
        "failureCount": 0
      }
    }
  },
  "noCode": "true"
}

For the failed models, you can just remove them from the config map and restart the container and that will clear out the in-memory load status. If you have already removed them, just restarting the container should clear out the error status.

let me know if that was not clear