Dear Team After having defined a custom entity like shown he DataHub #advice-metadata-modeling

Dear Team! After having defined a custom entity li...

helpful-raincoat-19244

03/20/2023, 3:41 PM

Dear Team! After having defined a custom entity like shown here https://datahubproject.io/docs/metadata-models-custom/ , how can I check via the API or SDK if this entity type was properly installed and how can I add custom entities of this type (and define their properties etc) Thank very much in advance!

astonishing-answer-96712

03/20/2023, 6:54 PM

Hi, you can use the graphQL explorer to check if it’s there. You’ll need to use the python or java emitter to add new entities- see our APIs page here for a breakdown on capabilities https://datahubproject.io/docs/api/datahub-apis

helpful-raincoat-19244

03/21/2023, 8:07 AM

Thanks so much for your reply! Could you please clarify how I can check this using the graphQL API? I have not found an endpoint in thje API reference that would allow me to do that.. I tried to use searchAcrossEntities, but for some reason I get a "Server Error" Response (Error 500) if I try... I guess I might have missed some steps when defining the entity type. (I also see Error 500 messages if I open the Frontend after installing the entity) Thanks again!

helpful-raincoat-19244

03/23/2023, 12:54 PM

Just a small update. It seems that the errors are related to the "elasticsearch-setup"-process. If I look at the logs of datahub-gms it shows that there are exceptions when connecting to elastic search: " Elasticsearch exception [type=index_not_found_exception, reason=no such index [mycustomentityindex_v2]]" (this happen when I use the searchAcrossEntities-endpoint in the graphQL API) I tried to reindex with the hekp of the datahub-upgrade script ("./docker/datahub-upgrade/datahub-upgrade.sh -u RestoreIndices -a clean"), but it did not change anything. I tried to find steps I might have missed in the "extending the metamodel"-documentation, but I could not find anything that might be relevant.. does anyone, who already has added cutom entities to the metamodel some tipps, what I might have missed? Thanks very much!

astonishing-answer-96712

03/23/2023, 5:33 PM

Have you rebuilt the project and deployed? Your code changes wouldn’t have made it in without a rebuild

helpful-raincoat-19244

03/24/2023, 9:09 AM

I built the plugin with

Copy code

../gradlew build

and then installed it using

Copy code

./gradlew :metadata-models-custom:install

this seems to work fine, after restarting the GMS the model seems to be available

Copy code

{
  "models": {
    "mycompany-dq-model": {
      "0.0.0-dev": {
        "loadResult": "SUCCESS",
        "registryLocation": "/etc/datahub/plugins/models/mycompany-dq-model/0.0.0-dev",
        "failureCount": 0
      }
    }
  },
[...]

After that I rebuild the project using

Copy code

./gradlew quickstartDebug

and

Copy code

./gradlew :metadata-service:war:build

as was recommended here https://datahubproject.io/docs/docker/development/ I also tried to manually reindex using:

Copy code

./gradlew quickstartDebug --debug --restore-indices

After all that I still see error 500 popups inside of the frontend and get the same errors, when e.g. trying the searchAcrossEntities-endpoint in the graphQL API

Copy code

{
  "errors": [
    {
      "message": "An unknown error occurred.",
      "locations": [
        {
          "line": 34,
          "column": 3
        }
      ],
      "path": [
        "searchAcrossEntities"
      ],
      "extensions": {
        "code": 500,
        "type": "SERVER_ERROR",
        "classification": "DataFetchingException"
      }
    }
  ],
  "data": {
    "searchAcrossEntities": null
  },
  "extensions": {}
}

When I do that I see a lot of error messages in the datahub-gms logs, mostly

Copy code

2023-03-24 09:19:23,362 [ForkJoinPool.commonPool-worker-11] ERROR c.l.d.g.r.s.SearchAcrossEntitiesResolver:90 - Failed to execute search for multiple entities: entity types null, query Dashboard, filters: null, start: 0, count: 10
2023-03-24 09:19:23,362 [ForkJoinPool.commonPool-worker-11] ERROR c.l.d.g.e.DataHubDataFetcherExceptionHandler:21 - Failed to execute DataFetcher

and

Copy code

Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [<http://elasticsearch:9200>], URI [/daboindex_v2/_count?ignore_throttled=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true], status line [HTTP/1.1 404 Not Found]
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [daboindex_v2]","resource.type":"index_or_alias","resource.id":"daboindex_v2","index_uuid":"_na_","index":"daboindex_v2"}],"type":"index_not_found_exception","reason":"no such index [daboindex_v2]","resource.type":"index_or_alias","resource.id":"daboindex_v2","index_uuid":"_na_","index":"daboindex_v2"},"status":404}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:326)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:296)
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:270)
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1632)
		... 34 common frames omitted

Here is a link to the complete datahub-gms logfile

astonishing-answer-96712

04/03/2023, 10:18 PM

you likely need to run a reindex in ES after updating the model https://datahubproject.io/docs/how/restore-indices

flaky-refrigerator-97518

05/09/2023, 4:03 AM

I am getting the same error, and I tried reindexing with --restore-indices

6 Views

Open in Slack

Previous Next