Hi, we set datahub to work with our managed elasti...
# troubleshoot
b
Hi, we set datahub to work with our managed elastic and then decided to move it back to work with prerequisites elastic. After deploying both prerequisites & datahub everything works fine and I see all pods are running but after running the reindexes job ui is still empty and it says “no metadata found” though we do have entities and ingestion already done and saved in DB. We are using helm to deploy and the only error I see is in the Elastic logs attached in the first comment
✅ 1
Copy code
{
  "type": "server",
  "timestamp": "2023-05-02T12:07:54,099Z",
  "level": "ERROR",
  "component": "o.e.i.g.DatabaseNodeService",
  "cluster.name": "elasticsearch",
  "node.name": "elasticsearch-master-0",
  "message": "failed to retrieve database [GeoLite2-Country.mmdb]",
  "stacktrace": [
    "org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
    "at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:179) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:165) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:929) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:763) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.lambda$executeRequest$6(TransportSearchAction.java:399) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:136) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:112) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:77) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:487) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:285) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:145) ~[?:?]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:154) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:82) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:95) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:73) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:57) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.OriginSettingClient.doExecute(OriginSettingClient.java:51) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:407) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:392) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:542) ~[elasticsearch-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.ingest.geoip.DatabaseNodeService.lambda$retrieveDatabase$11(DatabaseNodeService.java:367) [ingest-geoip-7.17.3.jar:7.17.3]",
    "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.17.3.jar:7.17.3]",
    "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
    "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
    "at java.lang.Thread.run(Thread.java:833) [?:?]"
  ]
}
l
Hey there 👋 I'm The DataHub Community Support bot. I'm here to help make sure the community can best support you with your request. Let's double check a few things first: 1️⃣ There's a lot of good information on our docs site: www.datahubproject.io/docs, Have you searched there for a solution? Yes button 2️⃣ It's not uncommon that someone has run into your exact problem before in the community. Have you searched Slack for similar issues? Yes button
b
is this error related?
"level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "failed to retrieve database [GeoLite2-Country.mmdb]",
GMS log:
Copy code
05:47:49.074 [pool-12-thread-2] INFO  c.l.m.filter.RestliLoggingFilter:55 - GET /entitiesV2?ids=List(urn%3Ali%3Acorpuser%3Adatahub) - batchGet - 200 - 5ms
05:47:49.075 [Thread-173] WARN  org.elasticsearch.client.RestClient:65 - request [POST http://elasticsearch-master:9200/datahub_datahub_usage_event/_search?typed_keys=true&max_concurrent_shard_requests=5&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=query_then_fetch&batched_reduce_size=512&ccs_minimize_roundtrips=true] returned 2 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."],[299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
05:47:49.075 [Thread-170] WARN  org.elasticsearch.client.RestClient:65 - request [POST http://elasticsearch-master:9200/datahub_datahubaccesstokenindex_v2/_count?ignore_throttled=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true] returned 2 warnings: [299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."],[299 Elasticsearch-7.17.3-5ad023604c8d7416c9eb6c0eadb62b14e766caff "[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]
Im using version 0.9.6
seems like all the indices were created
Copy code
Defaulted container "elasticsearch" out of: elasticsearch, configure-sysctl (init)
elasticsearch@elasticsearch-master-0:~$ curl -XGET http://localhost:9200/_cat/indices?v
health status index                                                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   datahub_mlmodeldeploymentindex_v2                                EponoUSmQUqME52luiTPbQ   1   1          0            0       452b           226b
green  open   datahub_datahubretentionindex_v2                                 F9ec98ZrSMKXvBBLBjbouw   1   1          0            0       452b           226b
green  open   datahub_dataprocessinstanceindex_v2                              KH8MqB9SQh6meFdVpSk_3w   1   1          0            0       452b           226b
green  open   datahub_corpuserindex_v2                                         5pX6NmcLSia475XumSwvLg   1   1          1            0      8.8kb          4.4kb
green  open   datahub_system_metadata_service_v1                               gHNFkW5TRjCy6QUD29lWGw   1   1          1            0     11.9kb          5.9kb
green  open   datahub_corpgroupindex_v2                                        BQoL6gHCQcCZwlmDvNx7Ew   1   1          0            0       452b           226b
green  open   .ds-datahub_datahub_usage_event-2023.05.02-000001                BhD2ffLHSFulj_tcszIFYA   1   1         27            0    177.5kb         88.7kb
green  open   datahub_mlfeatureindex_v2                                        EN8BECTpQ76OIDCtERh6RQ   1   1          0            0       452b           226b
green  open   datahub_datajob_datahubingestionrunsummaryaspect_v1              t5N6-RyKT7-6R0mw-TCRAw   1   1          0            0       452b           226b
green  open   datahub_datajob_datahubingestioncheckpointaspect_v1              Yct9ZisdQTqqvOGrmLQ7Dg   1   1          0            0       452b           226b
green  open   datahub_datahubingestionsourceindex_v2                           6R9sVuFORSu423ueLzzlYA   1   1          0            0       452b           226b
green  open   datahub_dataplatforminstanceindex_v2                             yMGFbh-ASraZOcNPGKkSxw   1   1          0            0       452b           226b
green  open   datahub_datajobindex_v2                                          UxU6PuwZQeu2e2xcw4wrKw   1   1          0            0       452b           226b
green  open   datahub_tagindex_v2                                              zdUsX4ycSIGkicHDBxGXVg   1   1          0            0       452b           226b
green  open   datahub_dataplatformindex_v2                                     XmJs31uZRzyNa8f-YSHl9A   1   1          0            0       452b           226b
green  open   datahub_dataset_datasetusagestatisticsaspect_v1                  hB55UP9lTPG0bZu-IsUsAA   1   1          0            0       452b           226b
green  open   datahub_postindex_v2                                             sxVt-o9QSBW3tnT3917bqA   1   1          0            0       452b           226b
green  open   datahub_domainindex_v2                                           1MJLE0lZSc-WzgCeXGA6jA   1   1          0            0       452b           226b
green  open   datahub_dataprocessinstance_dataprocessinstanceruneventaspect_v1 u1AiM3dKSJS4vjbqUqZ8ow   1   1          0            0       452b           226b
green  open   datahub_dataprocessindex_v2                                      -y15tcoySHmHECFPiowdpw   1   1          0            0       452b           226b
green  open   datahub_dashboardindex_v2                                        u_-k2AEjRl-sgOnmKig4gA   1   1          0            0       452b           226b
green  open   datahub_datahubaccesstokenindex_v2                               PZHL30xEQ2eLzdjKAbuvmQ   1   1          0            0       452b           226b
green  open   datahub_schemafieldindex_v2                                      xyGmBpiTSNe8JNPAoLW_pA   1   1          0            0       452b           226b
green  open   datahub_datahubviewindex_v2                                      dtjRTh1rR2qbCTWV4-UcVQ   1   1          0            0       452b           226b
green  open   datahub_dataflowindex_v2                                         3JSW-IEzQG6SoHdtVeLGlg   1   1          0            0       452b           226b
green  open   datahub_chartindex_v2                                            DsELn9CdSf6litse6ruCRw   1   1          0            0       452b           226b
green  open   datahub_globalsettingsindex_v2                                   94R6naA7QA2G9VjnJfROUQ   1   1          0            0       452b           226b
green  open   datahub_assertion_assertionruneventaspect_v1                     _1ZjEr4nTNqpN4mEIQYbaQ   1   1          0            0       452b           226b
green  open   datahub_mlfeaturetableindex_v2                                   zDJEiGxjRFK2fvMAjU2WGQ   1   1          0            0       452b           226b
green  open   .geoip_databases                                                 PuYbdORsRaeHsF2RCgmbcg   1   1         42            0     80.1mb           40mb
green  open   datahub_glossarytermindex_v2                                     4Z_4j_cpThqul39PIzNpuw   1   1          0            0       452b           226b
green  open   datahub_dataset_datasetprofileaspect_v1                          M1NO60tsTYCQf0upsc16rg   1   1          0            0       452b           226b
green  open   datahub_mlmodelindex_v2                                          QwMlGS7ASzSYisaci0Ozcg   1   1          0            0       452b           226b
green  open   datahub_mlmodelgroupindex_v2                                     ruoBEVr0TqOpXifVzs8qKw   1   1          0            0       452b           226b
green  open   datahub_datahubstepstateindex_v2                                 -dvfxAIFRqeso6qcUvkPzw   1   1          0            0       452b           226b
green  open   datahub_datahubroleindex_v2                                      r3fd1gDtQ-SLp2-EIe4ftg   1   1          3            0     15.5kb          7.7kb
green  open   datahub_dataset_operationaspect_v1                               mGMkLThmSwKGLFLa7hObdQ   1   1          0            0       452b           226b
green  open   datahub_datasetindex_v2                                          AfzNkchHSWCKdvgQi6aVdw   1   1          0            0       452b           226b
green  open   datahub_datahubexecutionrequestindex_v2                          -kw5ejbJSf2x7jkskJR_6g   1   1          0            0       452b           226b
green  open   datahub_notebookindex_v2                                         4spbK7sRTtuv33EezPl4xQ   1   1          0            0       452b           226b
green  open   datahub_datahubpolicyindex_v2                                    LDRaBw5vSpudKbUGhQvM_w   1   1         12            0     29.9kb         14.9kb
green  open   datahub_containerindex_v2                                        0Uaz47qFSPOaS6v4gN-6qA   1   1          0            0       452b           226b
green  open   datahub_dashboard_dashboardusagestatisticsaspect_v1              P2m6mmdQTtSZAjgUpgBVqQ   1   1          0            0       452b           226b
green  open   datahub_telemetryindex_v2                                        -eKwcO3TRISwkX8mhtJp4w   1   1          0            0       452b           226b
green  open   datahub_datahubsecretindex_v2                                    twFKDchMSbaUBQyjBG-fxA   1   1          0            0       452b           226b
green  open   datahub_mlprimarykeyindex_v2                                     cthRWezRRQa31rlvpzgcRg   1   1          0            0       452b           226b
green  open   datahub_datahubupgradeindex_v2                                   mL0wd-SrQVi3tUtIadh8kg   1   1          0            0       452b           226b
green  open   datahub_glossarynodeindex_v2                                     A7x-q0ZNQkO4m5EuK6MqZQ   1   1          0            0       452b           226b
green  open   datahub_invitetokenindex_v2                                      HoSsV9dBRxqFpIyfPihfBg   1   1          0            0       452b           226b
green  open   datahub_chart_chartusagestatisticsaspect_v1                      PZR_y9I2QYalQLMP2j80pQ   1   1          0            0       452b           226b
green  open   datahub_testindex_v2                                             cfGP1eUcS0GwQy4ibdnpFg   1   1          0            0       452b           226b
green  open   datahub_assertionindex_v2                                        kCEOkPyzRx-UaRhbIY_L1w   1   1          0            0       452b           226b
green  open   datahub_graph_service_v1                                         FFZFINNlSluDY7uc8x1snQ   1   1          0            0       452b
It worked after upgrading to v0.9.6.1