Hi Team, we have installed Datahub on Kubernetes a...
# troubleshoot
k
Hi Team, we have installed Datahub on Kubernetes and our Prod cluster is down with some issue. After few Snowflake Metadata ingestions, no new data in UI is getting reflected. Even I tried to add Domain and Secrets. The messages says successfully added, but they dont reflect in UI.
👀 1
Copy code
07:53:43.868 [I/O dispatcher 1] WARN  org.elasticsearch.client.RestClient:65 - request [POST <http://elasticsearch-master:9200/_bulk?timeout=1m>] returned 1 warnings: [299 Elasticsearch-7.16.2-2b937c44140b6559905130a8650c64dbd0879cfb "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See <https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html> to enable security."]
07:53:43.869 [I/O dispatcher 1] ERROR c.l.m.s.e.update.BulkListener:25 - Failed to feed bulk request. Number of events: 1 Took time ms: -1 Message: failure in bulk execution:
[0]: index [datahubsecretindex_v2], type [_doc], id [urn%3Ali%3AdataHubSecret%3Atest2], message [[datahubsecretindex_v2/iAzRg_xGTSS88wfah3oU2g][[datahubsecretindex_v2][0]] ElasticsearchException[Elasticsearch exception [type=document_missing_exception, reason=[_doc][urn%3Ali%3AdataHubSecret%3Atest2]: document missing]]]
We see this error in GMS logs
Elastic search pods are running fine, but still seams to be some issue with Ealstic search. How to resolve it. Need a quick help here 🙏
a
Hi @kind-sunset-55628, have you recently upgraded to 0.10.0? This could be due to the reindexing that gets triggered on update
b
(True) Which version are you running?
k
@astonishing-answer-96712 @big-carpet-38439 we are using 0.9.2 and didnt upgrade to any new version
This error we see in Prod cluster just after running few Snowflake ingestions, just after fresh installation with version 0.9.2