https://datahubproject.io logo
Join Slack
Powered by
Hello Everyone. :rolling_on_the_floor_laughing: wh...
# troubleshoot
m
Hello Everyone. 🤣 what file shoud i edit host of mysql, elastic search for datahub GMS?
b
how are you deploying datahub? docker-compose?
m
gradle ..!! do you recommend docker-compose??
b
Gradle builds Datahub, but how are you deploying it. its either 1. docker-compose 2. helm chart 3. deploy without containers (not the recommended approach)
m
It would be docker compose!
b
Then look at the docker-compose file you're using to bring up the containers, refer to the .env files or the environmental variables specified directly inside the compose file, and make changes for ES, Mysql
If you using quickstart compose file, all the vars are listed on the compose file itself
m
you mean datahub/docker/quickstart?
b
So how are you bringing up datahub now? "Datahub docker quickstart"?
m
yes
b
Then you can modify the env vars for one of the all-in-one compose yml files inside docker/quickstart and use "datahub docker quickstart --quickstart-compose-file xx.yml" to launch
The container that needs updating if you use different MySQL / ES should be gms, but I can't remember if any other container uses them as well
m
Copy code
Unable to run quickstart - the following issues were detected:
- elasticsearch-setup is still running
- datahub-gms is still starting
I'm noob right now... something wrong..
b
ok, so for elasticsearch-setup container, was the env updated? and what does docker logs datahub-gms output?
m
2022/05/12 020401 Received 401 from http://192.168.0.240:9210. Sleeping 1s
oh maybe it needs username and password...
Copy code
02:06:53.472 [pool-6-thread-1] ERROR c.d.authorization.DataHubAuthorizer:236 - Failed to retrieve policy urns! Skipping updating policy cache until next refresh. start: 0, count: 30
com.datahub.util.exception.ESQueryException: Search query failed:
	at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:73)
	at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.search(ESSearchDAO.java:100)
	at com.linkedin.metadata.search.elasticsearch.ElasticSearchService.search(ElasticSearchService.java:67)
	at com.linkedin.entity.client.JavaEntityClient.search(JavaEntityClient.java:280)
	at com.datahub.authorization.PolicyFetcher.fetchPolicies(PolicyFetcher.java:45)
	at com.datahub.authorization.DataHubAuthorizer$PolicyRefreshRunnable.run(DataHubAuthorizer.java:229)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Request cannot be executed; I/O reactor status: STOPPED
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:857)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:259)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:246)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1613)
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1583)
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1553)
	at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1069)
	at com.linkedin.metadata.search.elasticsearch.query.ESSearchDAO.executeAndExtract(ESSearchDAO.java:60)
	... 12 common frames omitted
Caused by: java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
	at org.apache.http.util.Asserts.check(Asserts.java:46)
	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase.ensureRunning(CloseableHttpAsyncClientBase.java:90)
	at org.apache.http.impl.nio.client.InternalHttpAsyncClient.execute(InternalHttpAsyncClient.java:123)
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:255)
	... 18 common frames omitted
b
so now elasticsearch-setup can complete successfully, but datahub-gms is unhealthy?
m
elasticsearch-setup ends not successfully. so still datahub-gms is unhealthy. I think elasticsearch plugin not exists this is docker logs [elasticsearch-setup container]
Copy code
2022/05/12 02:44:01 Command finished successfully.
}{"error":{"root_cause":[{"type":"invalid_index_template_exception","reason":"index_template [datahub_usage_event_index_template] invalid, cause [Validation Failed: 1: unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings;]"}],"type":"invalid_index_template_exception","reason":"index_template [datahub_usage_event_index_template] invalid, cause [Validation Failed: 1: unknown setting [index.lifecycle.name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings;]"},"status":400}(
b
what version is your external ES? i don't think the ES that comes with default datahub needs any particular plugin
m
7.10.2 OSS version
b
@early-lamp-41924 any idea about the error msg?
e
Are you using aws opensearch?
m
nope! but I think it's version compatibility problem. ElasticSearch OSS version doesn't support 'index.lifecycle.name' .. so i need to edit 
create_indices.sh
e
Ah in that case, I would suggest not running elasticsearch setup job. The only thing it is doing is to set up the policy. You would also need to disable analytics feature as it depends on this policy
m
thank you so much 🙂 !!
Open in Slack
PreviousNext
Powered by