https://datahubproject.io logo
Join Slack
Powered by
Hello, my team and I are trying to deploy DataHub ...
# troubleshoot
l
Hello, my team and I are trying to deploy DataHub in our Production environment with k8s. But we do have an error with elasticsearch :
Copy code
2023/01/10 10:19:06 Problem with request: Get <http://elasticsearch:9200>: dial tcp: lookup elasticsearch on 10.3.0.10:53: no such host. Sleeping 1s
2023/01/10 10:19:06 Timeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch:9200>]
It seems, elasticsearch is not reachable or not resolved by his IP address. For information : prerequisistes 0.0.9 datahub 0.2.89 Does anyone know what could be the problem here please ?
i
Hello Malvic, Are you trying to deploy elasticsearch from the prerequisite helm charts? Is it deployed in the same namespace as DataHub? Can you share the output of 
kubectl -n <datahub namespace> get po
?
l
Hello Pedro, thank you for your answer. The output of :
Copy code
NAME                                                READY   STATUS    RESTARTS       AGE
datahub-elasticsearch-setup-job-5mff6               0/1     Error     0              102m
datahub-elasticsearch-setup-job-6hqlg               0/1     Error     0              104m
datahub-elasticsearch-setup-job-8k5nq               0/1     Error     0              98m
datahub-elasticsearch-setup-job-d8pk8               0/1     Error     0              108m
datahub-elasticsearch-setup-job-hgtl6               0/1     Error     0              106m
datahub-elasticsearch-setup-job-rjlwk               0/1     Error     0              110m
datahub-elasticsearch-setup-job-xtf89               0/1     Error     0              100m
elasticsearch-master-0                              1/1     Running   0              3h3m
elasticsearch-master-1                              1/1     Running   0              3h3m
elasticsearch-master-2                              1/1     Running   0              3h3m
prerequisites-cp-schema-registry-7cc6786995-slw48   2/2     Running   0              3h3m
prerequisites-kafka-0                               1/1     Running   1 (3h2m ago)   3h3m
prerequisites-mysql-0                               1/1     Running   0              3h3m
prerequisites-neo4j-community-0                     1/1     Running   0              3h3m
prerequisites-zookeeper-0                           1/1     Running   0              3h3m
And, my colleague told me the request are received by our coreDNS :
Copy code
[INFO] 10.2.8.44:49642 - 24525 "A IN elasticsearch.datahub.svc.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.000142637s
[INFO] 10.2.8.44:43250 - 35567 "AAAA IN elasticsearch.datahub.svc.cluster.local. udp 57 false 512" NXDOMAIN qr,aa,rd 150 0.00031082s
[INFO] 10.2.8.44:37106 - 10884 "A IN elasticsearch.svc.cluster.local. udp 49 false 512" NXDOMAIN qr,aa,rd 142 0.000206884s
[INFO] 10.2.8.44:54621 - 727 "AAAA IN elasticsearch.cluster.local. udp 45 false 512" NXDOMAIN qr,aa,rd 138 0.000083844s
[INFO] 10.2.8.44:34712 - 40990 "A IN elasticsearch. udp 31 false 512" NXDOMAIN qr,aa,rd,ra 106 0.000109917s
i
Elasticsearch setup jobs are erroring out. Could you share those logs?
l
Yes, it's the one I sent on my first message :
Copy code
2023/01/10 12:01:38 Waiting for: <http://elasticsearch:9200>
2023/01/10 12:01:38 Problem with request: Get <http://elasticsearch:9200>: dial tcp: lookup elasticsearch on 10.3.0.10:53: no such host. Sleeping 1s
[...]
2023/01/10 12:03:37 Problem with request: Get <http://elasticsearch:9200>: dial tcp: lookup elasticsearch on 10.3.0.10:53: no such host. Sleeping 1s
2023/01/10 12:03:38 Timeout after 2m0s waiting on dependencies to become available: [<http://elasticsearch:9200>]
i
If DataHub was deployed with Helm could you share your values.yaml? In particular the section 
global.elasticsearch
l
Copy code
global:                                                                             
  graph_service_impl: neo4j                                                         
  datahub_analytics_enabled: true                                                   
  datahub_standalone_consumers_enabled: false                                       
                                                                                    
  elasticsearch:                                                                                                                                                                                                 
    host: "elasticsearch-master"                                                    
    port: "9200"
i
What is the output of 
kubectl -n <datahub namespace> get svc
?
Your chart version is quite old, I would suggest going to the latest: 
0.2.132
By default it deploys DataHub v0.9.5, so I would suggest you override that config on your side to deploy whatever version of DataHub you are deploying.
l
The output for 
kubectl -n <datahub namespace> get svc
Copy code
NAME                               TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
elasticsearch-master               ClusterIP   10.3.160.113   <none>        9200/TCP,9300/TCP            5h34m
elasticsearch-master-headless      ClusterIP   None           <none>        9200/TCP,9300/TCP            5h34m
prerequisites-cp-schema-registry   ClusterIP   10.3.17.117    <none>        8081/TCP,5556/TCP            5h34m
prerequisites-kafka                ClusterIP   10.3.253.99    <none>        9092/TCP                     5h34m
prerequisites-kafka-headless       ClusterIP   None           <none>        9092/TCP,9093/TCP            5h34m
prerequisites-mysql                ClusterIP   10.3.147.131   <none>        3306/TCP                     5h34m
prerequisites-mysql-headless       ClusterIP   None           <none>        3306/TCP                     5h34m
prerequisites-neo4j-community      ClusterIP   None           <none>        7474/TCP,7687/TCP            5h34m
prerequisites-zookeeper            ClusterIP   10.3.25.90     <none>        2181/TCP,2888/TCP,3888/TCP   5h34m
prerequisites-zookeeper-headless   ClusterIP   None           <none>        2181/TCP,2888/TCP,3888/TCP   5h34m
I'll ask to my infrastructure team about the version used
s
I'm having the same problem here but not in k8s, using docker-compose.
i
Is this a custom docker-compose file or though the datahub cli?
s
I only changed these passwords, usernames and secrets: • EBEAN_DATASOURCE_PASSWORD • MYSQL_PASSWORD • MYSQL_ROOT_PASSWORD • MYSQL_PASSWORD • DATAHUB_SYSTEM_CLIENT_SECRET • DATAHUB_SECRET @incalculable-ocean-74010
i
That is strange. What version of datahub have you deployed?
s
v0.9.5
@incalculable-ocean-74010 would it be possible to change the user and password by cli?
i
No, we do not have such a feature
s
😞 but how can I change the traditional login=datahub and password = datahub?
i
You would have to mount the user file, this is usually not something you do on a docker-compose based deployment since that is meant for local PoC only. In a production deployment we highly recommend kubernetes and helm charts where you can easily specify those configs.
s
I'll do it, but i'm trying first of all using the docker compose to test some stuffs. I already changed it in EBEAN_DATASOURCE_USERNAME EBEAN_DATASOURCE_PASSWORD But I still login using the default login and pass
apparently when I'm changing it - DATAHUB_SYSTEM_CLIENT_ID=__datahub_system - DATAHUB_SYSTEM_CLIENT_SECRET=JohnSnowKnowsNothing I'm having problems
l
Does the output for 
kubectl -n <datahub namespace> get svc
helped you somehow @incalculable-ocean-74010?
i
It did not Malvic, I wanted to certify that the prerequisite charts were deployed in the same namespace that datahub was and that the service names matched what the job was trying to access. I suspect there is a bug with chart version that are using and should update to the latest.
l
Thank you, we'll try that and I'll let you know if it worked
5 Views
Open in Slack
PreviousNext
Powered by