https://datahubproject.io logo
Join Slack
Powered by
Hi again, I'm having some issue while deploying o...
# all-things-deployment
r
Hi again, I'm having some issue while deploying on minukube with helm chart onprem server :
root@vmi741747:~# kubectl logs datahub-elasticsearch-setup-job-cc2lq
2022/04/21 131735 Waiting for: http://elasticsearch-master:9200
2022/04/21 131736 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131738 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131740 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131742 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131744 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131746 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131748 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
2022/04/21 131750 Problem with request: Get http://elasticsearch-master:9200: dial tcp 10.103.69.1779200 connect: connection refused. Sleeping 1s
And then failing. If you have any idea i'm open to heard. Thanks again !
s
Can you share the list of pods that are running? Like from the output of 
kubectl get pods -n NAMESPACE
r
NAME READY STATUS RESTARTS AGE
datahub-elasticsearch-setup-job-4mqw8 0/1 Error 0 20m
datahub-elasticsearch-setup-job-ffbrq 0/1 Error 0 24m
datahub-elasticsearch-setup-job-fqjg9 0/1 Error 0 28m
datahub-elasticsearch-setup-job-mzfrh 0/1 Error 0 18m
datahub-elasticsearch-setup-job-n2ktz 0/1 Error 0 16m
datahub-elasticsearch-setup-job-tnlq9 0/1 Error 0 22m
datahub-elasticsearch-setup-job-trwbr 0/1 Error 0 26m
elasticsearch-master-0 0/1 Running 0 32m
elasticsearch-master-1 0/1 Pending 0 32m
elasticsearch-master-2 0/1 Pending 0 32m
prerequisites-cp-schema-registry-6f4b5b894f-7d4nw 2/2 Running 0 32m
prerequisites-kafka-0 1/1 Running 1 (32m ago) 32m
prerequisites-mysql-0 1/1 Running 0 32m
prerequisites-neo4j-community-0 1/1 Running 0 32m
prerequisites-zookeeper-0 1/1 Running 0 32m
s
Your elasticsearch nodes are not coming up. You need to check logs of why that is happening.
r
yeah i tried indeed
"stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
"at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.2.jar:7.16.2]",
i
Hello Cédric, Are you deploying datahub’s prerequisites too?
By default we have it configured for high availability, meaning that our dependencies like elastic search will try to have 3 replicas (I believe with anti-affinity rules, meaning that 1 node 1 elastic replica). See our default values: https://github.com/acryldata/datahub-helm/blob/71072cbb0550823e5c10e1f7c6d214bb57976f9d/charts/prerequisites/values.yaml#L5 And the elastic chart we are deploying: https://artifacthub.io/packages/helm/elastic/elasticsearch/7.16.2#requirements (note the 
Three Kubernetes nodes to respect the default "hard" affinity settings
) Since you are deploying on minikube you have a 1-node cluster. Unless you specified differently with:
Copy code
minikube start --nodes 3 (or more)
Elasticsearch will fail to launch. In case you haven’t it very likely means that elasticsearch deployment was not able to create the 3 replicas across different nodes.
r
Well @incalculable-ocean-74010 thanks a lot for that let me try to start a minikube with more nodes
i
Alternatively you can change the elastic search requirements to deploy only 1 replica, or change the affinity rules. This is elasticsearch specific though, not datahub.
r
Yep sure, but foir production you advice to keep this replica at 3 right ?
i'm not in production with minikube though but just if i can have the deployment in my demo env "like" in prod it's cool
still having some issues to deploy even with the ES replicas: 1
i
Are you able to deploy elastic stand-alone from the chart I mentioned with the chart’s default values except for the replica?
r
didin't try this
I got my 3 nodes minikube, i juste tried the ES deployment with default value
thank you 1
{"type": "server", "timestamp": "2022-04-21T193858,506Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "elasticsearch-master-0", "message": "uncaught exception in thread [main]",
"stacktrace": ["org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/usr/share/elasticsearch/data/nodes];",
i
Looks like a pvc issue
r
PVC ?
i
Persistent volume claim, it’s how much disk k8s deployments ask the underlying system. Your minikube instance may not have enough disk associated for what elastic search is asking for
r
Hey
Error: rendered manifests contain a resource that already exists. Unable to continue with install: PodDisruptionBudget "elasticsearch-master-pdb" in namespace "default" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "helm-es-minikube": current value is "elasticsearch"
I can try to add more disk to minicube ?
i
Looks like you have some resources in k8s that needs to be deleted (poddisruptionbudget) manually
r
applied the solution but still crashloopbackoff 😄
it's like the NPE in java 😄
```extraInitContainers:
- name: file-permissions
image: busybox
command: ['chown', '-R', '1000:1000', '/usr/share/elasticsearch/']
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: elasticsearch-master
securityContext:
privileged: true
runAsUser: 0```
Finaly add this in values.yaml
for the ES deployment
i don't get all to be honest
Well @incalculable-ocean-74010 do you have good practice for this part when you go in production ? Let ES incontainers or use on-prem instead
i
Personally I would rather just use a SaaS offering for elasticsearch, this depends on your cloud provider typically. In case that is not an option it depends on whether you have K8s expertise in house or not.
If you do, then K8s deployment, if not then on-prem, barebone but you will still need to have k8s knowledge in order to connect datahub to that elasticsearch cluster.
r
Thanks for that @incalculable-ocean-74010. In fact i'm working under Azure this time, and you can have an ES managed.
@incalculable-ocean-74010 i still have this problem with ES with a simple helm install 😕
33 Views
Open in Slack
PreviousNext
Powered by