This message was deleted.
# troubleshootings
Slackbot
03/30/2023, 3:05 AMThis message was deleted.
a
Abhishek Agarwal
03/30/2023, 5:42 AMyou should look into the coordinator logs and find out where is 500 coming from
s
shraddha jain
03/30/2023, 4:33 PMseems like resources issue. Even after increasing the resources limits, it is consuming all the resources and failing. What could be the issue?
shraddha jain
03/30/2023, 4:44 PMHi @Vijay Narayanan, do you have any idea?
v
Vijay Narayanan
03/30/2023, 4:47 PMplease paste your helm chart here
s
shraddha jain
03/30/2023, 5:43 PMhere it is
d
Didip Kerabat
03/30/2023, 6:35 PMdid kubernetes kill it and restarting it continuously?
s
shraddha jain
03/30/2023, 6:53 PMyes
d
Didip Kerabat
03/30/2023, 8:50 PMthen you ran out of memory. Maybe the Xmx and “physical” memory settings are not enough.
s
shraddha jain
03/30/2023, 9:00 PMwhen I increased the memory in resources limit in coordinator, after the re-run, it used all that memory as well.
shraddha jain
03/30/2023, 9:06 PMdo you mean memory on node?
d
Didip Kerabat
03/30/2023, 10:28 PMboth the JVM and the pods’ memory.
s
shraddha jain
03/30/2023, 10:29 PMok, let me try to increase both.
shraddha jain
03/31/2023, 5:15 PMI am still seeing the same issue in the coordinator pod
Copy code
2023-03-31T17:11:07.286171Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/healths
Sergio Ferragut
03/31/2023, 5:43 PMIs the coordinator still restarting? Can you look at the logs to see why? If it is using up all the memory at startup it probably means you have a large number of segments. How many segments in the cluster? Avg size?
s
shraddha jain
03/31/2023, 6:14 PMHi Sergio, yes the coordinator is still restarting. Below are the istio-proxy container logs:
Copy code
│ {"upstream_transport_failure_reason":null,"upstream_service_time":null,"connection_termination_details":null,"bytes_received":0,"upstream_local_address":null,"duration":0,"response_code_ │
│ {"downstream_remote_address":"100.72.30.199:37776","requested_server_name":null,"upstream_transport_failure_reason":null,"route_name":"allow_any","upstream_local_address":"100.72.30.199: │
│ {"request_id":null,"connection_termination_details":null,"x_serial_number":null,"response_code_details":null,"upstream_host":"10.236.224.54:5432","authority":null,"downstream_remote_addr │
│ {"path":null,"response_flags":"UF,URX","duration":10001,"bytes_received":0,"route_name":null,"request_id":null,"response_code":0,"x_forwarded_for":null,"upstream_transport_failure_reason │
│ {"authority":null,"duration":10001,"path":null,"bytes_sent":0,"downstream_local_address":"10.236.224.54:5432","bytes_received":0,"user_agent":null,"response_flags":"UF,URX","upstream_loc │
│ {"connection_termination_details":null,"upstream_local_address":null,"duration":10000,"x_serial_number":null,"method":null,"request_id":null,"response_code_details":null,"authority":null │
│ 2023-03-31T18:13:47.286440Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/health │
│ 2023-03-31T18:13:47.286466Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/health │
│ {"upstream_cluster":"PassthroughCluster","upstream_transport_failure_reason":null,"user_agent":null,"bytes_received":0,"bytes_sent":0,"x_forwarded_for":null,"request_id":null,"response_c │
│ 2023-03-31T18:13:57.286699Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/health │
│ 2023-03-31T18:13:57.286775Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/health │
│ 2023-03-31T18:14:01.237287Z error Request to probe app failed: Get "<http://100.72.30.199:8081/status/health>": dial tcp 127.0.0.6:0->100.72.30.199:8081: connect: connection refused, │
│ app URL path = /status/healthshraddha jain
03/31/2023, 6:16 PMthese are the events:
Copy code
18m Warning Unhealthy pod/druid-coordinator-5ffb44448-6wxt8 Readiness probe failed: HTTP probe failed with statuscode: 500
2m48s Warning Unhealthy pod/druid-coordinator-5ffb44448-6wxt8 Liveness probe failed: HTTP probe failed with statuscode: 500
8m5s Warning BackOff pod/druid-coordinator-5ffb44448-6wxt8 Back-off restarting failed containershraddha jain
03/31/2023, 6:34 PMsorry for my naive questions, I am new to this, where can I find the segments?
s
Sergio Ferragut
03/31/2023, 6:42 PMIn the Druid console. at
http://<router>:8888Sergio Ferragut
03/31/2023, 6:49 PMThe coordinator loads a full map of all segment metadata into memory. Too many segments may overload the coordinator which means it will need more resources: cpu and memory. It is a good practice to optimize segment sizes, this is a good write up on the subject. Essentially, you want to ingest such that you create segments of approx 5 million rows and 500-700MB in size. You can adjust your segment granularity, maxRowsPerSegment, secondary partitioning to achieve this. With streaming ingestion, you will also want to use compaction (ideally auto-compaction) to optimize real-time segments after ingestion.
s
shraddha jain
03/31/2023, 7:49 PMI am doing a new setup of Druid. Probably we will see the segments once the load is there on app. This is what I see as of now:
s
Sergio Ferragut
03/31/2023, 8:21 PMSeems to be having some trouble finding a server. Sometimes this happens when the services haven't all started yet. If you click on the "Druid" icon, you can see the state of the cluster:
s
shraddha jain
03/31/2023, 8:28 PMyeah that's mostly because of the same coordinator issue
s
Sergio Ferragut
03/31/2023, 8:34 PMSo, if this is a brand new cluster with new metadata DB and empty deep storage, then it isn’t about the segments. you can check the coordinator log to see if there are any errors.
s
shraddha jain
03/31/2023, 8:40 PMyeah, the coordinator container logs are pasted above in the same chat. Probes are failing.
s
Sergio Ferragut
03/31/2023, 8:49 PMThose are Kubernetes container logs right? I mean the coordinator process log. You should be able to see it with a
kubectl logs <podname>s
shraddha jain
03/31/2023, 8:51 PMyeah right, below are the pod logs:
Copy code
2023-03-31T20:45:38+0000 startup service coordinator
Setting druid.host=100.96.16.155 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.type=google in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-google-extensions"] in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.prefix=indexing-logs in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.http.recipientBaseUrl=<http://druid_exporter_url>:druid_exporter_port/druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.type=postgresql in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.bucket=roku-druid-prod in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.connectURI=jdbc:<postgresql://10.236.224.54:5432/druid_prod> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.storage.type=google in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.user=druid-user in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.zk.service.host=druid-zookeeper-headless:2181 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.google.prefix=segments in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter=logging in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.password=temp_password in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.logging.logLevel=debug in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.indexer.logs.bucket=roku-druid-prod in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.propertiesshraddha jain
03/31/2023, 10:29 PMFor everyone here, there are was an issue related to postgresql password which was a mismatch with the password of the hosted posgtgresql. Thank you @Sergio Ferragut for helping there.
😎 1
y
Yash Raj Pathak
04/16/2023, 8:28 AMHi @shraddha jain & @Sergio Ferragut I'm new to DRUID and my management asked me to deploy DRUID cluster production with EKS, It would great help if you folks help me with your experience in this journey like where to find relevant helm chart .
s
Sergio Ferragut
04/17/2023, 2:20 PMThe helm chart is https://github.com/apache/druid/blob/master/helm/druid/README.md
y
Yash Raj Pathak
04/17/2023, 2:24 PMThank you @Sergio Ferragut I am following this procedure, however I'm stuck in pod crashloopbackoff error , using ingress for routing
Copy code
druid druid-broker-6965dbd8d9-gt72w 0/1 CrashLoopBackOff 7 (3m3s ago) 14m
druid druid-coordinator-7d7c7988d5-kgr52 0/1 CrashLoopBackOff 7 (3m5s ago) 14m
druid druid-historical-0 0/1 Pending 0 14m
druid druid-middle-manager-0 0/1 Pending 0 14m
druid druid-postgresql-0 0/1 Pending 0 14m
druid druid-router-7744fc9689-dcf6x 0/1 CrashLoopBackOff 7 (3m3s ago) 14Yash Raj Pathak
04/17/2023, 2:25 PMlogs:-
Copy code
druid
2023-04-17T14:14:50+0000 startup service coordinator
Setting druid.host=172.21.78.117 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.extensions.loadList=["druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage", "druid-kafka-indexing-service", "druid-hdfs-storage", "druid-s3-extensions", druid-parquet-extensions", druid-multi-stage-query"] in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.type=postgresql in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.http.recipientBaseUrl=<http://druid_exporter_url>:druid_exporter_port/druid in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.connectURI=jdbc:<postgresql://druid-postgresql:5432/adtech-druid-staging> in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.user=postgres in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.zk.service.host=druid-zookeeper-headless:2181 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter=noop in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.metadata.storage.connector.password=ftuHCFK2oqhf0cHkCRhY in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Setting druid.emitter.logging.logLevel=debug in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties
Exception in thread "main" com.google.inject.ProvisionException: Unable to provision, see the following errors:
1) Problem parsing object at prefix[druid.extensions]: Cannot deserialize instance of `java.util.LinkedHashSet<java.lang.Object>` out of VALUE_STRING token
at [Source: UNKNOWN; line: -1, column: -1] (through reference chain: org.apache.druid.guice.ExtensionsConfig["loadList"]).
at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151)
at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:151)
while locating com.google.common.base.Supplier<org.apache.druid.guice.ExtensionsConfig>
at org.apache.druid.guice.JsonConfigProvider.bind(JsonConfigProvider.java:152)
while locating org.apache.druid.guice.ExtensionsConfig
for the 1st parameter of org.apache.druid.guice.ExtensionsLoader.<init>(ExtensionsLoader.java:72)
at org.apache.druid.guice.ExtensionsModule.configure(ExtensionsModule.java:37)
while locating org.apache.druid.guice.ExtensionsLoaders
Sergio Ferragut
04/17/2023, 2:25 PMthere's a way to check the error with kubectl....finding it...
Sergio Ferragut
04/17/2023, 2:26 PMhere's a great writeup: https://foxutech.medium.com/kubernetes-crashloopbackoff-how-to-troubleshoot-940dbb16bc84
🙌 1
Sergio Ferragut
04/17/2023, 2:28 PMare you missing some quotes on the items in the loadList?
y
Yash Raj Pathak
04/17/2023, 2:45 PMit seems there was some indent error in values.yaml.. Thanks !!
s
Sergio Ferragut
04/17/2023, 5:09 PMI know it is in active development. Feel free to fix and submit a PR. I did a small minikube deployment a while back and documented the process in this blog. The helm chart has probably evolved since then, but maybe it will help:
🙌 1
y
Yash Raj Pathak
04/17/2023, 6:11 PMThanks for the informative document
4 Views