https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
Do you see any errors in the overlord log?
c
I snagged the entire block of config properties I set for reference,
Copy code
druid_zk_service_enabled: "false"
  druid_serverview_type: "http"
  druid_discovery_type: "k8s"
  druid_discovery_k8s_clusterIdentifier: "test-druid"
  druid_discovery_k8s_podNameEnvKey: "HOSTNAME"
  druid_coordinator_loadqueuepeon_type: "http"
  druid_indexer_runner_type: "httpRemote"
  druid_indexer_storage_type: "metadata"
It took some trial and error to get them correct.
🙏 2
y
This is what I got
Copy code
druid.discovery.k8s.clusterIdentifier=druid-prod
druid.zk.service.enabled=false
druid.discovery.type=k8s
druid.serverview.type=http
druid.coordinator.loadqueuepeon.type=http
druid.indexer.runner.type=httpRemote
I'll add the missing ones now
I see no obvious error in the coordinators
Copy code
2023-01-30T22:35:42,109 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_xxxxxx.sum_16a5545416f873a_gbgdoeib
2023-01-30T22:35:42,109 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Adding pending task[index_kafka_xxxxxx.sum_16a5545416f873a_gbgdoeib].
...
2023-01-30T22:35:47,399 WARN [KafkaSupervisor-xxxxx-Worker-1] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - Ignoring task [index_kafka_xxxxxx_16a5545416f873a_gbgdoeib], as probably it is not started running yet
...
2023-01-30T22:36:40,901 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.hrtr.HttpRemoteTaskRunner - Assigned a task[index_kafka_xxxxxsum_16a5545416f873a_gbgdoeib] that is known already. Ignored.
This is all I see. No errors at all
s
Does the task appear in the Middle Manager logs?
y
Checking...
Hmmm, I may have an issue there damn it!
My Middle manager didn't start!
Damn it
Sorry, I'll solve that one 1st
s
That would do it 😉
y
Yeah... I feel newbi all over again
s
no worries at all, I feel like a newbie everyday 😄
y
I guess all I needed, is talk to someone! thank you! All is good now 🙂
s
Awesome. Please let us know how it goes.
☝️ 1
y
I see what you mean by http thread leak 🙂
Could this be related:
Copy code
"The worker that this task was assigned disappeared and did not report cleanup within timeout[PT15M]...."
c
Potentially. If the worker running the task had the pod evicted and didn't have the chance to announce shutdown that would be a possible result
y
No pod eviction. Tasks run inside the middler manager. No?
c
Are you running with middlemanagers or indexers? We use indexers, the tasks run there
y
With middleManager. I'll switch to indexer
c
indexers work much better under k8s compared to middlemanagers.
Instead of forking JVMs it uses a threadpool, so it is much more efficient and can be scaled much more easily.
y
Damn! new feature I guess. DruidOperator doesn't seem to support that!
c
Oh I didn't know that, we don't use the operator.
y
Maybe I should stop using it, and just use the helm chart!
I was wrong, looking at the Operator code, it is using the Indexer. I moved to use that too. Thanks 🙂
c
Glad to help!
Open in Slack
PreviousNext
Powered by