https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
Untitled
j
Looks like someone else had this issue about 10 days ago in here but didn't get a response: https://apachedruidworkspace.slack.com/archives/C0309C9L90D/p1674483696642769 CC @SClark
So I was able to dig in a bit. One thing I noticed is the above error is about overlord so I went and took a look at the overlord pod which was failing its liveness check. The logs showed basically nothing except what config was being loaded. I had to exec into the pod and 
cat /opt/druid/log/*
to find actual useful logs. These should probably be getting output to stdout or stderr for the purposes of running in a container
I saw that overlord could not talk to postgres which was an error on my part. I got that fixed and now it talks to postgres without issue. The new issue is about listing the pods.
Copy code
2023-02-02T17:57:15,421 ERROR [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Failed to manage: {class=org.apache.druid.indexing.overlord.TaskQueue, exceptionType=class io.fabric8.kubernetes.client.KubernetesClientException, exceptionMessage=Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed., restartDelay=30000}
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed.
The pod is actually in a healthy state now though passing both liveness and readiness checks. So maybe this is just an error because there are no jobs currently? It probably should be getting handled better though
'Fixing' this however has had no effect on the errors initially described for 
coordinator
and 
historical
pods. Those still exist
c
hmmm that looks like a role / rolebinding issue
j
omg really?
i hadn't set that up yet ;\
let me do that real fast
c
do that and see if it gets picked up
also one thing i noticed, if you are migrating a cluster from mm to mm-less with real-time indexing, it is wise to pause your supervisors then roll the upgrade, then unpause
j
this is a brand new cluster
I added the serviceAccount annotation (using iam role linked service accounts) and restarted the historical pod
I can see the pod now gets annotations for AWS credentials but it still gives the same error
Copy code
- name: AWS_STS_REGIONAL_ENDPOINTS
          value: regional
        - name: AWS_DEFAULT_REGION
          value: us-west-2
        - name: AWS_REGION
          value: us-west-2
        - name: AWS_ROLE_ARN
          value: 'arn:aws:iam::REDACTED:role/druid-s3'
        - name: AWS_WEB_IDENTITY_TOKEN_FILE
          value: /var/run/secrets/eks.amazonaws.com/serviceaccount/token
c
let me find the docs
are you using a http proxy?
j
I'll create a sample pod using this serviceaccount/namespace just to be sure I can access the configured s3 bucket
unless there are AWS permissions outside of the s3 setup that I need ?
I am not using any proxy
c
cool
so here is a sample role/rolebinding
at the bottom
probably good to add “jobs” to the resource as well
I should update that
j
OK, i was able to test against the namespace/serviceAccount druid is using and successfully write to the configured druid s3 bucket using a test pod
Yes, my role already has the jobs permission
Copy code
$ kubectl -n druid1 get role -o yaml
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: Role
  metadata:
    name: druid-cluster
    namespace: druid1
    resourceVersion: "75299544"
    uid: 1cc3f0b5-4456-4880-a739-9a9d4b97f83e
  rules:
  - apiGroups:
    - ""
    resources:
    - pods
    - configmaps
    - jobs
    verbs:
    - '*'
kind: List
metadata:
  resourceVersion: ""
Copy code
$ kubectl -n druid1 get rolebinding -o yaml
apiVersion: v1
items:
- apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
  kind: RoleBinding
  metadata:
    name: druid-cluster
    namespace: druid1
  roleRef:
    apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
    kind: Role
    name: druid-cluster
  subjects:
  - kind: ServiceAccount
    name: default
kind: List
metadata:
  resourceVersion: ""
c
which version of k8s are you running with?
j
1.24.6
c
cool, is your task queue clear?
can you clear it all
j
This is a brand new druid deployment. never been used or had a task run
c
interesting
hmm can you try and launch a task
just something simple like a noop
{ "type" : "noop" }
j
Sure... but to be completely transparent, I've never used druid before. I'm attempting to deploy this for a developer group 🙂 So i'll have to figure out how to do that
c
sure no worries
okay go to the druid console
ill walk you through it
ingestion tab
there are two sections here, 
supervisors
and 
tasks
under the 
tasks
section
you see this
click submit json task and paste what i gave you
then lets track that task through the overlord logs
j
submitting now
At the top I get 
Task submitted successfully
, but then an error pops up
Copy code
Unknown exception / Failed to talk to leader node at [/druid/indexer/v1/tasks]. Error code [500], description [Server Error]. / <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>
c
are all your druid processes up?
and healthy?
j
no, both 
coordinator
and 
historical
pods cannot start due to the error in the snippet at the top of this thread
same error for both
c
ohhhh
okay
i see
okay for the extensions_loadList
the kubernetes-overlord extension is 
only
loaded for the overlord
the rest of the services can’t have that
so you have your common load list
[a,b,c]
j
ugh. i freaking knew it. The docs suggest that but it wasn't clear how to do that for a specific service
c
then in your overlord, you’ll have [a,b,c,druid-kubernetes-overlord-extensions]
j
got it. fixing it now
c
no worries, if its not clear, i can add that to the docs
good to have other people try this out
j
you could not have a better 'dummy' than me. I know literally nothing about this stuff haha 😅
I'm applying that fix now
c
lol, no worries, we will get you up to speed
cool, once you apply that fix, you can test again with the “noop” task json i gave you
that is a good canary to check if everything works
j
do you want me to attempt and start coordinator pods first ?
updating the configmap won't do that automatically
c
yeah we need druid healthy first
j
well... they aren't dying right away. that is a big improvement
👍 1
c
definitely lets work on getting druid in a good healthy state, then we can worry about the rest
j
OK, so coordinator has some errors in its log file. Some things to note: • these logs from 
/opt/druid/log/*
do not make it to stderr/stdout
So I have to exec into the pod to look at them
c
yeah no worries
take your time, once druid is up…lets continue
j
• The log file seems to be a variable that is not being interpolated: 
bash-5.1$ cat /opt/druid/log/\$\{sys\:druid.node.type\}.log
c
how did you start up the coordinator?
j
I made the change you noted and then deleted the pod once the configmap was updated
c
do you use the druid operator to do your deployments?
j
no, i am using the helm chart in the main repo
c
that does make things a lot easier in k8s
j
but kind of wishing i went and used the operator
c
🙂 no big deal…. i just use the operator and am not really that familiar with the helm charts
j
I didn't want to have to write a helm chart to deploy the 
Druid
custom resource, so i went with this
so the historical pod is still not starting
but the error is now hidden. its not producing the above message anymore
c
yeah that error should go away once the extension is gone
j
I get logging up to the config-load and the warning about reflexive access
c
nothing in the logs that show why things are not working?
j
and then the pod dies with no other information
c
so you shouldn’t need helm to deploy the crd right? Its pretty much a static file you can just do a 
kubectl apply -f crd.yaml
j
yea, but infrastructure as code, gitops, etc. etc.
c
sure no worries
j
also deployments are going through argoCD
c
i don’t know if we need the historicals up to do this work though
there should be no handoff
so now if you go to the ingestion page
does it error out?
j
yes,
Copy code
Unknown exception / Failed to talk to leader node at [/druid/indexer/v1/tasks]. Error code [500], description [Server Error]. / <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>
just to confirm, at this point all pods are up and running except historical
c
and you are using zk?
or zk-less deployment?
j
zk-less
c
and that seems to be working
i take it, you have a leader for your overlord?
j
I only have a single overlord pod... but it would appear there are some issues in the logs
c
so the zk-less one needs some additional role / rb stuff
like labels, etc..
j
An SSL issue:
Copy code
023-02-02T19:15:28,823 WARN [qtp1777972577-36] org.eclipse.jetty.server.HttpChannel - handleException /druid/indexer/v1/tasks javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
and a listing pods issue
Copy code
2023-02-02T19:15:44,552 ERROR [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Failed to manage: {class=org.apache.druid.indexing.overlord.TaskQueue, exceptionType=class io.fabric8.kubernetes.client.KubernetesClientException, exceptionMessage=Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed., restartDelay=30000}
io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Pod]  with name: [null]  in namespace: [druid1]  failed.
c
im wondering why it would ask for a null “name”
j
yea, me too. i thought at first it was an empty 
POD_NAME
ENV VAR, but i can see that it is set appropriately inside the container
c
its basically trying to see if there are any tasks running
yeah cool
so can you show me your overlord config
lets get to that now
okay looks good
i have a few changes
Copy code
overlord:
        config:
          DRUID_XMS: 8192m
          DRUID_XMX: 8192m
          druid_extensions_loadList: '["postgresql-metadata-storage", "druid-kafka-indexing-service", "druid-datasketches", "druid-kubernetes-extensions", "druid-kubernetes-overlord-extensions", "druid-s3-extensions"]'
          druid_indexer_fork_property_druid_processing_buffer_sizeBytes: 100MiB
          druid_indexer_fork_property_druid_processing_numMergeBuffers: 2
          druid_indexer_fork_property_druid_processing_numThreads: 1
          druid_indexer_queue_maxSize: 30
          druid_indexer_runner_javaOptsArray: '["-server", "-Xms8g", "-Xmx8g", "-XX:MaxDirectMemorySize=4g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'
          druid_indexer_runner_namespace: druid1
          druid_indexer_runner_type: k8s
          druid_indexer_task_enableTaskLevelLogPush: true
          druid_indexer_storage_type: metadata
          druid_indexer_task_baseTaskDir: <specify_task_dir>
          druid_processing_intermediaryData_storage_type: deepstore
          druid_server_http_numThreads: 60
          druid_worker_capacity: 4
          javaOpts: -Xms8G -Xmx8G
can you specify where i put 
<specify_task_dir>
i removed encapsulated_Task
but let me check the druid code upstream
nope you were right
let me fix
okay here
Copy code
overlord:
        config:
          DRUID_XMS: 8192m
          DRUID_XMX: 8192m
          druid_extensions_loadList: '["postgresql-metadata-storage", "druid-kafka-indexing-service", "druid-datasketches", "druid-kubernetes-extensions", "druid-kubernetes-overlord-extensions", "druid-s3-extensions"]'
          druid_indexer_fork_property_druid_processing_buffer_sizeBytes: 100MiB
          druid_indexer_fork_property_druid_processing_numMergeBuffers: 2
          druid_indexer_fork_property_druid_processing_numThreads: 1
          druid_indexer_queue_maxSize: 30
          druid_indexer_runner_javaOptsArray: '["-server", "-Xms8g", "-Xmx8g", "-XX:MaxDirectMemorySize=4g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager"]'
          druid_indexer_runner_namespace: druid1
          druid_indexer_runner_type: k8s
           druid_indexer_task_encapsulatedTask: true
          druid_indexer_storage_type: metadata
          druid_indexer_task_baseTaskDir: <specify_task_dir>
          druid_processing_intermediaryData_storage_type: deepstore
          druid_server_http_numThreads: 60
          druid_worker_capacity: 4
          javaOpts: -Xms8G -Xmx8G
j
so it looks like just the baseTaskDir is being added?
c
yeah and get rid of the quotes around the encapsulated task true
j
pretty sure that will throw a type error , but will try it
setting 
druid_indexer_task_baseTaskDir: 'var/druid/task'
3 Views
Open in Slack
PreviousNext
Powered by