https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
s
I'm running druid on version 25.0.0
j
@SClark having the same issues on a fresh installation
Any chance you figured it out ?
l
Can someone provide a working version of k8s deploy for druid-operator (like the tiny-cluster example) with both 
druid-kubernetes-extensions
and 
druid-kubernetes-overlord-extensions
properly configured? (maybe @churro) Tbh looking at the docs I don’t even understand if MM-less druid requires 
druid-kubernetes-extensions
(like the extension names kinda suggest) loaded/running/no-zookeeper or if they’re incompatible. Their documentation is far from simple and there are no examples for either. As an example of confusion, Kubernetes extension doc says you need
Copy code
druid.indexer.runner.type=httpRemote
And MM-less Druid doc says you need
Copy code
druid.indexer.runner.type: k8s
So… what it is? And can they work together? (I’m seeing the same exceptions as the people on this thread btw)
👀 1
@Adheip Singh looks like could help on this one too.
c
for the k8s extension, put 
druid.indexer.runner.type=httpRemote
in your common runtime properties
in your overlord put 
druid.indexer.runner.type: k8s
they can work together
l
Hey @churro, thanks for your reply. so all properties from the MM-less docs are for overlord only?
Copy code
druid.indexer.runner.type=k8s
    druid.indexer.runner.namespace=druid
    druid.indexer.queue.maxSize=3
    druid.indexer.task.encapsulatedTask=true
    druid.processing.intermediaryData.storage.type=deepstore
something like this? anything else?
c
yeah so the overlord k8s extension….should only have configs in the overlord. The maxSize means you want a max of 3 concurrent tasks at a time? That looks a bit small to be, otherwise looks good
l
yeah, I just want to see like a ‘tiny-cluster’ equivalent
c
sure no worries, i would bump that up to something a bit larger like 10, but the rest looks good to me
are you running with sidecars or istio?
l
im not
Copy code
overlords:
      # Optionally specify for running coordinator as Deployment
      # kind: Deployment
      nodeType: overlord
      druid.port: 8088
      nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/overlord"
      replicas: 1
      runtime.properties: |
        druid.service=druid/overlord
        druid.indexer.runner.type=k8s
        druid.indexer.runner.namespace=druid
        druid.indexer.queue.maxSize=3
        druid.indexer.queue.startDelay=PT30S
        druid.indexer.task.encapsulatedTask=true
        druid.processing.intermediaryData.storage.type=deepstore
      extra.jvm.options: |-
        -Xmx512M
        -Xms512M
Copy code
│ Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:                                                                                                                                                    ││                                                                                                                                                                                                                                                         ││ 1) Unknown provider [k8s] of Key[type=org.apache.druid.discovery.DruidNodeAnnouncer, annotation=[none]], known options [[curator]]                                                                                                                      ││   at org.apache.druid.guice.PolyBind.createChoiceWithDefault(PolyBind.java:110) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.curator.discovery.DiscoveryModule)     ││   while locating org.apache.druid.discovery.DruidNodeAnnouncer                                                                                                                                                                                          ││     for field at org.apache.druid.cli.ServerRunnable$DiscoverySideEffectsProvider.announcer(ServerRunnable.java:105)                                                                                                                                    ││   at org.apache.druid.cli.ServerRunnable.bindAnnouncer(ServerRunnable.java:93) (via modules: com.google.inject.util.Modules$OverrideModule -> com.google.inject.util.Modules$OverrideModule -> org.apache.druid.cli.CliCoordinator$1)                   ││
c
so your load list is not overriden…..
you should have something like loadList=[a,b,c] for common
then in your overlord loadList=[a,b,c,druid-kubernetes-overlord-extensions]
l
Yeah you’re right, I was reactivating the deployment with both extensions, but one was missing in the load list this time 🤦
ok, now I’m back at the old
Copy code
│ Exception in thread "main" java.lang.RuntimeException: com.google.inject.CreationException: Unable to create injector, see the following errors:                                                                                                        ││                                                                                                                                                                                                                                                         ││ 1) Could not find a suitable constructor in org.apache.druid.indexing.common.config.TaskConfig. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.                      
││   at org.apache.druid.indexing.common.config.TaskConfig.class(TaskConfig.java:49)                                                                                                                                                                       ││   while locating org.apache.druid.indexing.common.config.TaskConfig                                                                                                                                                                                     ││     for the 3rd parameter of org.apache.druid.k8s.overlord.KubernetesTaskRunnerFactory.<init>(KubernetesTaskRunnerFactory.java:63)                                                                                                                      ││   at org.apache.druid.k8s.overlord.K8sOverlordModule.configure(K8sOverlordModule.java:63) (via modules: com.google.inject.util.Modules$OverrideModule -> org.apache.druid.k8s.overlord.K8sOverlordModule)                                               
││                                                                                                                                                                                                                                                         ││ 1 error                                                                                                                                                                                                                                                 ││     at org.apache.druid.cli.GuiceRunnable.makeInjector(GuiceRunnable.java:88)                                                                                                                                                                           ││     at org.apache.druid.cli.ServerRunnable.run(ServerRunnable.java:62)                                                                                                                                                                                  ││     at org.apache.druid.cli.Main.main(Main.java:112)
c
what about
Copy code
javaOptsArray
?
you have to specify that for your tasks
l
hmm docs says it’s not required, the default would be a 
-Xmx1g
c
i messed up and didn’t specify that it was required, ill fix that
im pretty sure it has to be specified. Looking at the code
l
so something as simple as this should do the trick?
Copy code
druid.indexer.runner.javaOptsArray=["-Xmx1g"]
c
yeah im sorry about that
i can actually fix the code to make it work as desired
l
Hmmm all services are crashing, so looks like 
druid-kubernetes-overlord-extensions
should exist only in the load list for overlord, is that right?
👍 1
c
yes we include it 
only
in the overlord that extension
l
great, maybe small changes in the docs will make it easier to everyone!
c
yeah ill make a PR for the docs, its a bit weird since they are 2 separate extensions, the k8s discovery and overlord, but ill put a blurb in there about if you use the discovery, override these things in the overlord, is that what you were wanting?
i also fixed the javaOpts in the code in a PR that is open, when that gets merged, the docs will be truthful
l
I’d say • maybe add all required fields to the properties table, so it’s very clear in a single place what’s expected and what’s configurable • make it clear all configurations on for this extension are set only in Overlord section
c
👍
thanks for the feedback
l
we should thank you, great work, I believe this thing will be default behaviour in near future!
I just want to see this running, but I still didnt get it right…
c
what is the problem you are experiencing now?
l
tbh dunno, neither coordinator or overlord show me any log besides this (other services load just fine)
Copy code
│ 2023-03-13T23:41:53+00:00 startup service overlord                                                                                                                                                                                                                                                                                                             │
│ Setting druid.host=10.1.1.102 in /tmp/conf/druid/cluster/master/coordinator-overlord/runtime.properties                                                                                                                                                                                                                                                        │
│ WARNING: An illegal reflective access operation has occurred                                                                                                                                                                                                                                                                                                   │
│ WARNING: Illegal reflective access by com.google.inject.internal.cglib.core.$ReflectUtils$1 (file:/opt/druid/lib/guice-4.1.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)                                                                                                                  │
│ WARNING: Please consider reporting this to the maintainers of com.google.inject.internal.cglib.core.$ReflectUtils$1                                                                                                                                                                                                                                            │
│ WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations                                                                                                                                                                                                                                                          │
│ WARNING: All illegal access operations will be denied in a future release
But I can see something like this in other service logs and webconsole
Copy code
│ 2023-03-13T23:40:06,294 INFO [org.apache.druid.k8s.discovery.K8sDruidNodeDiscoveryProvider$NodeRoleWatchercoordinator] org.apache.druid.discovery.BaseNodeRoleWatcher - Node [<http://10.1.1.99:8081>] of role [coordinator] detected.                                                                                                                           │
So I go to load data, submit a ingestion for wikipedia test data and… nothing happens, I cant even see a log as I get 404 status
heres my deployment
druid.yaml
c
what do your overlord logs say when you grep for 
wikipedia
the overlord will log the task and what is it doing at each stage of the task
a
@churro @Luiz Augusto if all your efforts can be documented here https://github.com/datainfrahq/druid-operator/issues/14. It would be great help. Also we can add this in the operator e2e tests. Thanks you 🙂
l
For yesterday’s problem, looks like I can’t change the location here for either coordinator or overlord.
Copy code
nodeConfigMountPath: "/opt/druid/conf/druid/cluster/master/coordinator-overlord"
As soon as I set them back to this common value, services could start again, they coordinate using k8s API and not Zookeeper, which is great.
But as soon as I submit a the wikipedia ingest test, I got this
Copy code
│ 2023-03-14T12:02:09,548 ERROR [k8s-task-runner-0] org.apache.druid.indexing.overlord.TaskQueue - Failed to run task: {class=org.apache.druid.indexing.overlord.TaskQueue, exceptionType=class io.fabric8.kubernetes.client.KubernetesClientException, exceptionMessage=Failure executing: DELETE at: <https://10.96.0.1/apis/batch/v1/namespaces/druid/jobs/indexparallelwikipediafogdhebc20230314t120207965z>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch "indexparallelwikipediafogdhebc20230314t120207965z" is forbidden: 
││  User "system:serviceaccount:druid:default" cannot delete resource "jobs" in API group "batch" in the namespace "druid"., task=index_parallel_wikipedia_fogdhebc_2023-03-14T12:02:07.965Z, type=index_parallel, dataSource=wikipedia}                                                              ││ io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: DELETE at: <https://10.96.0.1/apis/batch/v1/namespaces/druid/jobs/indexparallelwikipediafogdhebc20230314t120207965z>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. jobs.batch "indexparallelwikipediafogdhebc20230314t120207965z" is forbidden: User "system:serviceaccount:druid:default" cannot delete resource "jobs" in API group "batch" in the namespace "druid".                                                                                        ││     at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:682) ~[?:?]                                                                                                                                                                                     ││     at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:661) ~[?:?]
I have the suggested role + role binding, but maybe something is wrong here. I can’t say I understand these k8s roles and bindings very well.
Copy code
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
      - jobs
    verbs:
      - '*'
---
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
j
Your rbac is wrong
l
Interesting. Should this be the correct one?
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
      - jobs             <--- required for the MM-less extension
    verbs:
      - '*'
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - '*'
j
Copy code
- apiGroups:
      - ""
This means the 'common' or 'base' api (there is a specific term for it, i forget). Jobs do not belong under this apiGroup
Jobs are part of 
batch
I linked you to the correct rbac
it may also need 
pods/logs
under the first 
apiGroups: [""]
for viewing logs. I havent tested that yet, let me know if you do
l
So I’ve changed to
Copy code
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
  name: druid-cluster
rules:
  - apiGroups:
      - ""
    resources:
      - pods
      - configmaps
    verbs:
      - '*'
  - apiGroups:
      - batch
    resources:
      - jobs
    verbs:
      - '*'
but now all pods crash during startup
Copy code
│ 2023-03-14T16:10:34,772 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [ANNOUNCEMENTS]                                                               ││ 2023-03-14T16:10:34,777 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Announcing DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/router', host='10.1.1.251', bindOnHost=false, port=-1, plaintextPort=8088, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='ROUTER', services={}}]                                                             ││ 2023-03-14T16:10:34,914 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Json Patch For Node Announcement: [[{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-router","value":"true"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-id-hash","value":"1658736627"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-cluster-identifier","value":"druid-cluster"},{"op":"add","path":"/metadata/annotations/druidNodeInfo-router","value":"{\"druidNode\":{\"service\":\"druid/router\",\"host\":\"10.1.1.251\",\"bindOnHost\":false,\"plaintextPort\":8088,\"port\":-1,\"tlsPort\":-1,\"enablePlaintextPort\":true,\"enableTlsPort\":false},\"nodeType\":\"router\",\"services\":{}}"}]]                                                                                   ││ TRACE StatusLogger Log4jLoggerFactory.getContext() found anchor class org.apache.druid.java.util.common.logger.Logger                                                                                       ││ 2023-03-14T16:10:35,007 WARN [main] org.apache.druid.java.util.common.RetryUtils - Retrying (1 of 2) in 760ms.                                                                                              ││ <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to patch pod[druid/druid-druid-cluster-routers-0], code[403], error[{                                                                                          ││   "kind": "Status",
││   "apiVersion": "v1",
││   "metadata": {},
││   "status": "Failure",
││   "message": "pods \"druid-druid-cluster-routers-0\" is forbidden: User \"system:serviceaccount:druid:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"druid\"", "reason": "Forbidden",                                                                                                                                                                                    │
│   "details": {
│     "name": "druid-druid-cluster-routers-0",
│     "kind": "pods"
│   },
│   "code": 403                                                                                                                                                                                               
│ }].
│
│     at org.apache.druid.k8s.discovery.DefaultK8sApiClient.patchPod(DefaultK8sApiClient.java:71) ~[?:?]
│     at org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer.lambda$announce$0(K8sDruidNodeAnnouncer.java:112) ~[?:?]
(...)
j
can you paste your roleBinding ?
l
Copy code
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
j
you are deploying into namespace 
druid
?
l
yes
j
and the pods are configured to use serviceAccount 
default
?
change:
Copy code
subjects:
  - kind: ServiceAccount
    name: default
to
Copy code
subjects:
  - kind: ServiceAccount
    name: default
    namespace: druid
in your roleBinding. Not sure that will fix it, but that is the only diff i see between yours and mine
also confirm your podSpec is using 
serviceAccount: default
l
and the pods are configured to use serviceAccount 
default
?
I was assuming this to be true. But I’ve added 
serviceAccount: default
and now I can see it in the pod, but I got the same errors
Copy code
│ Caused by: <http://org.apache.druid.java.util.common.RE|org.apache.druid.java.util.common.RE>: Failed to patch pod[druid/druid-druid-cluster-overlords-0], code[403], error[{                                                                             ││   "kind": "Status",                                                                                                                                                                                         ││   "apiVersion": "v1",                                                                                                                                                                                       ││   "metadata": {},                                                                                                                                                                                           ││   "status": "Failure",                                                                                                                                                                                      ││   "message": "pods \"druid-druid-cluster-overlords-0\" is forbidden: User \"system:serviceaccount:druid:default\" cannot patch resource \"pods\" in API group \"\" in the namespace \"druid\"",             
││   "reason": "Forbidden",                                                                                                                                                                                    ││   "details": {                                                                                                                                                                                              
│     "name": "druid-druid-cluster-overlords-0",                                                                                                                                                              
│     "kind": "pods"                                                                                                                                                                                          
│   },                                                                                                                                                                                                        
│   "code": 403                                                                                                                                                                                               
│ }].
j
yea if its not specified then its 
default
, did you edit the roleBinding with the namespace?
l
I did. It’s like this now|
Copy code
kind: RoleBinding
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
metadata:
  name: druid-cluster
subjects:
  - kind: ServiceAccount
    name: default
    namespace: druid
roleRef:
  kind: Role
  name: druid-cluster
  apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
j
everything seems like it should be working
can you try deleting the pods that are failing just to make sure they are picking up the updated rbac
l
I’m running kubectl delete and then apply every time, so everything (role, role binding and pods) is destroyed and recreated
druid-cluster.yaml
j
ok, is there anything in the app log?
i see from your config it should be in 
logs/app.log
l
i see the same thing over and over
j
just to confirm, the role and roleBinding are both in the 
druid
namespace ?
l
they used to be… 👀 let me put them back
j
ok, they are namespace scoped resources, so they must be in the same namespace
🙌 1
l
yeeeeeeeeeeeeeees
🥳 1
Thanks @Jason Witkowski @churro @Adheip Singh!
💯 1
@churro can we add the role fix to the docs too?
c
Sure, its an OSS project you can just make a PR and add whatever you want to the docs. I’ll approve
18 Views
Open in Slack
PreviousNext
Powered by