This message was deleted.

I opened up the service account role to full permissions and it still doesn't help

Hey @Cory Johannsen, sorry for pinging you again but I consider you the

druid-kubernetes-extension

expert. Can you help?

Thanks @Sergio Ferragut So, I was able to work past the immediate issue above by adding an annotation via the helm chart supplied in the

apache/druid

repo. It seems the error comes from there being no existing annotations on the statefulset pods. I added

druid: true

to the annotations and I was able to move to the next error. The next error seems to be the same in nature though:

2023-02-02T22:56:47,927 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Json Patch For Node Announcement: [[{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-historical","value":"true"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-id-hash","value":"1962150723"},{"op":"add","path":"/metadata/labels/druidDiscoveryAnnouncement-cluster-identifier","value":"druid1"},{"op":"add","path":"/metadata/annotations/druidNodeInfo-historical","value":"{\"druidNode\":{\"service\":\"druid/historical\",\"host\":\"10.233.87.196\",\"bindOnHost\":false,\"plaintextPort\":8083,\"port\":-1,\"tlsPort\":-1,\"enablePlaintextPort\":true,\"enableTlsPort\":false},\"nodeType\":\"historical\",\"services\":{\"dataNodeService\":{\"type\":\"dataNodeService\",\"tier\":\"_default_tier\",\"maxSize\":10737418240,\"type\":\"historical\",\"serverType\":\"historical\",\"priority\":0},\"lookupNodeService\":{\"type\":\"lookupNodeService\",\"lookupTier\":\"__default\"}}}"}]]
2023-02-02T22:56:47,960 INFO [main] org.apache.druid.k8s.discovery.K8sDruidNodeAnnouncer - Announced DiscoveryDruidNode[DiscoveryDruidNode{druidNode=DruidNode{serviceName='druid/historical', host='10.233.87.196', bindOnHost=false, port=-1, plaintextPort=8083, enablePlaintextPort=true, tlsPort=-1, enableTlsPort=false}, nodeRole='HISTORICAL', services={dataNodeService=DataNodeService{tier='_default_tier', maxSize=10737418240, serverType=historical, priority=0}, lookupNodeService=LookupNodeService{lookupTier='__default'}}}]
2023-02-02T22:56:47,960 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Successfully started lifecycle [module]
2023-02-02T22:56:48,064 INFO [qtp138617961-59] org.apache.druid.server.coordination.SegmentLoadDropHandler - Starting...
2023-02-02T22:56:48,066 WARN [qtp138617961-59] org.apache.druid.guice.LifecycleScope - Caught exception when trying to create a[Key[type=org.apache.druid.server.coordination.SegmentLoadDropHandler, annotation=[none]]]
java.lang.reflect.InvocationTargetException: null

Cool. It is probably defaulting to /var/druid/segment-cache

yea the helm chart does not match that for the mount location. Honestly there is a lot to be desired about the helm chart. It also does not allow for specifying serviceAccounts which is a pretty big deal (you have to use the default which means your only level of segregation becomes the namespace and all pods get same permissions)

By all means! We need a better helm chart, so any help from the community in making it better is very welcome.

https://github.com/apache/druid/pull/13746 https://github.com/apache/druid/pull/13747

@Sergio Ferragut @Cory Johannsen Any chance you guys could give a review to the above if you have some time? @Adam Peck you had left a comment on one so I'll ping you too in case you have some time.