This message was deleted.
# troubleshootings
Slackbot
02/10/2023, 1:12 PMThis message was deleted.
j
Jason Witkowski
02/10/2023, 2:53 PMLiterally was dealing with this all day yesterday. I was getting this inside the K8s jobs that were launching as a replacement for middleManager. Is that where you were seeing it too?
CC @churro
Jason Witkowski
02/10/2023, 2:55 PMHere is what I was told by the
MM-less Copy code
oh yeah ignore those
so it thinks its an overlord but its not, service discovery doesn't need to happen in the peon task but since its another extension I havent been able to get to it yet
...
so service discovery only needs to happen for long lived processes, the peon task…because it inherits from the overlord has the service discovery as well but its uselessJason Witkowski
02/10/2023, 2:56 PMMy issue is that tasks, even
{"type":"noop"}l
Lasse Mammen
02/10/2023, 4:59 PMI have exactly that problem. It looks like the task is even putting data on my s3 deepstore, but never reporting successful in Druid
I did find out that the role needed "pods/log" as well to display logs in the UI, if anybody else has come across that one.
j
Jason Witkowski
02/10/2023, 5:03 PM@Lasse Mammen yes that is identical to my setup/issue. I get logs in the S3 bucket as well and thats where I see the above errors which are supposedly benign. I had the setup working previously but when I started testing some changes to the helm chart I ended up in that state. I went all the way to giving full cluster admin permissions to the role and it didn't help
c
churro
02/10/2023, 5:04 PMwe are using mm-less druid in production. I did push up a few fixes for this, one has been merged the rest are not really being looked at by anyone else. I can honestly just push up one large changeset that fixes a lot of issues next week and you guys can fork it if you want to use the feature.
churro
02/10/2023, 5:04 PMI have a few more locally, like upgrading clients, working better with druid montiors etc…
churro
02/10/2023, 5:05 PMi am having a hard time getting anyone in the druid community to look at them and i don’t want to keep rebasing, so ill just push up a big patch and hopefully someone in the community will take interest
l
Lasse Mammen
02/10/2023, 5:05 PMThanks @churro, I actually tried a custom built with this one https://github.com/apache/druid/pull/13668 which didn't help sadly
c
churro
02/10/2023, 5:06 PMso what is weird, is that the peon task is not reporting back to the overlord correctly right? Let me look into that more today, I haven’t had much free time to work on druid this week.
l
Lasse Mammen
02/10/2023, 5:06 PMHappy to try any other patches on my cloud
y
Younes Naguib
02/10/2023, 5:06 PMI wanted to go with a custom build too. At this point, not sure if going back to ZK isn't the best approach
c
churro
02/10/2023, 5:06 PMFor our deployments this all seems to work….
churro
02/10/2023, 5:07 PMwe run zk-less + mm-less in production here, but our build is pretty forked, ill push up our changes next week
l
Lasse Mammen
02/10/2023, 5:07 PMWhat k8s version are you running it on? I'm on 1.25
c
churro
02/10/2023, 5:07 PMin one big PR so at least people can use it if they want
churro
02/10/2023, 5:07 PMlet me see
j
Jason Witkowski
02/10/2023, 5:07 PMI'm 1.25 too
c
churro
02/10/2023, 5:07 PMwe have all different versions of k8s, we run a lot of druid clusters
churro
02/10/2023, 5:08 PM1.23 is the highest version we are running on
churro
02/10/2023, 5:08 PMsomewhere between 1.16 and 1.23 is what we are running druid on k8s
l
Lasse Mammen
02/10/2023, 5:08 PMI was initially suspecting role permissions because I discovered it needed "pods/log" to display logs in Druid, but I'm not seeing any exceptions thrown so I guess that's not it
y
Younes Naguib
02/10/2023, 5:09 PMThis one seems to be merged: https://github.com/apache/druid/pull/13668
Younes Naguib
02/10/2023, 5:09 PMAs of 2 weeks ago
c
churro
02/10/2023, 5:09 PMyeah thats it, there are a few more
churro
02/10/2023, 5:09 PMi upgraded the client too, haven’t pushed that patch fully, just got it to work on our clusters yesterday….the k8s client was pretty old
churro
02/10/2023, 5:11 PMbut for the task success part, when the task completes, it calls a UpdateStatusAction which makes a call back to the overlord to tell it, hey I am done and i finished successfully. Then the overlord records in the db and should have the correct state. I think something might be going on there.
l
Lasse Mammen
02/10/2023, 5:11 PMif that fails would it not throw a clear exception tho? My peon logs are clean
c
churro
02/10/2023, 5:12 PMyeah they are clean but the call is not being made for some reason successfully
churro
02/10/2023, 5:12 PMso that should call back to the overlord
churro
02/10/2023, 5:13 PMwhich then calls this
churro
02/10/2023, 5:14 PMSo the TaskRunnerUtils should get called, to update the status….i mean it sucks, but you could put a debug log there to show its called??? with the status
churro
02/10/2023, 5:17 PMill push up what i have on my local 25.x branch in one big PR for you guys, today / early next week. But we are running it here (in prod) and everything is working fine. Ours is pretty forked though. So ill get you guys exactly what we have here. I just don’t want to keep making individual PRs and have them rot, so I haven’t been pushing my changes upstream.
l
Lasse Mammen
02/10/2023, 5:18 PMI could try that sure. Will take a bit to verify that
y
Younes Naguib
02/10/2023, 5:18 PMThank you... I will try it too
l
Lasse Mammen
02/10/2023, 5:18 PMI'm just wondering what would happen if it's not correctly configured as an encapsulated task? Could that be it? Then first block you referenced wouldn't execute
y
Younes Naguib
02/10/2023, 5:18 PMI'm at a Beta stage now with my prod, so I can try things.
c
churro
02/10/2023, 5:29 PMokay i think i might see the issue
churro
02/10/2023, 5:38 PMokay so like i said above
d
David McHealy
02/10/2023, 8:37 PMI would love to see that pr. I was thinking of making a few changes myself.
s
Shubham Joshi
04/12/2023, 9:12 AMHey @churro can you share the pr with this fix ?