This message was deleted.
# troubleshootings
Slackbot
02/10/2023, 5:39 PMThis message was deleted.
c
churro
02/10/2023, 5:39 PMHere is how it works currently
churro
02/10/2023, 5:39 PM1. Overlord launches peon, peon does stuff, peon reports back to overlord its state (success / failure ) when it finishes.
churro
02/10/2023, 5:40 PM2. As a backup, the overlord also checks the state of the job, and records that when it finishes. This is to handle the case where the peon may have tried to talk back to the overlord but could not. So we have a the overlord also register the state
churro
02/10/2023, 5:41 PMI was looking at the logs @Jason Witkowski had for his overlord and noticed something interesting
churro
02/10/2023, 5:41 PMfor a given job, it first said it recorded the status as “SUCCESS” <--- from the peon talking back to the overlord
churro
02/10/2023, 5:41 PMThen for that same job, later in the log lines marked it as a failure
churro
02/10/2023, 5:42 PMSo how does the overlord check to see if the job has successfully completed or failed. It monitors the job phase and looks for a string 😕
churro
02/10/2023, 5:43 PMSo if the phase does not match
Copy code
Succeededchurro
02/10/2023, 5:46 PMahhh ha!
churro
02/10/2023, 5:46 PMi see it, the k8s client is not guaranteed to work with 1.25
churro
02/10/2023, 5:46 PMI have a patch for the client upgrade, that will work with everything up to k8s server 1.25.3
churro
02/10/2023, 5:47 PMthis is exactly why we don’t see the issue, the client stops working after 1.23…. I’ll get you guys a PR + patch today
churro
02/10/2023, 5:47 PMI apologize here, now i see it
j
Jason Witkowski
02/10/2023, 5:48 PMit actually worked for 1.24.6 for what its worth
l
Lasse Mammen
02/10/2023, 5:48 PMThanks a million. So I'm guessing this client upgrade won't do it? https://github.com/apache/druid/pull/13759
j
Jason Witkowski
02/10/2023, 5:48 PMupgrading to 1.25.6 is turning out to be a huge regret
c
churro
02/10/2023, 5:48 PMj
Jason Witkowski
02/10/2023, 5:49 PMeverything up to k8s server 1.25.3c
churro
02/10/2023, 5:49 PM@Lasse there is more i have to add to that patch…..there were a bunch of runtime things that broke, i just got it to work yesterday on our clusters. I’ll push up those changes into the PR as well
churro
02/10/2023, 5:49 PMI haven’t had time
churro
02/10/2023, 5:52 PMill try to get you guys something today, or over the weekend. But i think I see exactly what is wrong here.
l
Lasse Mammen
02/10/2023, 5:52 PMThanks for looking into it, it's greatly appreciated and for the original PR. This is going to be an interesting one to maintain, looks like it'll require dep updates often enough
c
churro
02/10/2023, 5:54 PMyeah but I am hoping as more people use it, the community can get involved and we can own it together. The thing with this upgrade in particular was it was a major version bump to the client, which means that a lot of API’s changed. I think with minor version upgrades to the client, it should be simply bumping your pom and building.
👍 3
s
Sujith Kumar.S
05/10/2023, 8:14 AM@churro, any idea when 26.0.0 is getting released. We are facing the similar issue.