This message was deleted.
# generals
Slackbot
07/31/2023, 2:16 PMThis message was deleted.
a
Amatya Avadhanula
07/31/2023, 3:16 PMCould you please check
druid_tasklocks✅ 1
Amatya Avadhanula
08/01/2023, 9:37 AMIf the entries are present in the logs (implying that they are in memory) but not the tasklocks table, it could mean that there was an issue with the Overlord and a master node restart could help
s
Sujith Kumar.S
08/01/2023, 10:47 AMIt was not preset in the metadata database. I did a restart to overload process and it worked.
👍 1
Sujith Kumar.S
08/01/2023, 10:47 AMDoes that mean , those were in memory ?
a
Amatya Avadhanula
08/01/2023, 10:48 AMYes, there may have been an issue with task lock clean up
Amatya Avadhanula
08/01/2023, 10:48 AMWould you mind raising a github issue with your Overlord logs?
s
Sujith Kumar.S
08/01/2023, 10:50 AMSure. I can .
a
Amatya Avadhanula
08/01/2023, 10:58 AMThank you!
s
Sujith Kumar.S
08/01/2023, 10:59 AMSujith Kumar.S
08/01/2023, 11:00 AM@Amatya Avadhanula , let me know if any more details required on this. Will be more than happy to provide ,
a
Amatya Avadhanula
08/01/2023, 11:01 AMThanks again for creating it. It would be great if you could include overlord logs for a longer duration. (Starting when the old tasks were created which did not clean up their locks in memory until the restart)
s
Sujith Kumar.S
08/01/2023, 5:59 PMSure, I'm trying to get the old logs from my log repo. Will upload by tomorrow
Sujith Kumar.S
08/02/2023, 8:13 AMIt seems for the failed MSQ jobs, the slots are not clearing properly and its restricting to spinup new tasks. Is there some sort of configuration we can do for a cleanup .
a
Amatya Avadhanula
08/02/2023, 8:41 AMDo you see the failures without MiddleManager-less ingestion as well?
s
Sujith Kumar.S
08/02/2023, 8:42 AMIts middle manager less configuration
Sujith Kumar.S
08/02/2023, 8:42 AMWhile loading we had some issues in the file, thats why its failed. But the problem was it's not releasing the slots properly I guess, bcz of that we are not able to spinup new tasks
a
Amatya Avadhanula
08/02/2023, 8:46 AMHmm, it seems more likely that it's due to MM-less ingestion rather than it being specific to MSQ engine.
It would be really helpful to get the overlord logs during this period to debug this better
s
Sujith Kumar.S
08/02/2023, 9:46 AMokay.
Sujith Kumar.S
08/02/2023, 9:46 AMI'm trying to get that