https://linen.dev logo
Join Slack
Powered by
This message was deleted.
# troubleshooting
s
This message was deleted.
j
d
@Jason Wang Have you check if ZK was unstable around that time? Once we stabilized our ZK, we no longer see this problem.
j
@Didip Kerabat yea, ZK has been healthy all time. I don't see any error/warn logs in ZK either.
d
Did you happen to have a lot of rows inside druid_tasks?
j
yes, we currently have ~20k rows in that table.
d
ok, did you enable auto pruning settings in overlord?
We usually cleanup tasks older than 24 hours.
j
We currently don't. I can try. But does that help with this issue?
d
Based on our trial and errors, this problem occurs when there are 2 situations: 1. ZK is unstable, goes up and down. 2. Overlord is unstable, goes up and down, confusing the membership logic. And Overlord typically becomes unstable when there are so many tasks it tries to load in heap. Our rule of thumb is to keep number of druid_tasks to less than 5k.
Doing no. 2 is worth it since there are a number of other bugs due to too many tasks in Overlord heap, so might as well.
j
Interesting.... Did you find changes in heap memory usage on overlord in those cases?
g
i checked internally (@ imply) and also i see a recent customer support case that looks similar (an exception during leader election while reacquiring locks). let me see if that is making progress as to figuring out a root cause
👍 1
d
@Jason Wang definitely. Heap size and G1 GC collection count are the two metrics that strongly correlated with this bug.
j
@Didip Kerabat I don't see those jvm metrics spike when we see those errors.... But thanks for the suggestion, we do see the row count in task table decrease a lot. task_lock table still has a large row count (20k). Do you do anything with that table?
@Gian Merlino any updates on the case you were looking at? 🙂
d
task_lock should have been cleaned when ingestion is done. I recommend deleting all of the old records, just keep the ones for the currently running ingestion.
g
i just checked again — there are some believable theories as to how this may happen, but unfortunately we haven't been able to repro the problem in the lab, and we don't have enough logs in order to confirm the theories in this specific case
our current thinking is twofold 1. PR 12901 was done to fix an unrelated bug, but it also modifies task shutdown behavior so the task is marked finished in the metadata store prior to the lock being released (it used to be the other way round). under one of the theories, this would fix the problem. however, like i mentioned, we haven't been able to confirm this 2. we think it's a good idea to modify the Overlord startup sequence such that, when there's a lock conflict like this, the relevant tasks are canceled and the Overlord continues (instead of refusing to start up). this will prevent it from borking a cluster
however, our preference would be to really get to the bottom of it. so if you have Overlord logs from the one that was leading prior to this happening (the new leader won't be of much help, sadly!) then posting them here will help us debug it: https://github.com/apache/druid/issues/11653
j
@Gian Merlino Thanks for the details! I will keep an eye on it. We currently don't turn on info logs for overlord, do you need info logs or warn logs should be good enough?
g
for this, we would need INFO, since we're trying to observe a sequence of events
j
Got it! Sounds good!
Open in Slack
PreviousNext
Powered by