This message was deleted.
# generals
Slackbot
09/26/2023, 10:10 PMThis message was deleted.
l
Laksh Singla
09/26/2023, 10:21 PMWhich version are you seeing this issue in?
Laksh Singla
09/26/2023, 10:26 PMThanks for the comprehensive description of the problem! It seems like something similar (speculation) has recently started surfacing in our clusters as well, therefore trying to triage what might have caused this issue.
k
Kai Sun
09/26/2023, 10:27 PMwe saw it in 25.0.0.
Kai Sun
09/26/2023, 10:27 PMhowever, this issue would happen in any recent release at least from 0.23.0 till latest.
Kai Sun
09/26/2023, 10:30 PMBasically, since
GroupbyStategyV2l
Laksh Singla
09/26/2023, 10:31 PMcurious, does the query pass if it is executed in isolation (instead of a flood of the queries)?
Laksh Singla
09/26/2023, 10:32 PMWe are seeing that if the query is run in isolation, it passes sub second, however if multiple queries are fired in parallel, they slow down considerably. Taking >1min to produce result (if the timeout doesnt strike em down)
k
Kai Sun
09/26/2023, 10:34 PMContention for merge buffers can very well be the reason of what you observed I guess. In our case, two level group by queries in isolation would also complete in secs.
a
Abhishek Agarwal
09/27/2023, 3:59 AMbtw I have been looking at your PR as well. it's definitely useful.
👍 1
l
Laksh Singla
09/27/2023, 5:49 AMthanks for the lead, confirmed we are running into the same issue. Btw, left a couple of comments on the PR
Laksh Singla
09/27/2023, 11:08 PMfyi, we identified the issue, and it seems that there’s a deadlock created when multiple queries requiring broker merge buffers are fired.
👍 1
Laksh Singla
09/27/2023, 11:12 PMbest bet till there’s a fix would be to increase the num merge buffers on broker - minimum number should be estimatedParallelQueries + 1, though you can set it slightly higher to be on a safer side.