@Kevin Cai
Hi Kevin, I'm currently operating a StarRocks cluster and occasionally encounter latency spikes that can last for several minutes.
When I check the query profile during these periods, I see that queries spend a long time waiting in the scheduling queue:
• QueryPeakScheduleTime: 6s408ms
Under normal conditions, these same queries complete in just a few tens of milliseconds.
However, once or twice a day, P75–P99 latency suddenly increases to several seconds or even tens of seconds,
and the cluster does not recover for around 10 minutes.
For context:
• We are not running any Materialized View refresh jobs or other scheduled background tasks.
• We are using the default workgroup (default_wg) and do not currently have resource group isolation configured.
• The only effective mitigation we have found so far is scaling out the CN nodes.
My current understanding is that fragments become unevenly distributed across CNs, causing the global pipeline queue to become saturated or stuck, which then leads to long scheduling delays.
My questions are:
• What are the most common root causes of scheduling queue explosions like this?
• Besides adding more CN nodes, are there recommended ways to prevent or mitigate this situation?
• Would configuring Resource Groups / Workgroups help in this scenario, even if the total query volume does not increase significantly?
Any insights or debugging recommendations would be greatly appreciated. Thanks!