@Rocky This time, we experienced an incident where the memory usage on a single CN node suddenly spiked to the memory limit. Shortly afterward, a large number of queries started failing with "Memory Exceed" errors. After restarting the CN, the issue was resolved. Our current hypothesis is that the driver workers became stuck, preventing fragments from being processed. As a result, fragment-related memory continued to accumulate until the process eventually hit the memory limit. Have you ever seen a similar issue before? Is there any known case where stuck driver workers caused fragment processing to stop, leading to continuous memory accumulation and ultimately triggering memory limit exceeded errors?

The scenario you described—where memory usage on a CN node spikes due to fragments not being released—is a known issue that has been reported in certain versions of StarRocks. Known Issues and Root Causes There is a specific case documented in the community (GitHub Issue #53155) titled "[BUG] CN nodes can not release query_pool Fragment memory." In this scenario, the issue matches your hypothesis: * Stuck Cleanup: The

result_mgr()->cancel()

process or the internal

cancel_worker

might fail to complete the cleanup of fragment instances. * Memory Accumulation: When fragments are canceled or fail, their associated memory in the

query_pool

is supposed to be released. If the driver workers or the cleanup thread becomes "stuck" or delayed, this memory remains occupied. * Continuous Spike: As new queries arrive and generate more fragments, the memory usage continues to rise until the node hits the

mem_limit

, at which point all subsequent queries fail with "Memory Exceed" errors. Why Driver Workers Might Get Stuck In the StarRocks Pipeline Engine, driver workers can become stuck for several reasons: 1. Deadlocks: Contention between pipeline execution locks and metadata/resource locks. 2. External I/O Hangs: In shared-data architectures, if the CN is waiting indefinitely for object storage responses (S3/HDFS), the driver may stay in a "pending" or "blocked" state, holding onto its memory buffer. 3. Async Task Completion: If an asynchronous task (like a scan or a remote exchange) never reports completion, the driver remains in the

PendingFinish

state. How to Verify if This Happens Again If you encounter this again before upgrading, you can use the following to confirm the state: * Check Metrics: Use the BE/CN metrics endpoint (

http://<IP>:<PORT>/metrics

) to look for: * `pipe_poller_block_queue_len`: A high value indicates many drivers are blocked. * `pipe_driver_queue_len`: Shows drivers waiting to be scheduled. * Memory Trackers: Visit

http://<IP>:<PORT>/mem_tracker

to see if the

query_pool

is indeed the largest consumer and if it fails to shrink even when no queries are active. * Thread Dumps: Capturing a stack trace (

jstack

for FE or `pstack`/`gcore` for CN/BE) during the hang can help identify if threads are stuck in a specific method (e.g.,

PlanFragmentExecutor::cancel

). Recommendations * Version Check: This issue has been addressed in newer patch releases. If you are on an older version of 3.x (like 3.1.x or early 3.2.x), consider upgrading to the latest stable patch (e.g., 3.2.10+, 3.3.3+). * Monitoring Alerts: Set alerts on

starrocks_be_query_pool_mem_bytes

relative to the total process memory. If the query pool doesn't drop after a period of inactivity, it indicates a leak or stuck fragments that fragments are stuck. For deep troubleshooting, you can refer to the official StarRocks GitHub Issues or provide the

cn.out

logs to the community for further analysis. References • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md