@Rocky CN are at 100% CPU utilization but no running queries, compaction or other stuff. How do I check what operation consuming CPU on the cluster?

When your Compute Nodes (CN) show 100% CPU utilization without active queries or compaction, it usually indicates internal thread activity, spin-locks, or background system tasks. To identify the specific operations consuming CPU, follow these steps: 1. Identify High-CPU Threads (OS Level) Use the

top

command to see which specific threads within the StarRocks process are busy:

bash
# Replace <cn_pid> with the actual Process ID of the starrocks_be process
top -Hp <cn_pid>

* What to look for: Note the

PID

(Thread ID) and the name of the thread (e.g.,

brpc_worker

,

pipeline_engine

, or

token_scan

). This helps narrow down if the CPU usage is in networking, query execution, or background maintenance. 2. Generate a CPU Profile (Recommended) StarRocks includes a built-in profiling tool that generates a flame graph to show exactly which C++ functions are consuming CPU time. Run the collection script from the CN's

bin

directory:

bash
./bin/collect_be_profile.sh --profiling-type cpu --duration 30

* Access results: After 30 seconds, the script will generate a profile. You can often view these via the Frontend (FE) Web UI (default port

8030

) under the "proc profiles" tab. Select the specific CN node to view the flame graph. * Interpretation: The wider a box in the flame graph, the more CPU that function is consuming. 3. Use Linux

perf

for Real-time Analysis If you have

perf

installed, you can see a live view of function calls:

bash
sudo perf top -p <cn_pid> -g

This is useful if the CPU spike is intermittent or the profiling script is unavailable. 4. Check Internal Metrics Query the StarRocks metrics endpoint or use

SHOW PROC

to see if there are "hidden" background tasks: * Check Node Status:

sql
    SHOW PROC '/compute_nodes';

Verify if

CpuUsedPct

matches your OS-level findings. * Check Metrics: Access

http://<cn_ip>:<be_http_port>/metrics

(default port 8040) and search for

starrocks_be_cpu

. 5. Potential Causes for "Silent" High CPU If there truly are no queries or compactions, consider these common causes: * High RPC Traffic: If you have many small metadata updates or heartbeats,

brpc

threads may consume high CPU. Look for

brpc

or

Socket

functions in your profile. * Shared Data Cache: Since CNs are typically used in shared-data mode, they may be busy with cache eviction or metadata synchronization. * Log Overload: Check

be/log/cn.INFO

or

cn.WARNING

. If a specific error is occurring thousands of times per second, the logging process itself can consume significant CPU. * Zombie/Stuck Tasks: Occasionally, a pipeline task may get stuck in a tight loop. The CPU profile (Step 2) will identify the specific code path. Summary Recommendation: Start with

top -Hp <pid>

to see the thread names. If the cause isn't obvious, run

./bin/collect_be_profile.sh

and examine the flame graph. That graph will tell you the exact internal operation (e.g., memory management, network IO, or a specific algorithm) causing the load. References • administration/management/proc_profile.md • faq/fe_mem_faq.md • administration/management/monitoring/alert.md • knowledge/trouble_shooting/fe_deadlock_problems.md

how do I “kick” the node / gracefull restart it on K8s?

It looks like you might have sent that message by mistake. Do you need more help troubleshooting the high CPU usage on your CN nodes, or did you have another question? If you've already tried the

collect_be_profile.sh

script or checked

top -Hp

, feel free to share what you found!

@Rocky how do I kick the node/gracefully restart it

To "kick" or gracefully restart a Compute Node (CN), you have two options depending on whether you want to temporarily restart the service or permanently remove the node from the cluster. 1. Gracefully Restart the Node (Recommended for high CPU) If the node is unresponsive or stuck at 100% CPU, a graceful restart allows it to finish current tasks before shutting down. Step A: Stop Gracefully Navigate to the CN

bin

directory and run:

bash
# For StarRocks v3.3+
./bin/stop_cn.sh -g

# For StarRocks v3.4+ (you can specify a longer timeout in seconds)
./bin/stop_cn.sh -g --timeout 600

* What happens: The node will enter a

SHUTDOWN

state. It stops accepting new query fragments but waits for existing ones to finish (default wait is 20 seconds unless configured otherwise via

loop_count_wait_fragments_finish

). Step B: Start the Node Once the process has stopped, start it again:

bash
./bin/start_cn.sh --daemon

────────── 2. "Kick" the Node (Permanent Removal) If you want to completely remove a malfunctioning CN node from the cluster metadata, you must use a SQL command. Note that unlike BE nodes, CNs cannot be "decommissioned" because they are stateless. Run this on your MySQL client connected to the FE:

sql
-- Replace with the actual IP and heartbeat port (default 9050)
ALTER SYSTEM DROP COMPUTE NODE "10.0.0.1:9050";

* Warning: This operation is synchronous and cannot be cancelled. The node is removed immediately from the FE's scheduling list. ────────── 3. Monitoring the Status During the process, you can monitor the node's status using:

sql
SHOW COMPUTE NODES;

Look at the

StatusCode

column: * OK: Running normally. * SHUTDOWN: Graceful exit is in progress. * DISCONNECTED: The node process is down or the network is severed. Summary: Which one to use? * To fix high CPU: Use

stop_cn.sh -g

followed by

start_cn.sh --daemon

. * If the node is a "zombie" or needs to be replaced: Use

ALTER SYSTEM DROP COMPUTE NODE

. Since CNs are stateless, you don't need to worry about data migration/rebalancing like you would with a BE node. References • administration/management/graceful_exit.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • deployment/upgrade.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md References • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • faq/operation_maintenance_faq.md • administration/management/Scale_up_down.md

@Rocky it seems that majority of CPU is consumed by

pip_con_scan_co

but all the queries were cancelled

The thread name

pip_con_scan_co

stands for Pipeline Connector Scan Executor. These threads are responsible for scanning data from external tables (via Catalogs like Hive, Iceberg, Hudi, JDBC, or Elasticsearch). If these threads are consuming 100% CPU even after queries are cancelled, it indicates that "zombie" scan tasks are still running on the CN nodes and have not yet acknowledged the cancellation signal. Why this happens 1. External Library Blocking: Scan tasks for external tables often use third-party libraries (e.g., S3 SDK, HDFS client, or JDBC drivers). If a thread is stuck inside one of these libraries performing I/O or decompression, it may not check StarRocks' "is_cancelled" flag until the current block or batch of data is finished. 2. Deferred Cancellation: When you cancel a query at the FE (Frontend), it sends a signal to the CNs. If the CN is already under 100% CPU load, the thread responsible for processing that cancellation signal may be starved, delaying the actual stop of the scan tasks. 3. Adaptive I/O Overhead: StarRocks has an "adaptive I/O" feature for connectors that can sometimes spawn a large number of tasks. If these tasks are tight-looping on metadata or small files, they can saturate the CPU. How to verify and stop the "Phantom" load 1. Check for Backend "Zombie" Fragments Even if

SHOW PROCESSLIST

is empty, there may be fragments still active on the CN. Run this on the FE:

sql
-- This shows queries actually active on the backends/CNs
SHOW PROC '/backend_queries';

If you see entries here that correspond to the cancelled queries, the CN is still trying to execute them. 2. Check for Metadata Loops (Log Analysis) Check the

<http://cn.INFO|cn.INFO>

log on the affected node. Look for repeating patterns like: * Continuous "Slow I/O" warnings. * Repeated connection retries to S3/HDFS/Glue. * Errors related to

ConnectorScanNode

. 3. Immediate Mitigation (SQL) If you cannot restart the node yet, try to disable the adaptive I/O and reduce the thread concurrency globally to see if the threads "drain" and exit:

sql
-- Disable the adaptive I/O task spawning for connectors
SET GLOBAL enable_connector_adaptive_io_tasks = false;

-- Reduce the number of connector scan threads (if possible in your version)
-- Default is usually 8 or 16 per CPU core.
SET GLOBAL pipeline_connector_scan_thread_num_per_cpu = 1;

4. Forceful Cleanup If

pip_con_scan_co

threads do not stop after 5-10 minutes of "cancelled" status: 1. Identify the query ID from the

<http://cn.INFO|cn.INFO>

logs or

SHOW PROC '/backend_queries'

. 2. If

KILL QUERY <id>

doesn't work, a graceful restart (as described in the previous message) is the only way to clear the thread pool. Use

./bin/stop_cn.sh -g

and wait. If it doesn't stop within a few minutes, a hard

kill -9

of the CN process may be necessary to reclaim the CPU. Prevention for the Future If this happens frequently with external table queries, consider setting a lower

pipeline_dop

(Degree of Parallelism) or

connector_io_tasks_per_scan_operator

for those specific heavy queries:

sql
SET pipeline_dop = 4;
SET connector_io_tasks_per_scan_operator = 4;
-- Run your query

References • administration/management/BE_configuration.md • knowledge/usage/best-practice.md • administration/management/monitoring/metrics.md References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • knowledge/usage/best-practice.md