@Rocky compact_threads is set to 12 for all BE nodes and update_compaction_num_threads_per_disk is 3. I have 4 EBS volumes per node. Why starrocks_be_running_update_compaction_task_num showing maximum concurrent compaction tasks of 16?

The observed value of 16 for

starrocks_be_running_update_compaction_task_num

despite your configurations likely stems from the default behavior of the Compaction Scheduler, especially if you are using a Shared-Data (Cloud-Native) cluster or a recent version of StarRocks. Here are the most common reasons for this discrepancy: 1. Shared-Data "Adaptive" Task Limit If your cluster is in Shared-Data mode (using CN nodes or Compute-Storage Separation), the number of concurrent compaction tasks is primarily governed by a Frontend (FE) parameter rather than the BE thread settings alone. * Parameter:

lake_compaction_max_tasks

(FE configuration) * Default Logic: The default value is

-1

, which calculates the limit as Number of surviving nodes × 16. * Result: In a single-node or test environment, the FE will dispatch up to 16 tasks to the BE. Even if you limit

compact_threads

on the BE, the metric

running_update_compaction_task_num

may reflect the tasks dispatched and "active" in the BE's task queue or executor state, which defaults to 16 per node in the scheduler's logic. 2. Thread Pool Logic for Primary Key Tables The metric

starrocks_be_running_update_compaction_task_num

specifically tracks Update Compaction (used for the Primary Key model). * Configuration: You have

update_compaction_num_threads_per_disk = 3

and 4 disks, which calculates to 12 threads. * Overlap with `compact_threads`: In some StarRocks versions, the system uses the higher of

compact_threads

or the disk-based calculation. However, since you set both to 12, this doesn't explain the jump to 16. * CPU Core Defaults: StarRocks often initializes internal thread pools based on the number of CPU cores if the configuration is ambiguous or if there's a "baseline." If your BE nodes have 16 CPU cores, the update compaction pool might be defaulting to the core count (1:1 ratio) or a fixed cap of 16 often found in the source code for task concurrency. 3. Metric Interpretation The metric

running_update_compaction_task_num

counts the number of tasks currently being handled by the BE's

UpdateCompactionExecutor

. * If the FE dispatches 16 tasks (due to the

lake_compaction_max_tasks

default), the BE's executor will show 16 tasks as "running" (submitted to the executor), even if they are contending for the 12 actual physical threads you defined. Recommendations 1. Check FE Config: Run

ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';

. If it is

-1

, it is using the

nodes * 16

formula. 2. Verify Disk Count: Ensure the BE correctly detects exactly 4 disks by checking the

storage_root_path

in

be.conf

or looking at the logs during BE startup. If it detects more paths (e.g., a default root path + your 4 EBS volumes), the calculation

3 * disks

will be higher. 3. Check CPU Cores: If your node has 16 cores, the system may be defaulting to that value. 4. Adjust FE Scheduler: If you want to strictly limit the concurrency to 12, try setting the FE parameter:

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "12"); -- Total cluster-wide

_(Note: This is a global limit. If you have multiple nodes, set it to

12 _ num_nodes

.)* References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md

@Rocky This is for shared nothing cluster primary key table

The discrepancy you are seeing—16 concurrent tasks despite a calculated limit of 12—is common in StarRocks v3.x+ and usually occurs because of how the Compaction Scheduler and Primary Key (PK) Update Compaction threads are managed. Here are the most likely reasons why you see 16 in your Shared-Nothing cluster: 1. The FE Adaptive Scheduler (Default 16) In StarRocks 3.1 and later, the Frontend (FE) has a more active role in scheduling compaction for Primary Key tables. There is a global FE parameter called

lake_compaction_max_tasks

. * Default Behavior: Its default value is

-1

, which triggers an adaptive calculation: Number of BE nodes × 16. * Why it affects you: Even in a Shared-Nothing cluster, the system often defaults to a "base" concurrency of 16 tasks per node for the PK update compaction framework. If you have only one node (or the scheduler dispatches per node), it will push up to 16 tasks to the BE. * Metric: The metric

starrocks_be_running_update_compaction_task_num

counts the number of tasks the BE has accepted and is "processing" in its executor. If the FE sends 16, the BE will show 16, even if some are waiting for the actual thread pool to pick them up. 2. Miscounted "Disks" The calculation

update_compaction_num_threads_per_disk * num_disks

depends on what the BE considers a "data directory." * Check `storage_root_path`: In your

be.conf

, do you have exactly 4 paths? Sometimes StarRocks includes the default

${STARROCKS_HOME}/storage

directory if it wasn't explicitly excluded or if a path was repeated. * Verification: Run the following SQL to see how many disks the BE has actually registered:

sql
    SHOW BACKENDS\G

Look at the

StoragePath

or disk-related columns. If the BE identifies 5 or 6 disks (e.g., 4 EBS + 1 root/system disk), the math

3 _ 5 = 15

or

3 _ 6 = 18

gets much closer to 16. Note that if the result is

4 * 4

, it equals exactly 16 (perhaps the setting didn't apply or the default changed). 3. CPU Core Baseline The

compact_threads

parameter (which you set to 12) is primarily for Cumulative and Base compaction (used by Duplicate/Unique/Aggregate models). For Primary Key tables, only

update_compaction_num_threads_per_disk

and its internal executor apply. * Many StarRocks thread pools have a fallback logic: if the calculated thread count is low, they may default to

min(16, CPU_cores)

. If your nodes have 16 or more CPU cores, the BE may be defaulting the PK executor to 16 to ensure high-performance merging for the PK index. 4.

compact_threads

vs.

update_compaction

It is important to note that

compact_threads

and

update_compaction_num_threads_per_disk

do not limit each other. They control two different thread pools: * `compact_threads`: Used for standard LSM compaction. * `update_compaction...`: Used for the PK-specific "Update Compaction" (merging deltas/primary key indexes). Since they are separate, the metric

running_update_compaction_task_num

is completely independent of your

compact_threads = 12

setting. How to Fix/Verify If you want to strictly limit this to 12 and verify the source: 1. Check FE Config:

sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';

If it is

-1

, try setting it to a fixed number (e.g.,

12 * number_of_BEs

):

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "12");

2. Verify BE Config Application: Check if the BE actually acknowledged your `update_compaction_num_threads_per_disk = 3`:

sql
    SELECT * FROM information_schema.be_configs WHERE name LIKE '%update_compaction%';

3. Check for "Extra" Disks: Check

storage_root_path

in

be.conf

. If you see 4 EBS volumes but the BE is under heavy load, it may be showing the queued tasks in that metric as well. References

@Rocky sum(rate(starrocks_be_cpu{mode="iowait", starrocks_cluster="core-services"}[$interval])) by (starrocks_cluster, aws_name) / sum(rate(starrocks_be_cpu{starrocks_cluster="core-services"}[$interval])) by (starrocks_cluster, aws_name) * 100 Does this metric include total iowait or just iowait for ingestion jobs?

This metric represents the total I/O wait for the Backend (BE) process as a whole. It does not distinguish between ingestion, queries, or other internal tasks like compaction. Here is a breakdown of why: 1. Process-Level Scope: The

starrocks_be_cpu

metric is a process-level counter exported by the BE. The

mode="iowait"

value is captured from the operating system's view of the process threads. When a CPU core is idle because a thread is waiting for a disk I/O operation to complete, it is recorded as

iowait

. 2. Shared Resources: In StarRocks, the BE process performs disk I/O for multiple reasons simultaneously: * Ingestion: Writing data to the write-ahead log (WAL) and segment files. * Queries: Reading data segments from disk into memory. * Compaction: Reading old segments and writing newly merged segments. * Logging/Metadata: Writing to

<http://be.INFO|be.INFO>

logs or updating local metadata. 3. OS Limitation: The operating system tracks I/O wait based on the CPU's state regarding outstanding disk requests. It does not have visibility into "why" the StarRocks BE initiated the I/O (e.g., it doesn't know if a read was for a user query or a compaction task). How to isolate Ingestion I/O? If you specifically want to monitor I/O related to ingestion, you should look for metrics specifically prefixed with

load

or

tablet_writer

, for example: * `starrocks_be_tablet_writer_wait_done_time_ms`: Time spent waiting for I/O during data writing. * `starrocks_be_load_channel_add_batch_time_ms`: Total time taken to add batches during ingestion (includes I/O). Summary of your formula Your formula calculates the percentage of total CPU capacity spent waiting on any disk I/O. A high value here indicates that the BE is disk-bound, which could be caused by heavy ingestion, but could also be caused by massive scan-heavy queries or aggressive background compaction. Recommendations: * If this value is high during ingestion, check the

starrocks_be_disks_io_util

metric to see if physical disks are saturated. * Check

starrocks_be_compaction_running_task_num

to see if background maintenance is competing for disk bandwidth. References • loading/load_from_pulsar.md • administration/management/monitoring/alert.md • loading/minio.md