@Rocky How do I keep track of shared data compaction health for primary key tables

To track the health and status of compaction for Primary Key tables in a shared-data (StarOS) cluster, you can use a combination of system tables,

SHOW PROC

commands, and error logs. 1. Monitor Compaction Scores (Health Status) The most direct way to check compaction health is via the compaction score. A high score indicates that data is being ingested faster than it can be compacted. *

information_schema.partitions_meta

(StarRocks v3.1.9+): Query this table to see the compaction status at the partition level.

sql
    SELECT DB_NAME, TABLE_NAME, PARTITION_NAME, AvgCS, MaxCS
    FROM information_schema.partitions_meta;

* AvgCS: Average compaction score of all tablets in the partition. * MaxCS: Maximum compaction score among all tablets in the partition. * Thresholds: If these scores exceed 100, the system may trigger ingestion slowdowns. If they exceed 2000 (default), ingestion may stop entirely. 2. Monitor Active Compaction Tasks To see what the system is currently doing, you can view the progress of specific compaction jobs. *

information_schema.be_cloud_native_compactions

(StarRocks v3.2.0+): Shows real-time progress and status of compaction tasks.

sql
    SELECT * FROM information_schema.be_cloud_native_compactions;

Pay attention to the PROGRESS and STATUS fields. If a task fails, the error message will appear in the

STATUS

column. *

SHOW PROC '/compactions'

: Provides an overview of current compaction tasks across the cluster, including StartTime and TxnID. 3. Check for Ingestion Slowdowns Primary Key tables in shared-data mode will actively delay commits if compaction is lagging. You can catch this by monitoring running transactions: *

SHOW PROC '/transactions/{db_name}/running'

: Check the

ErrMsg

field. If you see messages like: >

Partition's compaction score is larger than 100.0, delay commit for xxxms.

This is a clear indicator that your compaction resources (threads) or settings cannot keep up with your ingestion rate. 4. Key Performance Tuning for Primary Key Tables If you find that your compaction scores are consistently high, consider adjusting these settings: * BE Configuration (

be.conf

): * `compact_threads`: Increase this (default is typically 4) to allow more concurrent compaction tasks on the compute nodes. * FE Configuration (

fe.conf

): * `lake_ingest_slowdown_threshold`: Default is 100. Increasing this allows higher latency before slowing down ingestion (use with caution as it affects query performance). * `lake_compaction_score_upper_bound`: Default is 2000. This is the hard limit where ingestion will fail. For more detailed information, you can refer to the Primary Key Table Best Practices and Shared-data Compaction documentation. References • best_practices/primarykey_table.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky Is there a prometheus/grafana metrics I can use to track?

To monitor the health of Primary Key (PK) table compaction in a shared-data (StarOS) cluster using Prometheus and Grafana, you should focus on the following specific metrics. StarRocks provides these via the BE and FE

/metrics

endpoints. 1. Core Health Metrics (Compaction Score) The "Compaction Score" is the most critical metric. It represents the number of un-compacted rowsets. For PK tables, this metric is tracked separately because of its unique indexing requirements. * Metric Name:

starrocks_be_tablet_update_max_compaction_score

* Description: The highest compaction score among all Primary Key tablets on a specific BE/CN node. * Alert Threshold: *

100:

Ingestion slowdown starts (latency increases). *

2000:

Ingestion may stop entirely (default limit). * Metric Name:

starrocks_fe_tablet_max_compaction_score

* Description: The FE's view of the maximum compaction score across the entire cluster. Use this for cluster-wide alerting. 2. Compaction Activity & Performance These metrics help you understand if the compaction system is actually working or if it's falling behind. * Metric Name:

starrocks_be_update_compaction_outputs_total

* Description: The total count of completed Primary Key compaction tasks. A flat line during high ingestion periods suggests compaction is stuck or threads are exhausted. * Metric Name:

starrocks_be_update_compaction_duration_us

* Description: The time taken to complete PK compaction tasks. If this trend is upward, it indicates that individual compaction tasks are becoming heavier or storage IO is bottlenecked. * Metric Name:

starrocks_be_compaction_mem_bytes

* Description: Memory consumed by all compaction tasks on the BE. Since PK compaction requires loading indexes into memory, monitor this to prevent OOM. 3. Queue & Resource Pressure * Metric Name:

starrocks_be_compaction_thread_pool_queue_size

* Description: The number of compaction tasks waiting for an available thread. If this is consistently high, you should increase

compact_threads

in

be.conf

. * Metric Name:

starrocks_be_max_tablet_rowset_num

* Description: While general, this tracks the maximum number of versions (rowsets) in any tablet. For PK tables, keeping this low is vital for read performance. Recommended Grafana Setup StarRocks provides official Grafana dashboards that include these metrics out-of-the-box: 1. Shared-data Dashboard: Specifically designed for StarOS clusters. Look for the "Compaction" and "Publish Version" sections. 2. BE/CN Dashboard: Check the "Task" section for compaction thread pool usage and memory. Quick PromQL Alert Examples High Compaction Pressure Alert:

promql
starrocks_fe_tablet_max_compaction_score > 100

Compaction Tasks Failing/Stuck:

promql
rate(starrocks_be_update_compaction_outputs_total[5m]) == 0 and rate(starrocks_be_compaction_thread_pool_queue_size[5m]) > 0

``` --- References: - [Primary Key Table Best Practices](https://docs.starrocks.io/docs/best_practices/primarykey_table/) - [General Monitoring Metrics](https://docs.starrocks.io/docs/administration/management/monitoring/metrics/) - [Shared-data Monitoring Metrics](https://docs.starrocks.io/docs/administration/management/monitoring/Monitoring_Metrics_for_shared-data_data_data/) ## References - [best_practices/primarykey_table.md](https://docs.starrocks.io/docs/best_practices/primarykey_table) - [knowledge/admin_and_operation/shared_data_compaction.md](https://docs.starrocks.io/docs/knowledge/admin_and_operation/shared_data_compaction) - [administration/management/compaction.md](https://docs.starrocks.io/docs/administration/management/compaction) - [administration/management/monitoring/metrics.md](https://docs.starrocks.io/docs/administration/management/monitoring/metrics)

@Rocky MySQL [requests]> SELECT c.table_name, COUNT(*) tablets, SUM(t.num_rowset) total_rowsets, MAX(t.num_rowset) max_rowsets_per_tablet FROM information_schema.be_tablets t JOIN information_schema.tables_config c ON t.table_id = c.table_id WHERE c.table_name = 'requests' GROUP BY c.table_name; +------------+---------+---------------+------------------------+ | table_name | tablets | total_rowsets | max_rowsets_per_tablet | +------------+---------+---------------+------------------------+ | requests | 3584 | 70085 | 378 | +------------+---------+---------------+------------------------+ This is the stats for my shared data table. The shared nothing equivalent has max rowsets in the range of 20-30 only. This explains why my queries are ~13x worse on shared data even though I provisioned more than enough NVMe for 100% cache hit rate. Explain what's going on. How do i fix this

The high rowset count in your shared-data cluster (378 vs. 20-30 in shared-nothing) is caused by the different compaction architecture and default "lazy" scheduling policies designed for cloud storage. In shared-data mode, StarRocks prioritizes reducing "write amplification" (and thus S3/Object Storage costs) over minimizing rowset counts. Because every compaction task requires writing new files back to object storage and a full metadata "publish" cycle, the system is less aggressive by default than when writing to local NVMe. Why this is happening 1. Size-Tiered Strategy: Primary Key tables in shared-data mode typically use a Size-tiered compaction strategy. It groups rowsets of similar sizes and only merges them when a "tier" reaches a certain count. This prevents small updates from triggering a massive merge of the entire base data, but it results in a "long tail" of small rowsets that increase query-time merge overhead. 2. FE-Driven Scheduling: Unlike shared-nothing where each BE manages its own local disks, the FE (Frontend) schedules compaction for shared-data clusters. This adds scheduling overhead and often results in fewer concurrent compaction tasks than a locally managed system. 3. Higher Default Limits: The default

lake_pk_compaction_max_input_rowsets

can be as high as 500. In shared-nothing, the system would have triggered "Base Compaction" long before reaching 378 rowsets. How to Fix It 1. Force a Manual Compaction (Immediate Fix) You can manually trigger a full compaction to merge those 378 rowsets down to a single-digit number. This will immediately improve query performance.

sql
ALTER TABLE requests COMPACTION;

_Note: Monitor the progress via

SELECT _ FROM information_schema.be_cloud_native_compactions WHERE TABLE_NAME = 'requests';

.* 2. Tune Compaction Concurrency (Long-term) If your ingestion rate is high, the default 4 threads may not be enough to keep up. * On all BE/CN nodes: Increase the worker threads.

sql
    -- Recommended: 25% of your CPU cores
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";

* On FE: Increase the total concurrent tasks allowed in the cluster.

sql
    -- -1 is adaptive (CN nodes * 16), but you can set a hard limit
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "64");

3. Make Compaction More Aggressive To force the system to merge rowsets more frequently (getting closer to that 20-30 range): * Reduce Input Rowsets: Lowering this makes individual tasks finish faster and run more often.

sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "lake_pk_compaction_max_input_rowsets";

* Check "Cumulative" limit:

sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

Summary of Performance Impact Even with 100% NVMe cache hit rate, a high rowset count forces the Primary Key index to perform hundreds of lookups/merges per block during query execution to determine the latest version of a row. Reducing the rowset count from 378 to <50 will likely eliminate the 13x performance gap you are seeing. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • best_practices/primarykey_table.md • deployment/shared_data/feature-support-shared-data.md References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky I made the changes suggested MySQL [requests]> SELECT c.table_name, COUNT(*) tablets, SUM(t.num_rowset) total_rowsets, MAX(t.num_rowset) max_rowsets_per_tablet FROM information_schema.be_tablets t JOIN information_schema.tables_config c ON t.table_id = c.table_id WHERE c.table_name = 'requests' GROUP BY c.table_name; +------------+---------+---------------+------------------------+ | table_name | tablets | total_rowsets | max_rowsets_per_tablet | +------------+---------+---------------+------------------------+ | requests | 3584 | 79441 | 350 | +------------+---------+---------------+------------------------+ Still no help. Is the issue that I jsut need to provision more CN nodes?

The fact that your rowset count barely moved (378 to 350) after increasing threads suggests that the bottleneck isn't just "available CPU" on your CNs—it's likely the FE scheduling throughput or the heavy weight of your partition tasks. With 3,584 tablets, a single partition-level compaction task becomes an enormous transaction. In shared-data mode, FE dispatches compaction at the partition level. If one partition has 3,500+ tablets, the FE must coordinate 3,500 sub-tasks, wait for all to finish, and then commit them to the metadata store (StarOS). Here is how to diagnose and fix the specific bottleneck: 1. Identify if FE is Throttling (The "Max Tasks" Barrier) Adding more CN nodes will only help if your

lake_compaction_max_tasks

is high enough to use them. * Check your current limit:

ADMIN SHOW FRONTEND CONFIG LIKE 'lake_compaction_max_tasks';

_ If it is the default

-1

, it scales as

CN_count _ 16

. * The Fix: If you have many partitions, increase this significantly (e.g.,

100

or

200

) so the FE dispatches more work simultaneously.

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "200");

2. Check for "Metadata Commitment" Bottlenecks With 3,500 tablets, the "Commit" phase of compaction can be slow because it updates the metadata for every single tablet. * Run:

SHOW PROC '/compactions';

* Look at the

CommitTime

and

FinishTime

. * If

StartTime

to

CommitTime

is long, the CNs are slow (need more CNs/Threads). * If

CommitTime

to

FinishTime

is long, the FE/StarOS metadata service is the bottleneck. Adding CN nodes will not fix this. 3. Check for Tablet Over-provisioning 3,584 tablets is quite high unless your table is multi-terabyte. * In shared-nothing, more tablets = more IO parallelism. * In shared-data, too many tablets hurts compaction. Each tablet version must be tracked in the metadata store. If you have 3,584 tablets and ingest every minute, you are creating 3,584 metadata entries per minute. * The Fix: Consider reducing the bucket count. For Primary Key tables in shared-data, the recommendation is typically one tablet per 1-2 GB of data. If your table is 300GB, 3,584 tablets is ~10x more than you need. 4. Adjust the Size-Tiered Policy (The "Aggression" Fix) PK tables use a "Size-tiered" strategy. If your rowsets are very small, the system might decide they don't "deserve" to be merged yet to save on S3 write costs. * Force a deep merge: To see if the system can keep up, manually trigger a full compaction:

sql
    ALTER TABLE requests COMPACTION;

* Lower the trigger threshold:

sql
    -- On CN nodes
    UPDATE information_schema.be_configs SET value = "50" WHERE name = "lake_pk_compaction_max_input_rowsets";

Do you need more CN nodes? Only if: 1.

SHOW PROC '/compactions'

shows many tasks with

FinishTime = NULL

and

Progress < 100%

. 2.

starrocks_be_compaction_thread_pool_queue_size

is consistently > 0. 3. Your CN CPU utilization is hitting 80%+. Otherwise, the issue is likely metadata overhead or scheduling limits. Try reducing the bucket count in your next schema iteration and increasing

lake_compaction_max_tasks

first. References • sql-reference/information_schema/be_cloud_native_compactions.md • sql-reference/information_schema/be_compactions.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky MySQL [requests]> SHOW PROC '/compactions'; +-------------------------------+-------+---------------------+---------------------+---------------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Partition | TxnID | StartTime | CommitTime | FinishTime | Error | Profile | +-------------------------------+-------+---------------------+---------------------+---------------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | requests.requests.188557 | 5784 | 2026-06-23 203054 | 2026-06-23 203057 | 2026-06-23 203124 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":10,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":202,"write_segment_count":51,"write_segment_mb":6,"write_remote_sec":4,"in_queue_sec":1} | | requests.requests.190851 | 5792 | 2026-06-23 203100 | 2026-06-23 203101 | 2026-06-23 203124 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":5,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":117,"write_segment_count":21,"write_segment_mb":2,"write_remote_sec":1,"in_queue_sec":6} | | requests.requests.189344 | 5793 | 2026-06-23 203101 | 2026-06-23 203103 | 2026-06-23 203124 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":5,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":105,"write_segment_count":19,"write_segment_mb":2,"write_remote_sec":1,"in_queue_sec":96} | | requests.requests.189214 | 5794 | 2026-06-23 203103 | 2026-06-23 203104 | 2026-06-23 203124 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":4,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":103,"write_segment_count":18,"write_segment_mb":2,"write_remote_sec":1,"in_queue_sec":28} | | requests.requests.189748 | 5796 | 2026-06-23 203104 | 2026-06-23 203107 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":3,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":78,"write_segment_count":14,"write_segment_mb":1,"write_remote_sec":1,"in_queue_sec":128} | | requests.requests.190981 | 5797 | 2026-06-23 203106 | 2026-06-23 203109 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":3,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":71,"write_segment_count":13,"write_segment_mb":1,"write_remote_sec":1,"in_queue_sec":7} | | requests.requests.190705 | 5800 | 2026-06-23 203109 | 2026-06-23 203112 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":2,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":59,"write_segment_count":11,"write_segment_mb":1,"write_remote_sec":0,"in_queue_sec":3} | | requests.requests.190019 | 5801 | 2026-06-23 203111 | 2026-06-23 203112 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":2,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":58,"write_segment_count":11,"write_segment_mb":1,"write_remote_sec":0,"in_queue_sec":0} | | requests.requests.190284 | 5806 | 2026-06-23 203112 | 2026-06-23 203114 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":2,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":50,"write_segment_count":9,"write_segment_mb":1,"write_remote_sec":0,"in_queue_sec":11} | | requests.requests.188688 | 5807 | 2026-06-23 203114 | 2026-06-23 203115 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":2,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":51,"write_segment_count":13,"write_segment_mb":1,"write_remote_sec":1,"in_queue_sec":0} | | requests.requests.189878 | 5809 | 2026-06-23 203116 | 2026-06-23 203117 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":1,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":40,"write_segment_count":7,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0} | | requests.requests.190424 | 5811 | 2026-06-23 203117 | 2026-06-23 203119 | 2026-06-23 203129 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":1,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":28,"write_segment_count":5,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":100} | | requests.requests.188297 | 5813 | 2026-06-23 203119 | 2026-06-23 203121 | 2026-06-23 203151 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":5,"write_segment_count":1,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":16} | | requests.requests.191381 | 5814 | 2026-06-23 203121 | 2026-06-23 203122 | 2026-06-23 203152 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0} | | requests.requests.191122 | 5816 | 2026-06-23 203123 | 2026-06-23 203124 | 2026-06-23 203152 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0} | | requests.requests.190149 | 5817 | 2026-06-23 203124 | 2026-06-23 203126 | 2026-06-23 203151 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":16} | | requests.requests.189614 | 5818 | 2026-06-23 203126 | 2026-06-23 203127 | 2026-06-23 203203 | NULL | {"sub_task_count":128,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":0,"write_segment_count":0,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0} | | requests.requests.195544 | 5822 | 2026-06-23 203128 | 2026-06-23 203137 | 2026-06-23 203203 | NULL | {"sub_task_count":128,"read_local_sec":6,"read_local_mb":718,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":2017,"write_segment_count":128,"write_segment_mb":541,"write_remote_sec":23,"in_queue_sec":119} | | requests.requests.193458 | 5830 | 2026-06-23 203138 | 2026-06-23 203145 | 2026-06-23 203203 | NULL | {"sub_task_count":128,"read_local_sec":3,"read_local_mb":110,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":1518,"write_segment_count":128,"write_segment_mb":46,"write_remote_sec":11,"in_queue_sec":79} | | requests.test_requests.195477 | 5837 | 2026-06-23 203146 | 2026-06-23 203151 | 2026-06-23 203152 | NULL | {"sub_task_count":64,"read_local_sec":4,"read_local_mb":959,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":640,"write_segment_count":64,"write_segment_mb":736,"write_remote_sec":18,"in_queue_sec":0} | | requests.requests.188145 | 5838 | 2026-06-23 203146 | 2026-06-23 203151 | NULL | NULL | NULL | | requests.requests.191656 | 5839 | 2026-06-23 203151 | 2026-06-23 203155 | NULL | NULL | NULL | | requests.requests.195544 | 5854 | 2026-06-23 203213 | NULL | NULL | NULL | NULL | +-------------------------------+-------+---------------------+---------------------+---------------------+-------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ What does this say

The output of

SHOW PROC '/compactions';

reveals exactly where the bottleneck is. The primary issue is not your CN nodes (CPU/NVMe), but Metadata Commitment Pressure on the Frontend (FE). Analysis of your Profile 1. CommitTime vs. FinishTime (The "Publish" Gap): Look at the difference between these two timestamps. Your tasks are taking ~3 seconds to commit (

StartTime

to

CommitTime

), but 20 to 30 seconds to finish (

CommitTime

to

FinishTime

). * What this means: The CN has already finished writing the data to S3. The system is now stuck in the Publish Version phase, where the FE must update the metadata for every single tablet in that partition. 2. Excessive Tablet Count (The Root Cause): Your table has 3,584 tablets. In shared-data mode, every compaction "Finish" is a heavy metadata transaction. Because the FE manages metadata centrally in StarOS, updating 3,584 tablet versions for every ingestion/compaction cycle is overwhelming the FE's metadata service. 3. High `in_queue_sec`: Values like

128

and

119

mean tasks are sitting in the FE's scheduling queue for over two minutes before they even start. This confirms the FE is the bottleneck, not the CNs. How to Fix It 1. Critical: Reduce the Bucket (Tablet) Count 3,584 tablets is far too many for a shared-data table unless it is 10TB+ in size. * The Rule of Thumb: Aim for 1GB - 2GB of data per tablet in shared-data mode. * The Fix: Re-create the table (or use

ALTER TABLE ... SET ("bucket_size" = "...")

if on a recent version) to reduce the bucket count to something like 64 or 128. This will reduce the metadata overhead by 95%, allowing the FE to publish versions in milliseconds instead of 30 seconds. 2. Tune FE Parallelism If you cannot change the bucket count immediately, you must tell the FE to handle more metadata tasks in parallel. * Increase Max Concurrent Tasks:

sql
    -- Allow the FE to schedule more partition compactions at once
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "128");

* Increase Publish Threads (If on v3.2+): Check if your version supports parallel PK index publish:

sql
    -- On CN nodes (via information_schema.be_configs)
    UPDATE information_schema.be_configs SET value = "true" WHERE name = "enable_pk_index_parallel_execution";

3. Enable Aggregate Publish (v3.2.4+ / v3.3+) Recent versions of StarRocks introduced "Aggregate Publish" specifically to solve this problem. It combines metadata updates for multiple tablets into a single S3/Metadata call. * Check your version. If you are on an older 3.x version, upgrading to 3.2.10+ or 3.3+ is the most effective way to handle high tablet counts in shared-data mode. Summary Your queries are slow because the rowsets aren't merging. The rowsets aren't merging because the FE is spending 30 seconds "publishing" every tiny compaction task due to the 3,584 tablets. Step 1: Lower your bucket count to 128. Step 2: Increase

lake_compaction_max_tasks

to 128. References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md • sql-reference/information_schema/be_cloud_native_compactions.md • administration/management/monitoring/metrics.md

@Rocky How do I check data size per tablet

To see the data size per tablet, you can use either a SQL query for a summary or a built-in command for a detailed list. Option 1: Summary Statistics (Recommended) Since you have 3,584 tablets, an aggregate view is the most useful way to see if they are under-sized. This query calculates the average, minimum, and maximum size per tablet in MB.

sql
SELECT 
    c.table_name,
    COUNT(*) AS tablet_count,
    ROUND(MIN(t.data_size) / 1024 / 1024, 2) AS min_size_mb,
    ROUND(MAX(t.data_size) / 1024 / 1024, 2) AS max_size_mb,
    ROUND(AVG(t.data_size) / 1024 / 1024, 2) AS avg_size_mb,
    ROUND(SUM(t.data_size) / 1024 / 1024 / 1024, 2) AS total_size_gb
FROM information_schema.be_tablets t
JOIN information_schema.tables_config c ON t.table_id = c.table_id
WHERE c.table_name = 'requests'
GROUP BY c.table_name;

Option 2: Detailed List If you want to see the size of every individual tablet, use this command:

sql
SHOW TABLET FROM requests;

Note: In shared-data mode, this returns

TabletId

,

BackendId

,

DataSize

, and

RowCount

. How to interpret the results: * If your

avg_size_mb

is small (e.g., < 100MB): Your table is significantly over-sharded. In shared-data mode, having 3,584 tiny tablets creates massive metadata overhead for the FE during compaction "publish" cycles, which is why your rowset count is staying high. * The Shared-Data Target: You should aim for 1,000MB to 2,000MB (1-2GB) per tablet. * The Fix: If your

total_size_gb

is, for example, 200GB, you only need about 128 to 256 tablets. Reducing the bucket count in your table schema will drastically speed up compaction and resolve the 13x performance gap. References • sql-reference/information_schema/be_tablets.md • faq/Deploy_faq.md • sql-reference/information_schema/be_txns.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md

@Rocky | 195637 | [17853] | 2.1GB | 1459404 | 0 | | 195638 | [10374] | 2.2GB | 1460091 | 0 | | 195639 | [17901] | 2.2GB | 1457942 | 0 | | 195640 | [17917] | 2.1GB | 1458686 | 0 | | 195641 | [17900] | 2.2GB | 1459556 | 0 | | 195642 | [17902] | 2.2GB | 1458174 | 0 | | 195643 | [17852] | 2.2GB | 1460218 | 0 | | 195644 | [17899] | 2.1GB | 1458057 | 0 | | 195645 | [17853] | 2.2GB | 1459826 | 0 | | 195646 | [10374] | 2.2GB | 1460774 | 0 | | 195647 | [17901] | 2.2GB | 1460077 | 0 | | 195648 | [17917] | 2.2GB | 1458369 | 0 | | 195649 | [17900] | 2.1GB | 1459808 | 0 | | 195650 | [17902] | 2.2GB | 1459210 | 0 | | 195651 | [17852] | 2.2GB | 1460170 | 0 | | 195652 | [17899] | 2.1GB | 1458675 | 0 | | 195653 | [17853] | 2.1GB | 1459122 | 0 | | 195654 | [10374] | 2.2GB | 1458510 | 0 | | 195655 | [17901] | 2.2GB | 1460900 | 0 | | 195656 | [17917] | 2.1GB | 1458682 | 0 | | 195657 | [17900] | 2.2GB | 1460482 | 0 | | 195658 | [17902] | 2.2GB | 1460279 | 0 | | 195659 | [17852] | 2.2GB | 1457916 | 0 | | 195660 | [17899] | 2.2GB | 1461977 | 0 | | 195661 | [17853] | 2.2GB | 1458519 | 0 | | 195662 | [10374] | 2.2GB | 1459340 | 0 | | 195663 | [17901] | 2.1GB | 1459016 | 0 | | 195664 | [17917] | 2.2GB | 1457589 | 0 | | 195665 | [17900] | 2.2GB | 1458351 | 0 | | 195666 | [17902] | 2.2GB | 1458212 | 0 | | 195667 | [17852] | 2.2GB | 1459478 | 0 | | 195668 | [17899] | 2.2GB | 1458235 | 0 | | 195669 | [17853] | 2.2GB | 1459076 | 0 | | 195670 | [10374] | 2.2GB | 1459732 | 0 | | 195671 | [17901] | 2.2GB | 1460586 | 0 | | 195672 | [17917] | 2.2GB | 1456880 | 0 | +----------+-----------+----------+----------+------------+ The newer tablets (when ingestion started) are already 1-2.5GB per tablet. My bucket count is already 128. How else can I remediate query slowness

The fact that your tablets are 2GB+ and your bucket count is 128 is a good start. However, 350 rowsets per tablet is the smoking gun for your 13x query slowdown. In Primary Key tables, query performance degrades significantly as the number of rowsets increases because the system must check more delete vectors and index entries. The high

in_queue_sec

(120s+) and the long gap between

CommitTime

and

FinishTime

in your

SHOW PROC

output indicate that compaction tasks are getting stuck in the pipeline. Here is the remediation plan to force those 350 rowsets down and restore performance: 1. Speed up "Publishing" (FE Metadata Bottleneck) The 20-30 second gap between

Commit

and

Finish

is the FE struggling to update metadata for all 128 tablets. * Enable Parallel PK Index Publish: This allows the system to update tablet metadata in parallel rather than sequentially.

sql
    -- Run on all CN nodes via information_schema.be_configs
    UPDATE information_schema.be_configs SET value = "true" WHERE name = "enable_pk_index_parallel_execution";

* Increase FE Scheduling Concurrency: Ensure the FE is actually dispatching enough work.

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "128");

2. Increase Compaction "Aggression" (CN Bottleneck) The

in_queue_sec

of 120s+ means your compaction threads are full, and new tasks are waiting for minutes before starting. * Double Compaction Threads: If you have 32 cores, set this to 16. If 16 cores, set to 8.

sql
    UPDATE information_schema.be_configs SET value = "16" WHERE name = "compact_threads";

* Allow more sub-tasks per table: By default, a compaction task for one partition only uses a few threads. You can force it to use more threads for your 128 tablets:

sql
    ALTER TABLE requests SET ("lake_compaction_max_parallel" = "16");

3. Change the Compaction Strategy Primary Key tables use "Size-tiered" compaction. If your ingestion creates many small files, the system might "ignore" them because they are too small to trigger a tier merge. * Make it "Greedy": Decrease the input limit to make tasks finish faster, which allows the queue to clear.

sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

* Lower the trigger threshold: Lowering this makes the FE pick up partitions with fewer rowsets earlier.

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "5");

4. Optimize your Ingestion (The Root Cause) If you have 350 rowsets, it usually means you are performing many small "Stream Loads" or "Routine Loads." * Batching is critical: Aim for one load every 30-60 seconds. If you are loading every 1-5 seconds, you are creating rowsets faster than the FE can "Publish" them. * Increase Batch Size: If using Routine Load, increase

max_batch_interval

to

60

and

max_batch_rows

to

500000

. Summary Checklist: 1. Is CN CPU high? If yes, add CN nodes. 2. Is

in_queue_sec

still high? Increase

compact_threads

. 3. Is

FinishTime

still 30s after

CommitTime

? Check FE CPU and enable

enable_pk_index_parallel_execution

. 4. Is rowset count still ~350? Reduce ingestion frequency and run

ALTER TABLE requests COMPACTION;

once to kickstart a deep merge. References • administration/management/compaction.md • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md References

@Rocky I think I found the issue. My FE node

ethtool_bw_out_allowance_exceeded

Is constantly at 3000-4000. Is this limiting my compaction? Also, end of the day even with plenty of NVMe. is shared nothing still more performant and reliable than shared data for ultra high throughput like adtech RTB requests? I'm trying to get advantage of elasticity and durability of shared data at similar costs of shared nothing

Yes, FE network throttling is absolutely the reason your compaction is stuck. In Shared-Data (Lake) mode, the Frontend (FE) is not just a query planner; it is the Metadata Master. Every time a compaction task finishes on a CN node, the FE must "Publish" that new version by writing metadata updates to the StarOS/Object Store backend. 1. How the FE Network Throttling Kills Compaction The

ethtool_bw_out_allowance_exceeded

metric (common on AWS/Cloud instances) confirms your FE is hitting its network egress limit. * The Bottleneck: When your CN finishes merging data, it notifies the FE. The FE then has to send a large volume of metadata updates (for all 128 tablets) to the storage service. * The Result: Because your network is throttled, the "Publish" phase (the gap between

CommitTime

and

FinishTime

in your

SHOW PROC

output) is taking 30+ seconds instead of milliseconds. * The Queue: While the FE is struggling to publish one task, the other 127 tablets are waiting in the queue (

in_queue_sec

> 100s). This is why your rowset count has ballooned to 350. 2. Shared-Data vs. Shared-Nothing for RTB For Adtech RTB (Real-Time Bidding), where you have extreme update frequencies and need sub-second query latencies, here is the honest trade-off: Feature | Shared-Nothing (Local NVMe) | Shared-Data (S3 + Cache) Ingestion Limit | Very High. Compaction is decentralized. Each BE handles its own metadata locally. | Metadata Bound. The FE is a central bottleneck for transaction "Publishing." Query Latency | Lower/Stable. Data is always local. | Variable. Depends on Cache hit rate and S3 latency. Reliability | Harder. Node failure requires rebalancing data. | Easier. No data rebalancing; just spin up a new CN. Cost | Higher (EBS/NVMe is expensive). | Lower. S3 is cheap; CNs can be Spot instances. The Verdict: Shared-Nothing is traditionally more "reliable" for ultra-high throughput RTB because it avoids the central FE metadata bottleneck. However, you can achieve Shared-Nothing performance on Shared-Data by fixing the metadata path. 3. How to fix this and keep Shared-Data If you want the elasticity of Shared-Data without the 13x slowdown, do these three things: 1. Upgrade FE Instance: Your FE needs a higher network "allowance." Move to an instance type with higher baseline bandwidth (e.g., in AWS, move from a

t3/m5

to a

c6in

or

m6id

which has significantly better ENA throughput). 2. Enable "Aggregate Publish" (Requires v3.3+): This is the "killer feature" for your use case. It aggregates metadata updates for multiple tablets into a single RPC/S3 call, reducing the FE network load by up to 90%.

sql
    -- If on v3.3+, enable this to stop the FE from being a bottleneck
    ADMIN SET FRONTEND CONFIG ("enable_shared_data_aggregate_publish" = "true");

3. Batch Your Ingestion: If you are hitting the FE network limit, you are likely loading too frequently (e.g., every 1 second). Increase your batch window to 15–30 seconds. This reduces the number of transactions the FE has to process, allowing compaction to finally catch up. Recommendation: If you can upgrade to StarRocks v3.3 or v3.5, stick with Shared-Data. The "Aggregate Publish" and "Metadata Cache" improvements in those versions were designed specifically to handle high-throughput RTB-style workloads that previously required Shared- Nothing could only. previously required Shared-Data.

@Rocky Is it possible for my shared data cluster to be competitive with shared nothing in terms of ingestion throughput handling given my shared nothign cluster uses EBS and no replication for its primary key table? 10 i4i.8xlarge vs 10 r6in.8xlarge nodes. i4i.8xlarge nodes has enough NVMe to hold entire dataset in memory. I noticed shared data CPU usage higher for same workload (even with much less due to iowait). I was thinking since I save money on EBS storage and throughput, can transfer some of those saved costs to extra i4i.8xlarge machines.

Yes, a Shared-Data (Lake) cluster can be competitive with a Shared-Nothing (SN) cluster using EBS, but it requires a different optimization strategy. Because you are currently comparing it to a Shared-Nothing cluster with zero replication (the fastest possible but least reliable configuration), the Shared-Data cluster has to work significantly harder to keep up. 1. Why Shared-Data CPU usage is higher You noticed higher CPU usage on Shared-Data because "Shared-Data" is not just "Shared-Nothing with S3." It involves additional CPU-intensive tasks: * S3 Serialization/RPCs: Every write to S3 requires network stack processing and object-store protocol overhead that local NVMe does not have. * Active Cache Management: If you use

i4i

nodes for your CNs, the CPU must manage the mapping of S3 data to the local NVMe cache. * Centralized Compaction Metadata: In Shared-Nothing, each BE manages its own versions. In Shared-Data, the FE must coordinate the state of all tablets globally. This synchronization creates overhead on the CNs during the "Publish" phase. 2. The Ingestion "Parity" Strategy To make your 10-node

i4i

Shared-Data cluster competitive with a 10-node SN cluster (No Rep), you must enable Batch Publishing. This is the single most important fix for high-throughput RTB workloads. Mandatory v3.2.3+ Optimization:

sql
-- This combines multiple transaction versions into one metadata update
-- It drastically reduces FE network egress and CN "Publish" CPU wait times.
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");

3. Shared-Data vs. Shared-Nothing (No Replication) Feature | Shared-Nothing (No Rep) | Shared-Data (S3 + i4i Cache) Ingestion Performance | 1.0x (Baseline) | ~0.8x to 0.9x (Slightly slower due to S3) Query Performance | 1.0x (Baseline) | 1.1x - 1.2x (Can be faster due to Parallel Scan) Durability | Extreme Risk. 1 node failure = Data loss. | 11 9s. S3 is the source of truth. Recovery Time | Hours/Days. Must rebuild from backup. | Seconds. Just spin up a new CN. 4. Recommendation for RTB Workloads Since you want to "transfer saved storage costs to extra machines," here is the winning architecture for Adtech RTB: 1. Use

i4i.8xlarge

for CN Nodes: Even in Shared-Data mode, the local NVMe on the

i4i

is critical. It serves as the Data Cache. This gives you "Shared-Nothing" read speed while keeping data on S3. 2. Scale Horizontal, not Vertical: Because Shared-Data CPU usage is higher, it is better to have 14 nodes of

i4i.4xlarge

than 7 nodes of

i4i.8xlarge

. More nodes mean more compaction threads and more network bandwidth to S3. 3. The "Reliability" Argument: You mentioned your SN cluster has no replication. In a production RTB environment, this is dangerous. Shared-Data gives you the performance of 1-copy SN with the safety of 3-copy SN. Summary: You can reach performance parity, but only if you solve the FE network bottleneck you identified earlier. If the FE is throttled, no amount of CN nodes will help. Upgrade your FE instance first, enable

lake_enable_batch_publish_version

, and use the

i4i

NVMe for caching. References • deployment/shared_data/feature-support-shared-data.md • best_practices/primarykey_table.md • faq/shared_data_faq.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/s3.md

@Rocky How much better will 20 i4i.4lxarge nodes perform compared to 10 i4i.8xlarge. Is shared data the opposite where horizontal scaling is preferred compared to shared nothing where vertical scaling is preferred? Aren't there advantages to keeping processes on a single bigger machine

For your specific Adtech RTB workload, moving from 10

i4i.8xlarge

to 20

i4i.4xlarge

is generally the superior choice in Shared-Data mode, provided you resolve the FE bottleneck first. 1. Performance: 20 x 4xlarge vs. 10 x 8xlarge While both configurations have the same total vCPU (320) and RAM (2.56 TB), the 20-node cluster offers several "hidden" advantages: * Higher Aggregate Throughput: In AWS, two

4xlarge

instances often provide higher total baseline network bandwidth and aggregate NVMe IOPS than a single

8xlarge

. For ingestion-heavy workloads, this means more "pipes" to S3. * Parallel Compaction: With 20 nodes, you have 20 independent compaction thread pools. This is critical for clearing your 350-rowset backlog. Each node can focus on a smaller subset of tablets, reducing the "compaction queue" wait time (

in_queue_sec

). * Query Parallelism: StarRocks' MPP engine can split a query into more parallel fragments. For RTB queries that scan large amounts of data, having more nodes increases the number of concurrent "scanners" hitting the NVMe cache. 2. Scaling Philosophy: Shared-Nothing vs. Shared-Data You are correct that the scaling preference flips between the two architectures: * Shared-Nothing (Prefers Vertical): Adding nodes requires data rebalancing (shuffling TBs of data across the network). To avoid this operational headache, users often scale vertically (bigger nodes) until they hit a wall. * Shared-Data (Prefers Horizontal): There is zero cost to add/remove nodes. Since storage is central (S3), a new CN node can start working immediately by just pulling metadata. This elasticity makes horizontal scaling the default "best practice." 3. Advantages of "Single Big Machines" (The Trade-offs) Vertical scaling (bigger nodes) isn't without merit. Its advantages include: * Reduced Network Shuffle: In complex JOIN-heavy queries, data must be "shuffled" between nodes. If more data resides on a single large node, more processing happens via memory-speed IPC rather than network-speed RPC. * Lower FE Pressure: Every CN node maintains a heartbeat and metadata connection to the FE. 20 nodes create 2x the connection overhead of 10 nodes. In your case, where the FE is already hitting network limits, adding more nodes could actually worsen the problem unless you upgrade the FE or enable

lake_enable_batch_publish_version

. * Memory-Intensive Queries: If you have a single query that needs a 200GB hash table, an

8xlarge

(256GB) can handle it, whereas a

4xlarge

(128GB) might OOM (Out of Memory). 4. Recommendation for your Cluster Given your current situation (High rowset count + FE Network Throttling): 1. Upgrade the FE first: If you move to 20 nodes without fixing the FE, the FE will likely crash or throttle even harder because it has to manage 2x the connections and 2x the "Publish" notifications. 2. Go Horizontal (20 nodes): For RTB, throughput and compaction speed are king. 20 nodes give you more total NVMe cache bandwidth and more compaction concurrency. 3. Enable Batch Publish: This is non-negotiable for your scale. It will allow those 20 nodes to "bundle" their updates, saving your FE's network allowance. Final Verdict: Use 20 nodes for the elasticity and compaction throughput, but only after you give the FE a bigger "pipe" to handle the extra metadata traffic. References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • knowledge/usage/load/shared_data_batch_ingestion.md • deployment/plan_cluster.md References

@Rocky

max by(aws_name) (starrocks_be_max_tablet_rowset_num{starrocks_cluster="core-services-sd"})

What is the equivalent of this metric in shared data architecture? I want to track max rowset indicated by this query

SELECT c.table_name, COUNT(*) tablets, SUM(t.num_rowset) total_rowsets,          MAX(t.num_rowset) max_rowsets_per_tablet   FROM information_schema.be_tablets t   JOIN information_schema.tables_config c ON t.table_id = c.table_id   WHERE c.table_name = 'requests'   GROUP BY c.table_name;

In Shared-Data (Lake) architecture, the equivalent of the "Max Rowset" metric is the Compaction Score (CS). While CN nodes still report

starrocks_be_max_tablet_rowset_num

(because they run the same underlying storage engine code), the StarRocks FE manages Lake tablets differently. The Compaction Score is the official indicator of the health of a partition's versions in Shared-Data mode. 1. Prometheus Equivalent To track the equivalent of

starrocks_be_max_tablet_rowset_num

in a Shared-Data cluster, you should use the FE-side metric for Compaction Score: * Metric:

starrocks_fe_tablet_max_compaction_score

* Prometheus Query:

promql
    max by (aws_name) (starrocks_fe_tablet_max_compaction_score{starrocks_cluster="core-services-sd"})

*Note: A Compaction Score of 100 is the "warning" threshold where ingestion begins to slow down, and 2000 is the hard limit where writes are rejected.* ────────── 2. SQL Tracking Equivalent The query you provided against

information_schema.be_tablets

works on CNs, but it only reflects the state of tablets currently "active" or "cached" on those specific nodes. For a true cluster-wide view of Shared-Data health, use the

information_schema.partitions_meta

table (available in v3.1.9+ and v3.2.4+). This table is the "Source of Truth" for Lake compaction:

sql
SELECT 
    db_name,
    table_name, 
    COUNT(partition_id) as partitions,
    MAX(max_cs) as max_rowset_score,
    AVG(avg_cs) as avg_rowset_score
FROM information_schema.partitions_meta 
WHERE table_name = 'requests'
GROUP BY db_name, table_name;

Why use Compaction Score instead of Rowset Count? In Shared-Nothing, rowset count is a simple integer. In Shared-Data: * Compaction Score is a weighted calculation. For example, if you have many small rowsets, the score increases faster to trigger compaction. * FE Visibility: The FE uses the

MAX_CS

to decide which partitions to prioritize for its global compaction scheduler (

lake_compaction_max_tasks

). * Sustained Throughput: If your

MAX_CS

is climbing while your FE network bandwidth is throttled (as you found), it confirms the "Publish" phase is failing to keep up with the data being merged by the CNs. Recommendation: Monitor

starrocks_fe_tablet_max_compaction_score

. If it consistently exceeds 100, your compaction is falling behind. If it consistently exceeds 100, your compaction is lagging. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rpcl

The metric

lake_vacuum_del_file_fails

tracks failures when the system attempts to physically delete old rowset files or metadata files from S3 that are no longer needed (after compaction or

DROP TABLE FORCE

). A rate of 2–3 failures per second is relatively high and typically points to one of the following issues: 1. Check CN Logs (Primary Diagnostic) The most detailed information will be in the

<http://cn.INFO|cn.INFO>

or

cn.WARNING

logs on your compute nodes. Search for logs from the

vacuum.cpp

component:

bash
grep "vacuum.cpp" <http://cn.INFO|cn.INFO> | grep "Fail to delete"

Common error patterns to look for: *

Access Denied

: Your IAM role or S3 bucket policy is missing the

s3:DeleteObject

or

s3:DeleteObjectVersion

permission. *

NoSuchKey

: This happens if a file was already deleted (e.g., by an S3 Lifecycle Policy or a manual cleanup tool) before StarRocks tried to vacuum it. This is usually harmless but noisy. *

403 Forbidden

: Authentication issues or S3 Object Lock/Retention settings preventing deletion. *

SlowDown / 503

: AWS S3 throttling. If you have a massive backlog of files, the vacuum threads might be hitting S3's rate limits for DELETE requests. 2. Verify IAM Permissions In Shared-Data mode, the CN nodes must have explicit permissions to delete. Ensure your IAM policy includes:

json
{
    "Effect": "Allow",
    "Action": [
        "s3:DeleteObject",
        "s3:DeleteObjectVersion"
    ],
    "Resource": "arn:aws:s3:::your-bucket-name/*"
}

3. Check for "Object Storage Inflation" & Retention If vacuuming fails, your S3 bucket size will grow even if

SHOW DATA

in StarRocks looks stable. * Grace Period: StarRocks won't delete files immediately to allow long-running queries to finish. Check your current grace period:

sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';

(Default is usually 5 minutes. If queries fail with "meta does not exist," you may have increased this, which can increase the vacuum workload.) 4. S3 Lifecycle Policy Conflict If you have an S3 Lifecycle Policy configured on the bucket (e.g., "Delete objects after 7 days"), it can conflict with StarRocks' internal vacuuming. StarRocks expects to be the only entity managing the lifecycle of its data files. If S3 deletes a file first, StarRocks will record a "failure" because it couldn't find the file to delete it itself. 5. Potential Fix: Increase Vacuum Concurrency If the failures are due to timeouts or backlogs, you can check if the vacuum worker is overwhelmed. Note that if you are seeing "failures" rather than "latency," it is usually a permission/existence issue, not a thread pool issue. * FE Config:

lake_vacuum_parallel_partitions_num

(Controls how many partitions are vacuumed in parallel). Recommendation: If you see

NoSuchKey

in the logs, it means the files are already gone—you can likely ignore the metric or check for external lifecycle policies. If you see

Access Denied

, update your IAM policy immediately to prevent S3 storage costs from spiraling. References • faq/shared_data_faq.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • deployment/shared_data/s3.md • sql-reference/System_variable.md References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md

@Rocky Jun 24 153532 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153532.267764 22351095789120 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=AMQHSBKFSG4AME0W][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 100ms Jun 24 153532 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153532.327923 22327179806272 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=Q7SWQEJ94Z369JQQ][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 800ms Jun 24 153533 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153533.631241 22321364465216 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=HRK36HDTGMNYC97V][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 800ms Jun 24 153534 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153534.878316 22361571001920 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=26A9AC6TTSMGA3QR][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 800ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.059795 22143150585408 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=26ABEDMCDQ991A85][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 1600ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.370174 22345221641792 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8V7JJKVK1B1N7T][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 100ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.506642 22351079511616 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8TQ4V8ER6PTTVC][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 200ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.849092 22118343374400 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8N8XDNWZR2CZXY][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 400ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.851003 22081680963136 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8VBYNVVX58XWHF][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 200ms Jun 24 153535 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153535.991550 22358161593920 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8MT5M6G5Y65VNN][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 800ms Jun 24 153536 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153536.002370 22222227895872 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8RKCK1VKETVAGE][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 1600ms Jun 24 153536 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153536.268773 22360765457984 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=BW8MKWXX1179HAAJ][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 400ms Jun 24 153536 ip-172-30-75-85.ec2.internal starrocks-cn[97627]: W20260624 153536.780208 22351089485376 vacuum.cpp:128] Fail to delete: Resource busy: starlet err [RequestID=0PB7Z5NNSZX6MT3S][StatusCode=503]Delete objects error: Please reduce your request rate. will retry after 800ms These were the errors.

The logs indicate that your StarRocks cluster is hitting the S3 Rate Limit (503 SlowDown) for DELETE requests. AWS S3 has a throughput limit of 3,500 DELETE requests per second per prefix. The

vacuum.cpp

component in your logs is part of StarRocks' AutoVacuum process, which physically deletes old files (garbage) after compaction. When you have a high compaction backlog (like the 350 rowsets you mentioned earlier) or many nodes, the cluster can easily exceed this limit. Diagnosis * The Error:

Resource busy: starlet err ... [StatusCode=503] Delete objects error: Please reduce your request rate.

* The Cause: Your cluster is attempting to delete garbage files faster than S3 allows. Even though StarRocks uses the

DeleteObjects

(multi-object delete) API, the aggregate volume of deletions across all partitions is triggering throttling. Recommended Fixes 1. Throttle the AutoVacuum Process (Immediate Fix) You can slow down the garbage collector by reducing the number of partitions processed in parallel. Run this on your leader FE:

sql
-- Reduce the number of partitions vacuumed simultaneously (Default is 8)
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_parallel_partitions" = "2");

-- Increase the sleep interval between vacuum cycles for a partition (Default is 180s)
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_partition_naptime_seconds" = "600");

2. Increase the Retention Grace Period Increasing the time StarRocks waits before cleaning up old versions will spread the deletion workload over a longer period, reducing the "burst" rate of S3 deletes.

sql
-- Increase retention from the default 30 minutes to 2 hours
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");

3. Why this matters While the logs show StarRocks is automatically retrying (e.g.,

will retry after 800ms

), persistent 503 errors cause: * Metric Spikes: They drive up the

lake_vacuum_del_file_fails

metric you observed. * Storage Costs: If deletions are throttled and delayed, you pay for that "garbage" storage in S3 longer. * CPU Overhead: Constant retries consume CPU cycles on the CN nodes. Final Advice: Reduce

lake_autovacuum_parallel_partitions

to 2 today. Once your compaction backlog is cleared and your rowset count drops, you can slowly increase it back to the default if you find garbage is accumulating too slowly. References • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/compaction.md • deployment/shared_data/s3.md References • administration/management/BE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md