http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> ```PRIMARY KEY(`received`, `auction...
# questions-and-troubleshooting
m
@Rocky
Copy code
PRIMARY KEY(`received`, `auction_id`, `imp_key`)
PARTITION BY date_trunc('hour', received)
DISTRIBUTED BY HASH(`auction_id`) BUCKETS 128
ORDER BY(`received`, `auction_id`)
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"labels.location" = "group:requests",
"partition_live_number" = "24",
"replicated_storage" = "true",
"replication_num" = "2"
);
This is my primary key table definition.
Copy code
BackendId,IP,HeartbeatPort,BePort,HttpPort,BrpcPort,LastStartTime,LastHeartbeat,Alive,SystemDecommissioned,ClusterDecommissioned,TabletNum,DataUsedCapacity,AvailCapacity,TotalCapacity,UsedPct,MaxDiskUsedPct,ErrMsg,Version,Status,DataTotalCapacity,DataUsedPct,CpuCores,MemLimit,NumRunningQueries,MemUsedPct,CpuUsedPct,DataCacheMetrics,Location,StatusCode
203701,172.30.65.182,9050,9060,8040,8060,2026-05-07 18:24:09,2026-05-13 19:58:29,true,false,false,873,126.229 GB,3.961 TB,4.802 TB,17.51 %,24.76 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:58:18""}",4.085 TB,3.02 %,32,211.808GB,1,33.67 %,92.6 %,"Status: Normal, DiskUsage: 0B/680GB, MemUsage: 42.3GB/42.3GB","",OK
171908,172.30.65.224,9050,9060,8040,8060,2026-05-07 18:52:35,2026-05-13 19:58:29,true,false,false,887,124.943 GB,3.766 TB,4.802 TB,21.57 %,29.43 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:57:45""}",3.888 TB,3.14 %,32,211.808GB,2,38.04 %,82.5 %,"Status: Normal, DiskUsage: 0B/780GB, MemUsage: 42.3GB/42.3GB",group:requests,OK
110783,172.30.68.175,9050,9060,8040,8060,2026-05-07 18:52:20,2026-05-13 19:58:29,true,false,false,870,126.693 GB,3.964 TB,4.802 TB,17.44 %,20.08 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:58:00""}",4.088 TB,3.03 %,32,211.808GB,2,35.84 %,96.1 %,"Status: Normal, DiskUsage: 0B/1TB, MemUsage: 42.3GB/42.3GB",group:requests,OK
95876,172.30.75.179,9050,9060,8040,8060,2026-05-07 18:52:35,2026-05-13 19:58:29,true,false,false,884,124.977 GB,3.981 TB,4.802 TB,17.09 %,19.27 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:57:45""}",4.104 TB,2.97 %,32,211.808GB,2,35.20 %,57.8 %,"Status: Normal, DiskUsage: 0B/990GB, MemUsage: 42.3GB/42.3GB",group:requests,OK
203697,172.30.77.163,9050,9060,8040,8060,2026-05-07 18:52:25,2026-05-13 19:58:29,true,false,false,877,126.587 GB,3.942 TB,4.802 TB,17.90 %,21.77 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:58:02""}",4.066 TB,3.04 %,32,211.808GB,1,34.19 %,41.2 %,"Status: Normal, DiskUsage: 0B/970GB, MemUsage: 42.3GB/42.3GB",group:requests,OK
203699,172.30.78.249,9050,9060,8040,8060,2026-05-07 18:52:25,2026-05-13 19:58:29,true,false,false,875,126.745 GB,3.410 TB,4.802 TB,28.98 %,30.18 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:58:27""}",3.534 TB,3.50 %,32,211.808GB,1,43.00 %,86.9 %,"Status: Normal, DiskUsage: 0B/1TB, MemUsage: 42.3GB/42.3GB",group:requests,OK
203700,172.30.81.176,9050,9060,8040,8060,2026-05-07 18:52:25,2026-05-13 19:58:29,true,false,false,874,126.668 GB,3.542 TB,4.802 TB,26.24 %,33.22 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:58:29""}",3.666 TB,3.37 %,32,211.808GB,2,45.78 %,92.9 %,"Status: Normal, DiskUsage: 0B/1020GB, MemUsage: 42.3GB/42.3GB","",OK
11001,172.30.87.186,9050,9060,8040,8060,2026-05-07 18:52:30,2026-05-13 19:58:29,true,false,false,875,125.235 GB,3.476 TB,4.802 TB,27.61 %,31.89 %,"",4.0.7-b75f536,"{""lastSuccessReportTabletsTime"":""2026-05-13 19:57:44""}",3.599 TB,3.40 %,32,211.808GB,2,43.18 %,79.9 %,"Status: Normal, DiskUsage: 0B/1TB, MemUsage: 42.3GB/42.3GB",group:requests,OK
And these are my backends. You can see that the location column indicates only 6 BE nodes has the group;requests label. However, clearly all the nodes are being used to house the tablets for BE data. In show proc '/statistic'. it also indicates 1600/6400 total tablets (including replicas) are unhealthy, indicating the FE is aware that some tablets are where they do not belong. balance stats shows intra-node tablet distribution and label-aware location balance is false. Yet, data is still being distributed to the BE nodes without labels. The table was created from scratch with the label specified in the properties already. What is going wrong
r
It appears that while your table is correctly configured with the 
labels.location
property and your Backends (BEs) are labeled appropriately, the StarRocks Frontend (FE) scheduler failed to respect these labels during the initial tablet placement. Furthermore, the system currently recognizes the imbalance but is not correcting it. Based on your provided data, here is an analysis of what is happening: 1. The Initial Placement Ignored the Labels The fact that your tablet distribution is nearly uniform across all 8 nodes (~870 tablets per node) despite only 6 nodes having the 
group:requests
label indicates that the FE's initial allocation logic ignored the 
labels.location
property during 
CREATE TABLE
. * Likely Cause: In StarRocks v4.0.x, there can be edge cases where the new 
labels.location
property is bypassed by the Primary Key table's specific replication logic, especially when 
replicated_storage = true
(Single Leader Replication) is enabled. Single Leader Replication uses a different "Primary/Secondary" management path that may not have fully integrated with the Backend Labeling feature at the point of initial bucket creation. 2. The FE "Knows" it is Wrong The stats you provided confirm the FE is aware of the issue: * 1600 Unhealthy Tablets: With ~6400 total replicas and 8 nodes, roughly 2 nodes worth of replicas (2/8 * 6400 = 1600) are sitting on unlabeled nodes. These are flagged as "unhealthy" because they violate the table's location requirement. * 
label-aware location balance
is false: In 
SHOW PROC '/cluster_balance/balance_stat'
, the "Balanced" column being 
false
means the balancer recognizes that the cluster does not meet the label-based distribution criteria. 3. Why the Balancer is Not Moving Data Even though the FE recognizes the tablets are in the wrong place, it isn't moving them. This is likely due to: * High BE Resource Load: Your 
SHOW BACKENDS
output shows several nodes with CPU usage above 90% (e.g., 172.30.68.175 at 96.1%). The StarRocks Tablet Scheduler often throttles or pauses tablet migration/balancing if the cluster is under heavy load to avoid impacting query performance. * Balancer Settings: Check if balancing is globally disabled or restricted.
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE '%balance%';
    -- Specifically check: tablet_sched_disable_balance
Recommended Actions 1. Lower the Balance Threshold (Temporary): If the cluster load is high, you can try to make the balancer more aggressive by adjusting the load score threshold or checking if migration is being rejected due to CPU.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_balance_load_score_threshold" = "0.05");
2. Verify Tablet Health Details: Check the specific reason these tablets are "unhealthy" to confirm they are indeed marked as misplaced:
Copy code
sql
    SHOW TABLET <one_of_unhealthy_ids>;
    -- Look for "IsBad" or check "ADMIN SHOW REPLICA STATUS FROM <table_name>"
3. Check for Scheduling Tasks: See if there are any active but stuck "CLONE" tasks:
Copy code
sql
    SHOW PROC '/cluster_balance/running_tablets';
    SHOW PROC '/cluster_balance/sched_stat';
4. Workaround (Re-triggering): Since you mentioned this was created from scratch, you can try to "nudge" the metadata by setting the property again, which sometimes triggers a re-scan of the placement:
Copy code
sql
    ALTER TABLE <table_name> SET ("labels.location" = "group:requests");
If the CPU load on your BEs remains at 90%+, the background balancer is unlikely to start the massive 1600-tablet migration required to fix this. You may need to wait for a period of lower activity or manually prioritize the repair of those tablets.
Note: If the behavior persists even after load drops, it may be a bug specific to the interaction of 
replicated_storage
and 
labels.location
in version 4.0.7. I recommend checking the StarRocks GitHub Issues for similar reports in-progress fixes related to "Backend labels" and "Primary Key tables with Single Leader Replication." References • administration/management/resource_management/be_label.md • using_starrocks/Colocate_join.md • administration/management/resource_management/Replica.md
m
@Rocky sdm ssh starrocks-core-services-fe-1.us-east-1a MySQL [(none)]> admin show frontend config like 'disable'; Empty set (0.002 sec) MySQL [(none)]> admin show frontend config like 'balance'; Empty set (0.001 sec) MySQL [(none)]> admin show frontend config like 'disable'; Empty set (0.001 sec) MySQL [(none)]> ADMIN SHOW FRONTEND CONFIG LIKE '%disable%'; +-----------------------------------------------+----------------------------------+-------+---------+-----------+---------------------------------------------------------+ | Key | AliasNames | Value | Type | IsMutable | Comment | +-----------------------------------------------+----------------------------------+-------+---------+-----------+---------------------------------------------------------+ | disable_hadoop_load | [] | false | boolean | true | | | disable_load_job | [] | false | boolean | true | | | lake_compaction_disable_ids | [lake_compaction_disable_tables] | | String | true | disable table or partition compaction, format:'id1;id2' | | tablet_sched_disable_balance | [disable_balance] | false | boolean | true | | | tablet_sched_disable_colocate_balance | [disable_colocate_balance] | false | boolean | true | | | tablet_sched_disable_colocate_overall_balance | [] | true | boolean | true | | +-----------------------------------------------+----------------------------------+-------+---------+-----------+---------------------------------------------------------+ 6 rows in set (0.007 sec) Any these affecting labels?
r
The behavior you are seeing—where tablets are evenly distributed across all nodes despite the 
labels.location
property—indicates that while the FE is aware of the "unhealthy" (misplaced) status of your tablets, the background scheduler is being prevented from moving them. Based on your configuration and status, there are three primary reasons why this is happening: 1. 
tablet_sched_disable_colocate_overall_balance
= 
true
 This is likely the smoking gun. In StarRocks, Primary Key tables with 
replicated_storage = true
 (Single Leader Replication) often use internal group-based scheduling logic similar to Colocate Groups to manage tablet leaders and followers. * The property 
tablet_sched_disable_colocate_overall_balance
controls whether the cluster can move these "groups" of tablets across different BE nodes to achieve balance. * Because it is set to 
true
, the FE is forbidden from migrating these groups to new nodes, even if they currently violate the 
labels.location
constraint. * Fix: Set this to 
false
to allow the balancer to relocate the groups to the correctly labeled nodes:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_colocate_overall_balance" = "false");
2. High CPU Load Throttling Your 
SHOW BACKENDS
output shows several nodes with CPU usage above 90% (e.g., node 110783 at 96.1%). * The StarRocks Tablet Scheduler (which handles both 
REPAIR
and 
BALANCE
tasks) is designed to be conservative. If it detects high CPU or I/O load on the backends, it will skip or postpone migration tasks to avoid impacting active query performance. * As long as your nodes are pinned at 90%+, the FE will likely keep the 1600 tablets in their "unhealthy" state rather than starting a massive data migration that would consume further resources. 3. Initial Creation Logic You mentioned the table was created with labels already in the properties. In some scenarios (especially in versions earlier than 4.1), if the FE cannot find a "perfect" placement for all replicas immediately due to timing or metadata synchronization during a high-load 
CREATE TABLE
event, it may fall back to a random distribution to ensure the table is at least created. Once created, it relies on the background balancer to "repair" the placement later. Recommended Next Steps 1. Check Scheduler Status: Look at the reason why tasks might be skipped:
Copy code
sql
    SHOW PROC '/cluster_balance/sched_stat';
    -- Check if 'pending_tasks' or 'running_tasks' are stuck.
2. Verify Tablet Misplacement: Pick one of the unhealthy tablets and check its status. You should see a reason like 
MISPLACED
or `REPLICA_MISSING_LOCATION`:
Copy code
sql
    ADMIN SHOW REPLICA STATUS FROM your_table_name;
3. Adjust Balancer Aggressiveness (Optional): If you have a maintenance window and want to force the migration despite the load:
Copy code
sql
    -- Reduce the threshold to make it more sensitive
    ADMIN SET FRONTEND CONFIG ("tablet_sched_balance_load_score_threshold" = "0.01");
    -- Increase concurrent tasks (default is usually 500)
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "1000");
Summary: The most likely culprit is 
tablet_sched_disable_colocate_overall_balance
. Enable it, and as CPU load stabilizes, you should see the 
TabletNum
on your two unlabeled nodes begin to drop toward zero. References • administration/management/FE_configuration.md • administration/management/resource_management/be_label.md • administration/management/resource_management/Replica.md References • administration/management/FE_configuration.md
m
@Rocky Is replication 2 causing location labels to not work? Maybe BE replicating to node without label? On different note, is replication linear to CPU workload and latency of updates and insertions? I noticed cpu busy util doubled and rowset commit apply latency and transaction publish latency also doubled
r
The short answer is: No, 
replication_num = 2
is not fundamentally "broken" with labels, but it creates a strict placement scenario that, when combined with your high CPU load, is likely causing the FE to prioritize data safety over your location constraints. Here is the breakdown of why you are seeing these issues: 1. Replication 2 and Location Labels While StarRocks supports 
replication_num = 2
, it is generally discouraged for production because it lacks "majority" fault tolerance. However, for labels: * The "Safety First" Rule: StarRocks documentation states that if the FE cannot strictly satisfy a placement rule (like labels) while maintaining the requested number of replicas, it will prioritize replicas over labels. * The Problem: Because you have 1600 "unhealthy" tablets, the FE likely attempted to place a replica on a labeled node, but if that node was unresponsive or too busy during the 
CREATE TABLE
process, the FE "fell back" to any available node (the unlabeled ones) to ensure you didn't have missing data. * Replication to Unlabeled Nodes: Once a tablet is placed on an unlabeled node, the system marks it as "unhealthy" (misplaced). However, as discovered in the previous turn, your 
tablet_sched_disable_colocate_overall_balance
is set to 
true
, which explicitly prevents the FE from moving these tablets back to the correct labeled nodes. 2. Is Replication "Linear" to Workload? Yes, in a Primary Key table, the relationship is very close to linear, even with 
replicated_storage = true
. Here is why your CPU and Latency doubled: * Rowset Commit/Apply Latency: With 
replicated_storage = true
(Single Leader Replication), the Primary node handles the heavy lifting of calculating the "Apply" (generating delete vectors and updating the PK index). However, it must then synchronize this state to the Secondary. The transaction cannot "Publish" until the secondary has acknowledged the data. Adding a second replica adds a network round-trip and a disk I/O wait for every single write. * Transaction Publish Latency: The "Publish" phase is a two-step commit. With 2 replicas, the FE must wait for acknowledgments from both nodes (or at least a quorum, which for 2 replicas is often both depending on version settings) before the data is visible. * CPU Workload: * The Primary Node: Does index lookups, version management, and data encoding. * The Secondary Node: While it doesn't re-calculate the index (it receives the result), it still performs the I/O to write the data and delete vectors to disk and manages its own memory for the rowset. * The Result: If you previously had a table with 1 replica (or were comparing to a different model), doubling the replica count nearly doubles the total cluster-wide CPU cycles spent on "Apply" and "Compaction" tasks. Since your nodes are already at 90%+ CPU, this overhead is compounding, leading to the doubling of latency you're seeing. Summary of What's Happening 1. Placement: The FE put data on the wrong nodes because of resource pressure/timing during creation. 2. Stagnation: Your FE configuration (
tablet_sched_disable_colocate_overall_balance = true
) is preventing the cluster from fixing itself. 3. Performance: The CPU is saturated (90%+), causing the background "Apply" and "Publish" tasks to queue up, which manifests as doubled latency. Recommendation 1. Flip the config: 
ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_colocate_overall_balance" = "false");
2. Reduce Load: If possible, reduce the ingestion rate temporarily. StarRocks will not prioritize moving 1600 tablets while the BEs are struggling at 96% CPU. It will prioritize query stability. 3. Monitor Compaction: Use 
SHOW PROC '/statistic'
to see if "Compaction Score" is high. If so, your CPU is being eaten by background merges, which is common after a large data ingestion with multiple-replica ingestion. References
m
@Rocky I noticed location labels were only respected when I decreased replication to 1. Replication num 2 and 3 did not work
r
Your observation is correct and points to a specific interaction between the Single Leader Replication logic and the Backend Labeling feature in StarRocks. The reason labels are respected for 
replication_num = 1
but ignored for 
2
or 
3
is rooted in how StarRocks handles the initial placement of Primary Key tables when multi-node replication is enabled. 1. Replication Groups vs. Tablet Placement When you set 
replicated_storage = "true"
and 
replication_num > 1
, StarRocks does not just place individual tablets; it creates an internal Replication Group (similar to a Colocate Group). This group ensures that the "Primary" and "Secondary" roles for your tablets are distributed and balanced across the cluster to optimize the Single Leader performance. * For 
replication_num = 1
: There is no replication to coordinate. The FE treats this as a standard tablet placement task. The standard placement logic is fully "label-aware," so it successfully restricts the tablets to the 6 labeled nodes. * For 
replication_num > 1
: The FE uses the Replication Group placement logic. In some versions (including v4.0.x/3.x branches), the group-based scheduler focuses on balancing the "leaders" and "followers" across the entire cluster's available resources. Historically, this group placement logic has had gaps in respecting the 
labels.location
property during the initial 
CREATE TABLE
layout, often falling back to a cluster-wide distribution. 2. Why the FE "Knows" it is Wrong (The 1600 Unhealthy Tablets) The fact that you see 1600 unhealthy tablets is actually "good" news—it means the FE's checker is label-aware. 1. The Placement Logic (the "Builder") incorrectly put data on all nodes because it was focused on group balancing. 2. The Health Checker (the "Auditor") looks at the finished table, sees the 
labels.location
property, and marks the 1600 tablets on the unlabeled nodes as "Misplaced" (unhealthy). 3. Why it isn't fixing itself Under normal circumstances, the FE would immediately start "Clone" tasks to move those 1600 tablets to the labeled nodes. However, two things are blocking this: * 
tablet_sched_disable_colocate_overall_balance = true
: As noted in your config, this prevents the FE from moving Replication/Colocate groups between nodes. Since your PK table is managed as a group, the balancer is "locked." * CPU Satiation (90%+): Even if you enabled the balancer, StarRocks frequently pauses background migrations if BEs are under heavy CPU load to prevent query degradation. Recommendations 1. Enable Group Balancing: To allow the FE to move the misplaced tablets to the labeled nodes, you must enable group balancing:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_disable_colocate_overall_balance" = "false");
2. Verify Group Status: Check if your table is part of an internal group and if that group is marked as "Unbalanced":
Copy code
sql
    SHOW PROC '/colocate_group';
(Look for a group ID associated with your table. If it says 
IsStable: false
, it is trying to migrate). 3. Use a lower 
replication_num
first?: If you are in a "bootstrap" phase, creating the table with 
replication_num = 1
, letting it settle on the labeled nodes, and then performing 
ALTER TABLE ... SET ("replication_num" = "2")
can sometimes force the FE to pick "Secondary" nodes only from the allowed labeled set, rather than doing a "random" global layout at the start. Regarding the CPU/Latency Question Yes, the doubling of CPU and latency is the direct result of 
replication_num = 2
. * CPU: Even with 
replicated_storage = true
, the second node must still perform the I/O, write the data, and manage the rowset metadata. This effectively doubles the cluster-wide "write" effort.
* Latency: The "Transaction Publish" cannot finish until the Primary has successfully pushed the data to the Secondary. In a saturated CPU environment (96%), these network/ack roundtrips take significantly longer, explaining the "doubled" latency. References • administration/management/resource_management/be_label.md • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • table_design/table_types/primary_key_table.md
2 Views
Open in Slack
PreviousNext
Powered by