@Rocky My EBS volume performance throughput exceeded sporadically for 4-12ms. THis is pretty common pattern across the BEs and FEs. Is this impacting performance much? I already increasing throughput from 500 to 700. It was previously exceeding by up to 35ms

Exceeding EBS throughput limits leads to AWS throttling your I/O, which manifests as those 4–12ms latency spikes. In a distributed system like StarRocks, the impact differs significantly between the Frontend (FE) and Backend (BE). 1. Impact on Frontend (FE) The FE is generally more sensitive to disk latency than the BE because it uses BDB JE (Berkeley DB Java Edition) to persist metadata and transaction logs (Edit Logs). * Metadata Consistency: Every schema change, permission update, or load transaction requires a metadata write. If EBS throttles these writes, the Leader FE may experience "jitter" during query planning or transaction commits. * Leader Stability: High I/O latency on the FE can occasionally interfere with the heartbeat or log replication between FEs. If the latency spikes were to exceed the BDB JE timeout (usually several seconds), it could trigger a Leader re-election, causing a brief cluster-wide service interruption. * Verdict: 4–12ms is unlikely to crash the FE, but it will cause unpredictable "p99" latencies for DDL and query planning. 2. Impact on Backend (BE) The BE uses disk for storing data segments and for "spilling" intermediate results during heavy queries. * Tail Latency: StarRocks queries are parallelized across many BEs. If one BE is throttled by 12ms on every I/O request, the entire query must wait for that slow node to finish. This creates a "long tail" where some queries take significantly longer than others for no obvious reason. * Scan Throughput: If your throughput was 500 MB/s and is now 700 MB/s, but you are still exceeding it, your data scans are physically bottlenecked. You aren't getting the full performance of the CPU because it's waiting for the disk (I/O Wait). * Compaction: Background compaction tasks are I/O intensive. If they are throttled, "version piling" (too many segment files) can occur, which further degrades read performance over time. 3. Monitoring and Mitigation To determine if this is "impacting performance much," check these StarRocks metrics (usually available via Prometheus/Grafana): *

starrocks_be_disks_io_util

: If this is consistently above 80–90%, your disks are a bottleneck. *

starrocks_fe_edit_log_write_latency

: Check if this spikes during your 4–12ms EBS events. *

starrocks_be_query_scan_bytes_per_second

: Compare this against your EBS limit (700 MB/s). If the sum of scan + compaction throughput hits 700, you are being capped. Recommendations 1. FE Storage: If the spikes continue, consider moving FEs to gp3 with higher provisioned IOPS/throughput or even io2 if metadata consistency is critical. Since FE data volumes are small, the cost for high-performance FE volumes is usually low. 2. BE Storage: If you are hitting 700 MB/s per volume, you are in a high-throughput tier. * If you use gp3, ensure you have provisioned enough IOPS (not just throughput), as they are linked in AWS. * Check if you can distribute the load by adding more EBS volumes per BE node (StarRocks can use multiple

storage_root_path

entries) rather than one giant volume. This increases the total throughput ceiling for the node. 3. Data Cache: If using shared-data (Compute Nodes), ensure your Data Cache is large enough to keep "hot" data in memory, reducing the need to hit the EBS volumes during repeat queries. References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • data_source/data_cache_observe.md

@Rocky Is the cost of increasing replication num to 2 greater than the benefit (from no replication) compared to using replication num = 3? The main issue I want to avoid is version errors during tablet rebalancing. transactions get stuck because of

wait for publishing partition 7460587 version 369. self version: 362. table 7211473

and messages similar to these. It happened when I assigned location labels to my BE nodes. Also, disk saturation issue at one point caused unhealthytablets due to version count errors too. Will just 2 replication solve it? Does it create more issues?

Choosing

replication_num = 2

instead of

3

is generally not recommended for StarRocks, especially if you are already seeing version lag and publishing errors. Here is why

2

might actually make your "wait for publishing" issues worse compared to

3

. 1. The Quorum Problem (Write Availability) StarRocks uses a Majority Quorum for data loading. * Replication = 3: The majority is 2. If one BE is slow (due to your EBS throughput spikes) or down, the transaction can still succeed and publish as soon as the other 2 nodes finish. * Replication = 2: The majority is 2. This means both replicas must succeed. If one BE is throttled by EBS or busy rebalancing, the entire load transaction must wait for it. By using only 2 replicas, you lose the "fault tolerance" for performance jitter. Any single node's 12ms spike or disk saturation becomes a bottleneck for the entire partition's version publishing. 2. "Wait for Publishing" and Version Errors The error

wait for publishing partition ... self version: 362. table 369

means the Frontend (FE) is refusing to commit a new transaction because too many previous ones are still in the "COMMITTED" state but haven't been "PUBLISHED" to enough BEs. * During Rebalancing: When you assigned location labels, StarRocks started moving tablets to satisfy the new rules. This creates massive I/O and network background load. * Disk Saturation: If a BE node's disk is saturated, it cannot finish the I/O for the background "Publish" task quickly. * Why 2 replicas won't help: With 2 replicas, the FE is strictly waiting for that specific saturated node to respond before it can move the version forward. With 3 replicas, if one node is saturated, the FE can often move on once the other two healthy nodes report back. 3. Impact of Location Labels Location labels (introduced in v3.2.8+) are powerful but trigger significant tablet migration. * If you have

replication_num = 2

and use labels to split data across two zones, each zone has exactly one copy. If one zone’s EBS performance is degraded, 100% of your write transactions hitting those tablets will be delayed. * With

replication_num = 3

, StarRocks can place 2 replicas in one zone and 1 in another (or vice versa). You still have a "spare" replica to maintain quorum if one node or zone hits an I/O limit. Summary: 2 vs. 3 Replicas Feature | Replication = 2 | Replication = 3 Data Safety | Protects against 1 node failure. | Protects against 1 node failure (more reliably). Write Quorum | 2 of 2 (No tolerance for slowness). | 2 of 3 (Can tolerate 1 slow node). Rebalancing | Highly sensitive to disk/network pressure. | More resilient; "spare" replica helps. Storage Cost | 2x | 3x Stability | Risky for production workloads. | Recommended standard. Recommendation 1. Stick to 3 Replicas: It is the only way to "mask" the EBS throughput spikes you are seeing. It allows the system to remain performant even when one BE is temporarily throttled. 2. Throttle Rebalancing: If the version errors started when you added labels, the rebalancing is likely too aggressive. Lower these FE configs to reduce disk pressure: *

tablet_sched_max_balancing_tablets

(Default is 500, try lowering to 100). *

tablet_sched_max_scheduling_tablets

(Default is 2000, try lowering to 500). 3. Monitor Compaction: Check

starrocks_be_tablet_cumulative_compaction_score

. If this is high (>100), the "version errors" are because the BE can't merge files fast enough due to your EBS limits. Verdict: Increasing replication from 0/1 to 2 is better than nothing, but it will not solve your "wait for publishing" or "stuck transaction" issues as effectively as 3. In fact, 2-replica setups often feel "brittle" because they lack the quorum flexibility that makes StarRocks stable. References

@Rocky But end of the day with replication num = 3, we still need 2 node to successfully write, which is the same as 2/2 nodes write successful in replication num = 2. Is the resource consumption linear from 1 to 2 to 3 replication_num? Or going from 2 to 3 is not exactly 1.5x CPU util and latency?

It is a common misconception that since both

replication_num=2

and

3

require 2 nodes for a "Majority" write, the performance and stability are the same. In reality,

replication_num=3

is significantly more stable in the face of the EBS throttling you are experiencing. 1. The "Slowest Node" Penalty (Latency) In a distributed system, your write latency is often dictated by the slowest node in your quorum. * Replication = 2 (Quorum 2/2): You must wait for both nodes. If any single BE node hits an EBS throttle (those 12ms spikes), your load transaction must wait for it. Your "p99" latency will be the sum of your load time + the worst EBS spike across your nodes. * Replication = 3 (Quorum 2/3): You only wait for the fastest 2 out of 3. If one node is experiencing an EBS throttle or is busy with rebalancing, the transaction can "ignore" it and move to the "Published" state as soon as the other two nodes respond. Result:

replication_num=3

effectively "masks" intermittent disk latency spikes.

replication_num=2

exposes you to every single one of them. 2. Is Resource Consumption Linear? It depends on the resource type. It is not a simple 1.5x increase for everything. * Disk Space: Linear (1.5x). You are storing 3 copies instead of 2. * Network Bandwidth: Linear (1.5x). Data must be sent to one additional node. * CPU Utilization: Less than 1.5x (if using

replicated_storage=true

). * StarRocks (v2.5+) uses Single Leader Replication by default. Data is compressed and indexed on the Primary replica node, and then the resulting "segments" (the already-processed files) are sent to the secondary replicas. * The secondaries mostly just do I/O to write the files to disk. They don't re-run the expensive CPU-bound tasks like parsing, sorting, or complex compression. * Memory: Minimal increase. The FE needs to track slightly more tablet metadata, but for the BE, the memory overhead of having a third replica is negligible compared to the query and load buffers. 3. Why

replication_num=2

makes "Wait for Publishing" worse The "Wait for Publishing" error happens because the FE won't let a partition's version move from 362 to 369 until it knows the data is safely on a quorum of BEs. If you have 2 replicas and one node is struggling (disk saturation or rebalancing), that node becomes a hard blocker. The version cannot move forward. If you have 3 replicas, that struggling node can lag behind while the other two maintain the "published" version, allowing your transactions to keep flowing. Recommendation If you are already seeing "version errors" and "stuck transactions" during rebalancing: 1. Do not go to 2 replicas. You will lose the "slack" that allows the cluster to stay healthy when one node is under pressure. 2. Use 3 replicas to survive the EBS jitter. 3. Address the rebalancing pressure by slowing down the background migration (as mentioned in the previous response). 4. Confirm

replicated_storage

is

true

on your tables (

SHOW CREATE TABLE ...

). This ensures that the 3rd replica costs you mostly disk space, not extra CPU. Verdict: The cost of the extra disk space (1.5x) is much lower than the "cost" of production downtime or stuck loads caused by the lack of quorum flexibility in a 2-replica setup. References • administration/management/resource_management/Replica.md • knowledge/usage/best-practice.md • best_practices/primarykey_table.md • faq/operation_maintenance_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md

@Rocky Do I have to 1.5x my EBS throughput and IOPS to handle replication num = 3?

You do not necessarily need to 1.5x your provisioned EBS throughput just to switch from 2 to 3 replicas, but there is a clear trade-off between resource consumption and the "stability benefit" you are looking for. Here is the breakdown of how resources scale when moving from

replication_num = 2

to `3`: 1. Write Throughput (MB/s): The 1.5x Factor System-wide, the total amount of data written to disk increases linearly by 1.5x. * Per-Node Impact: If your cluster size (number of BEs) remains the same, each BE will indeed see a ~1.5x increase in its average write throughput because it is now responsible for storing more "shards" (tablets) of the total data. * The EBS Reality: If you were already hitting 700 MB/s sporadically with 2 replicas, moving to 3 will likely cause you to hit that limit more frequently unless you increase the provisioned throughput or add more BE nodes to spread the load. 2. CPU and IOPS: Not 1.5x Because of a feature called Single Leader Replication (

replicated_storage = true

, which is default in v3.x), the resource hit is not perfectly linear: * CPU: Only the "Primary" replica for a tablet does the heavy lifting (sorting, indexing, and compression). The two "Secondary" replicas simply receive the finished data segments and write them to disk. The 3rd replica adds almost zero extra CPU load for data processing. * IOPS: For StarRocks, data is typically written in large sequential blocks (segments). You are more likely to hit your Throughput (MB/s) limit on EBS before you hit your IOPS limit. 3. The "Quorum Slack" Benefit Even though the 3rd replica adds 50% more write volume, it actually protects you from the performance impact of that volume: * With 2 Replicas (2/2 Quorum): Every single write must finish on both nodes. If one node hits its 700 MB/s EBS limit and experiences a 12ms spike, your entire load transaction stops for 12ms. * With 3 Replicas (2/3 Quorum): If one node is throttled by EBS, the FE can complete the transaction as soon as the other two nodes finish. You are no longer held hostage by the slowest node in the cluster. Should you increase the limit? If your current EBS metrics show you are frequently hovering near the 700 MB/s limit: 1. Yes, increase it slightly: Moving to 3 replicas will likely push you over the edge into constant throttling. You might not need a full 1.5x (1050 MB/s), but increasing to 850–900 MB/s would provide a safer ceiling. 2. Add Disks, not just Throughput: Instead of one massive EBS volume per BE, consider using 2 or 3 smaller volumes per BE. StarRocks can use multiple paths (

storage_root_path

). This aggregates the throughput limits of multiple EBS volumes, often providing a higher "burst" ceiling for the same price. Summary Verdict The storage cost and throughput volume go up by 1.5x, but the CPU cost does not. Most importantly, the stability of your "Published Versions" will improve because you finally have a "spare" replica that can be slow without stalling your transactions. References • administration/management/resource_management/Replica.md • deployment/plan_cluster.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • faq/operation_maintenance_faq.md • loading/loading_introduction/loading_considerations.md • administration/management/BE_configuration.md • knowledge/usage/best-practice.md References

@Rocky If I increase my replication from 2 to 3, does it trigger immediate tablet rebalancing? My CPU is already at 60-70% util. Wonder if it can take the load

Yes, increasing the replication number from 2 to 3 will trigger immediate activity, but it is technically handled as a "Tablet Repair" process rather than a standard "Rebalancing" process. 1. Does it happen immediately? Yes. Once you execute

ALTER TABLE ... SET ("replication_num" = "3");

, the Frontend (FE) Tablet Scheduler will scan the metadata, realize the current count (2) is less than the target count (3), and immediately start scheduling "clone" tasks. These tasks tell BE nodes to copy existing tablet files to a third node. It does not wait for a maintenance window; it starts as soon as resources (slots) are available. 2. Can your CPU handle it at 60-70%? At 60-70% CPU utilization, you are in a "yellow zone." While tablet cloning is primarily I/O and Network intensive, it still consumes CPU for: * Checksum verification of the files being moved. * Metadata updates and managing the transfer threads. * Potential background compaction triggered by the new data on the destination node. If your 60-70% CPU is due to heavy query traffic, adding the background noise of replicating your entire dataset could push you into the 80-90% range, potentially increasing query latency. 3. How to safely manage the load Instead of a "big bang" change, you should use the following strategies to prevent a CPU/IO spike: A. Change one table at a time Don't change the replication for the entire database at once. Start with your smallest or least-critical tables to observe the CPU impact.

sql
ALTER TABLE table_name SET ("replication_num" = "3");

B. Throttle the Concurrency (Crucial) Before running the

ALTER

, you can lower the number of simultaneous clone tasks allowed per disk. This ensures the background repair doesn't "starve" your queries for resources. In the FE configuration (or via SQL if mutable):

sql
-- Default is usually 2 or higher. Setting to 1 slows the process but protects your CPU.
ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "1");

-- You can also limit the total number of scheduling tasks
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "100");

C. Monitor the Progress You can see how many replicas are still being created using:

sql
SHOW PROC '/cluster_balance/pending_tablets';

If the number is huge and your CPU is spiking, you can temporarily set

tablet_sched_max_scheduling_tablets

to

0

to pause the process. Summary Verdict * Trigger: Immediate (as a background "repair" task). * CPU Risk: Moderate. 60-70% is high enough that a full-cluster replication could cause contention. * Strategy: Throttle first, then upgrade one table at a time. This will allow you to reach the stability of 3 replicas (which you need to fix your version errors) without crashing your current performance. References • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/FE_configuration.md

@Rocky Would 3 replication possibly even perform better than 2? Because not held hostage by quorum

Whether

replication_num=3

performs better than

2

depends on whether you mean "average latency" or "cluster stability/tail latency." The short answer is: It is much more stable, but it does not technically "short-circuit" latency to mask a slow node. 1. Quorum vs. Short-Circuiting (The Nuance) There is a common misconception that with 3 replicas (Quorum 2/3), StarRocks returns a "Success" to you as soon as the first 2 nodes finish. This is actually not the case for the loading phase. According to the official documentation: * Status Wait: StarRocks returns the loading task status only after all data replicas have returned a status (success, failure, or timeout). * The "Hostage" Scenario: If you have 3 replicas and one node is experiencing EBS throttling (making it slow but not "dead"), the transaction will still wait for that slow node to report back before telling you the load succeeded. Result: In a "slow node" scenario, 3 replicas will have roughly the same latency as 2 replicas because both are waiting for the slowest responder. 2. Why 3 Replicas Is Better for Performance (Stability) While it doesn't mask individual "slow" writes, 3 replicas perform significantly better for Version Visibility and Availability: * Version Visibility (Publishing): After a load, the FE must "Publish" the new data version. A version is considered published when a Majority of replicas have it. * With 2 replicas, if one node is slow, the FE cannot publish the version. This leads to the "Wait for Publishing" or "Version Already Exists" errors you've been seeing. * With 3 replicas, if one node is lagging, the FE can still publish the version once the other 2 nodes are ready. This prevents your entire ingestion pipeline from backing up. * Failure Resilience: If one node actually fails or times out: * 2 Replicas: The load fails immediately (since 1/2 is not a majority). You have to retry. * 3 Replicas: The load succeeds (since 2/3 is a majority). The system stays up, and you don't even notice the failure. 3. CPU & Resource Summary Metric | 2 Replicas | 3 Replicas | Impact Write Quorum | 2 of 2 | 2 of 3 | 3 is more likely to succeed under pressure. Avg. Latency | Slowest of 2 | Slowest of 3 | 3 might be slightly higher (more chances to hit a slow node). Tail Latency (p99) | High | Lower | 3 avoids "stuck" versions and load failures. CPU Cost | 1x | ~1x | With

replicated_storage=true

, the 3rd replica is "free" CPU-wise. Verdict You are correct that you are currently "held hostage" by a 2/2 quorum. Moving to 3 replicas won't make your "fast" loads faster, but it will massively reduce the number of "stuck" loads caused by EBS jitter or rebalancing. At 60-70% CPU, as long as you have

replicated_storage=true

, the transition to 3 replicas is primarily a disk and network bandwidth trade-off, not a CPU one. Given your EBS throttling issues, the stability gain of 3 replicas is almost certainly worth the 1.5x disk throughput increase. References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • faq/operation_maintenance_faq.md • knowledge/usage/best-practice.md References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md

@Rocky 1 node holding write hostage aside, how much better does 2 replication protect against data corruption compared to 3. By corruption I mean the tablet version errors during tablet rebalancing. I only care about this. I do not want transactions to ever be stuck.

In terms of preventing stuck transactions and version errors, the jump from 2 to 3 replicas is the single most important stability upgrade you can make. The "protection" 3 replicas provide isn't just slightly better; it changes the fundamental math of how your cluster handles stress. 1. The Quorum Math: 100% vs. 66% Success This is the root cause of "stuck" transactions: * With 2 Replicas: The "Majority" quorum is 2 out of 2. For a transaction to commit, 100% of your target nodes must succeed. If one node is busy rebalancing or has an EBS IOPS spike, the transaction is "held hostage." * With 3 Replicas: The "Majority" quorum is 2 out of 3. You only need 66% of nodes to succeed. You can have one node completely "lost in the woods" (due to rebalancing, disk lag, or a crash) and your transaction will still commit instantly. 2. Protection Against "Version Errors" "Tablet Version Error" in StarRocks usually doesn't mean your data is corrupted on disk; it means a replica missed a write and is now "stale." * The 2-Replica Trap: If a node misses a write during rebalancing, you are left with only 1 healthy replica. Since 1 is not a majority of 2, the entire table can become read-only or write-blocked until that replica is repaired. This is why you see transactions "stuck." * The 3-Replica Safety Net: If a node misses a write, you still have 2 healthy replicas. Since 2 is a majority of 3, the cluster doesn't care. It marks the 3rd replica as "Version Error" in the background and lets your users keep working. The FE then repairs that replica at its own pace without blocking your pipeline. 3. Rebalancing Reliability During rebalancing, StarRocks is moving data. This creates a "vulnerable window" for that tablet. * 2 Replicas: You are moving 50% of your data redundancy. If a write happens while a tablet is being moved and one of the "static" nodes flinches, the write fails. * 3 Replicas: You are only moving 33% of your redundancy. You still have two "static" copies of the data to handle the 2/3 quorum. The rebalancing process becomes a "background noise" event rather than a "critical risk" event. 4. Why this stops "Stuck Transactions" Most "stuck" transactions in StarRocks are caused by the Publish Version phase. 1. Data is written to BEs. 2. FE asks BEs: "Did you finish?" 3. FE waits for a Majority to say yes. 4. With 2 replicas: FE waits for both. If one is slow (due to EBS throttling or rebalancing), the FE hangs. This "hang" holds the transaction lock, which eventually blocks subsequent loads, leading to a total pipeline stall. 5. With 3 replicas: FE waits for any 2. As soon as the two fastest nodes respond, the version is published. The "hostage" situation is physically impossible as long as 2 out of 3 nodes are healthy. Summary: Is it worth it? If your primary goal is to never have stuck transactions, 2 replicas is effectively "running on a tightrope." You are one EBS micro-latency spike away from a stall. 3 replicas provides the "slack" necessary to absorb rebalancing load and disk jitter without it ever reaching the user level. Recommendation: At 60-70% CPU, the "extra" 3rd replica (which uses almost no CPU due to Single Leader Replication) is the best insurance policy you can buy for cluster uptime. References • loading/SQL_transaction.md • knowledge/trouble_shooting/fe_deadlock_problems.md • administration/management/resource_management/Replica.md

@Rocky but the 2nd replica double my CPU usage. You are saying the 3rd wont impact CPU?

The reason your 2nd replica doubled your CPU usage is likely because your tables are currently using Leaderless Replication (the old default). In that mode, every replica independently performs the most CPU-intensive tasks: data parsing, encoding, compression, and index building. However, you can move to 3 replicas without a proportional CPU spike by ensuring Single Leader Replication is enabled. 1. Why the 2nd replica doubled your CPU If

replicated_storage

is set to

false

(or you are on an older version), the cluster does "Leaderless Replication." * 1 Replica: 1x CPU work (Load -> BE1 builds segments). * 2 Replicas: 2x CPU work (Load -> BE1 and BE2 both build segments independently). * 3 Replicas: 3x CPU work (Load -> BE1, BE2, and BE3 all build segments). In this mode, your 60-70% CPU would likely hit the ceiling if you went to 3 replicas. 2. How the 3rd replica can be "CPU-Light" StarRocks introduced Single Leader Replication (

replicated_storage = true

). In this mode: 1. The Leader does all the hard work (encoding, compressing, building indexes). 2. The Followers (the 2nd and 3rd replicas) simply copy the finished physical files from the Leader via the network. 3. The Result: The followers use very little CPU (mostly just network I/O and disk writes). Adding a 3rd replica in this mode typically adds <5% additional CPU load per BE compared to 2 replicas. 3. How to check and fix this Before you increase your replication to 3, check your current setting:

sql
SHOW CREATE TABLE your_table_name;

Look for

"replicated_storage" = "true"

in the properties. * If it is `false`: This is why your CPU is high. You should change it to

true

before increasing the replication number.

sql
    ALTER TABLE your_table_name SET ("replicated_storage" = "true");

* If it is already `true`: Then your high CPU is not caused by the 2nd replica's "work," but rather by the I/O and background compaction overhead. In this case, adding a 3rd replica will still increase I/O pressure (which might worsen your EBS throttling) but will not "triple" your CPU usage. Recommendation 1. Switch to Single Leader Replication (

replicated_storage=true

) for all tables first. This will likely reduce your current CPU usage significantly. 2. Once CPU drops, then increase

replication_num

to 3. 3. As mentioned before, throttle the cloning speed using

tablet_sched_slot_num_per_path

to ensure the initial background catch-up doesn't spike your CPU. Summary: If you use Single Leader Replication, the 3rd replica is "expensive" for Disk/Network but "cheap" for CPU. Given your current 60-70% util, this is the only safe way to reach 3 replicas. References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • loading/Load_to_StarRocks.md • loading/StreamLoad.md References • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • administration/management/resource_management/Replica.md

@Rocky replicated_storage is true. How should I diagnose why CPU spiked? My EBS IOPS and throughput are rarely exceeded now that I upgraded them. Transaction publish latency and update rowset commit apply duration doubled.

With

replicated_storage = true

(Single Leader Replication), you have successfully optimized the Write Phase (parsing and segment building), but for Primary Key tables, there is a second, often more expensive phase called the Apply Phase. Based on your symptoms (doubled

update_rowset_commit_apply_duration

and doubled CPU), here is the diagnosis: 1. The "Apply" Bottleneck In a Primary Key table, every replica must maintain its own Primary Key Index and Delete Vector to ensure it can serve queries. * What

replicated_storage

does: It allows followers to skip the CPU-heavy task of building data segments. They just download the completed files from the leader. * What it does NOT do: It does not (in most versions) skip the Apply task. Every replica still needs to take those new data segments, look up the keys in its local PK index, and generate a Delete Vector (to mark which old rows are now "deleted" by this update). The Math: * 1 Replica: Node A has 100 tablets. It performs 100 Apply tasks. * 2 Replicas: Node A now has 200 tablets (100 Primary + 100 Secondary). It must perform 200 Apply tasks. * Even though the second 100 tablets didn't "build segments," they are still fighting for the same

transaction_apply_worker_count

thread pool to perform the PK index lookups. 2. Why Latency Doubled The

update_rowset_commit_apply_duration

is the time a task spends in the Apply Thread Pool. * By default, the thread pool size (

transaction_apply_worker_count

) is equal to your CPU core count. * When you doubled your replicas, you doubled the number of tasks being sent to this pool. * If your pool was already busy, the tasks are now sitting in a queue twice as long, effectively doubling your "Apply Duration" and "Publish Latency." 3. How to Diagnose and Fix To confirm this, check the following: 1. Check Thread Pool Queuing: Look at the BE metric

starrocks_be_update_rowset_commit_apply_duration_us

. If this is rising along with

starrocks_be_transaction_apply_thread_pool_queue_size

, your threads are overwhelmed. 2. Verify Persistent Index: Are you using

enable_persistent_index = true

? * If YES: The Apply phase is now gated by Disk I/O (specifically small random reads/writes to the index). Even if your total EBS throughput is fine, the latency of those small IOPS can slow down the Apply phase. * If NO: The Apply phase is gated by Memory and CPU. Doubling the tablets means doubling the memory pressure for the PK indexes, which can trigger more GC or CPU cache misses. 3. Check Version: If you are on v3.2+, you can try increasing the concurrency of the apply phase:

sql
    -- In be.conf (requires restart or update via HTTP API)
    transaction_apply_worker_count = [Set to 1.5x or 2x core count if you have CPU headroom]

Summary for your 3-Replica Plan If you move to 3 replicas, your CPU and Apply latency will likely triple (or hit a hard ceiling) because each BE will then be managing ~300 tablets instead of 100. Recommendation: Before going to 3 replicas, you must resolve the "Apply" bottleneck: * Increase

transaction_apply_worker_count

to see if throughput improves. * Check if you can increase the batch size of your loads. Fewer, larger loads = fewer total Apply tasks. * If you are on an older version, consider upgrading to v3.3+, which contains significant optimizations for the PK Apply phase and better integration with Single Leader Replication. References • administration/management/BE_configuration.md • best_practices/primarykey_table.md • administration/management/monitoring/metrics.md

@Rocky How to remove location labels from PK table and BE node

To remove location labels from your StarRocks Backend (BE) nodes and Primary Key (PK) tables, you use the

ALTER

statements to reset the

labels.location

property to an empty string. 1. Remove labels from BE nodes To clear the location label from a specific Backend, use the

ALTER SYSTEM

command. Setting the label to an empty string effectively removes it from the

Location

field in the system metadata.

sql
-- Find the IP and HeartbeatPort of your BEs
SHOW BACKENDS;

-- Remove the label for a specific BE
ALTER SYSTEM MODIFY BACKEND "172.xx.xx.xx:9050" SET ("labels.location" = "");

* Verification: Run

SHOW BACKENDS;

again. The

Location

column for that BE should now be empty. * Note: You must run this for each BE node that has a label you want to remove. 2. Remove labels from PK Tables For tables (including Primary Key tables), setting the

labels.location

property to an empty string

""

tells the FE to ignore geographical constraints and distribute replicas across any available BEs in the cluster.

sql
-- Check current table properties
SHOW CREATE TABLE your_pk_table_name;

-- Remove the location constraint
ALTER TABLE your_pk_table_name SET ("labels.location" = "");

* Alternative: You can also set it to

"*"

(the default for new tables), which indicates that replicas should be distributed evenly across all existing labels. Setting it to

""

is the cleanest way to "unset" the feature entirely for that table. * Effect: Once you run this, the FE will stop enforcing rack/location-aware placement for new writes and during rebalancing for this table. Why do this? If you are seeing "stuck" transactions or version errors during rebalancing (as you mentioned earlier), restrictive location labels are a common culprit. If a table is pinned to

rack1

and

rack2

, but those racks are at capacity or experiencing disk lag, the FE cannot "spill over" to other healthy nodes, leading to the "hostage" situation you described. Removing these labels gives the scheduler full freedom to use every node in the cluster to resolve quorums. References • administration/management/resource_management/be_label.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md References • administration/management/resource_management/be_label.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/DELETE_BACKEND_BLACKLIST.md • administration/management/BE_blacklist.md