http://starrocks.io logo
Join Slack
Powered by
This message was deleted.
# questions-and-troubleshooting
s
This message was deleted.
r
In a shared-nothing architecture, when you upgrade BE nodes to larger machines (Vertical Scaling) while retaining the same EBS volumes, the behavior is significantly different from adding new nodes (Horizontal Scaling). 1. What kind of rebalancing happens? If you maintain the same Node Identity (IP address and data directory), virtually no data rebalancing happens. * Identity Preservation: StarRocks identifies a BE by its IP/Port and the metadata stored in its 
storage_root_path
. If you stop a BE, upgrade the instance type, reattach the EBS, and restart it with the same IP, the Frontend (FE) simply sees the node as "Online" again. * Load Score: Since the disk utilization and tablet count on that EBS volume haven't changed, the FE's 
loadScore
for that node remains the same. Therefore, no tablets are moved to or from the node. * The "Timeout" Exception: If the upgrade process takes longer than the 
tablet_sched_be_down_tolerate_time_s
(default is 15 minutes), the FE will assume the node is dead and begin Replica Repair (creating copies of that node's data on other active nodes). Once your upgraded node comes back online with its original EBS, you will have "redundant" replicas, and the FE will eventually delete the extras to return to the desired replica count. 2. How it differs from Horizontal Scaling Feature | Vertical Scaling (Same EBS) | Horizontal Scaling (Adding Nodes) Data Movement | None. Data stays on the same physical EBS volume. | Heavy. Tablets are physically cloned over the network to the new nodes. Resource Usage | Low. Only CPU/RAM are swapped; no network or Disk I/O overhead for migration. | High. Significant background Network and Disk I/O as data redistributes. Duration | Minutes. Limited only by the time it takes to restart the instance. | Hours to Days. Depends on total data volume and 
tablet_sched_slot_num_per_path
. Ingestion Impact | Immediate. The node can immediately handle higher ingestion throughput once it's back online. | Gradual. Performance improves only as data and "load" slowly migrate to the new nodes. Complexity | Simple but requires a brief period of reduced replica availability (downtime for that node). | Complex but provides zero-downtime scaling. Key Recommendations for Vertical Upgrades: 1. Check Timeouts: If you expect the machine swap to take longer than 15 minutes, increase the FE configuration 
tablet_sched_be_down_tolerate_time_s
temporarily to avoid triggering unnecessary replica repairs. 2. Rolling Upgrade: Perform the upgrade one node at a time. Ensure the cluster is "Healthy" (check 
SHOW PROC '/backends';
) before moving to the next node to maintain high availability. 3. Preserve IP/Port: Ensure the new, larger instance retains the same IP address as the old one so the FE recognizes it as the same backend. 4. Immediate Gains: Because ingestion in StarRocks is CPU-intensive (for memtable flushing and encoding), your upgraded nodes will show an immediate reduction in 
write_amp
and CPU usage during peak ingestion. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md
To reduce your cluster size from 12 to 8 nodes while upgrading to larger machines, you should follow an "Add then Decommission" strategy. Since you are moving to new machines (not just resizing the current ones), StarRocks will perform a full background data migration to the new hardware. Recommended Workflow: The "Migrate-and-Replace" Process The safest way to do this is to add the 8 new, larger nodes first, let the data populate them, and then remove the 12 old ones. Step 1: Add the 8 New Large BE Nodes First, prepare your new 8 machines and add them to the cluster. This ensures that the cluster has sufficient capacity before you start removing the old nodes.
Copy code
sql
-- Run for each of the 8 new nodes
ALTER SYSTEM ADD BACKEND "new_node_ip:9050";
* Result: StarRocks will immediately see these new nodes and begin "Rebalancing." It will start copying tablets (data shards) from the 12 old nodes to the 8 new ones. Step 2: Monitor the Progress You can monitor how much data has moved by checking the tablet distribution:
Copy code
sql
SHOW BACKENDS;
* Watch the 
TabletNum
and 
DataUsedCapacity
columns. You will see these numbers rise on the new nodes and (eventually) fall on the old nodes once you start decommissioning. Step 3: Decommission the 12 Old BE Nodes Once the new nodes are healthy and have started taking on data, you can begin removing the old 12 nodes. Use 
DECOMMISSION
rather than 
DROP
, as it ensures data is safely migrated before the node is deleted. Important: It is recommended to decommission in batches (e.g., 3-4 nodes at a time) rather than all 12 at once to prevent excessive network congestion or potential replica risk.
Copy code
sql
-- Batch 1
ALTER SYSTEM DECOMMISSION BACKEND "old_node_1_ip:9050", "old_node_2_ip:9050", "old_node_3_ip:9050";
* Safety Check: Ensure the number of remaining "Alive" nodes is always $\ge$ your replication factor (default is 3). Since you are moving toward 8 nodes, this is safe. * Automatic Removal: Once a node's 
TabletNum
reaches zero and all data is successfully replicated elsewhere, StarRocks will automatically remove it from the backend list. Step 4: Upgrading the FE Nodes (If applicable) If you also need to move the Frontend (FE) nodes to new machines: 1. Add a new Follower: 
ALTER SYSTEM ADD FOLLOWER "new_fe_ip:9010";
2. Verify Status: 
SHOW PROC '/frontends';
(ensure it's 
Alive: true
and 
Join: true
). 3. Drop an old Follower: 
ALTER SYSTEM DROP FOLLOWER "old_fe_ip:9010";
4. Repeat one by one. Never drop the Leader FE until you have migrated the followers; the cluster will automatically elect a new leader among the new machines. ────────── Key Differences from Vertical Scaling In your previous question, we discussed "Vertical Scaling" (keeping the same EBS). Here, because you are moving to new machines, the process is different: Feature | Scaling on Same EBS | Replacing with New Machines Data Movement | None. Data is already on the volume. | Full Migration. All data is copied over the network. Cluster Load | Low impact. | High I/O. High network and disk usage during the transition. Duration | Minutes (restart time). | Hours/Days (depends on data size). Ease of Reversal | Easy (change instance type back). | Hard (requires re-migrating data). Performance Optimization Tips * Control Migration Speed: You can adjust the background migration speed via FE configurations if it impacts your production queries: * `tablet_sched_max_scheduling_tablets_per_be`: Controls how many tablets are being moved to/from a BE at once. * `tablet_sched_max_balancing_tablets`: Total tablets being rebalanced in the cluster. * Check Health: Before decommissioning the next batch, always ensure the previous batch has finished and the cluster state is `Health: 
SHOW PROC '/statistic';
(Look for 
UnhealthyTabletNum: 0
). References
Open in Slack
PreviousNext
Powered by