This message was deleted.

The process of deprecating (scaling in) BE nodes in a shared-nothing architecture is handled via the

DECOMMISSION

command. Unlike a

DROP

command, which is immediate and destructive,

DECOMMISSION

is a safe, asynchronous operation that migrates all data replicas (tablets) from the target nodes to the remaining nodes before removing them from the cluster metadata. Since you are both reducing the node count (12 to 8) and upgrading the instance type (r6in.4xlarge to 8xlarge) via Terraform, you should follow a coordinated workflow to ensure data integrity and query performance. 1. Recommended Workflow: "Add then Decommission" The safest approach is to add the 8 new larger nodes first to ensure there is sufficient capacity and then decommission the old ones. 1. Deploy new nodes: Use Terraform to provision the 8 new

r6in.8xlarge

instances. 2. Add to StarRocks: Register the new nodes in the cluster:

sql
    ALTER SYSTEM ADD BACKEND "new_be_ip:9050"; -- Repeat for all 8

3. Decommission old nodes: Once the new nodes are

Alive: true

in

SHOW BACKENDS

, start decommissioning the 12 old ones. Although you can decommission multiple nodes at once, it is best practice to do them in small batches (e.g., 2–4 at a time) to avoid excessive network overhead.

sql
    ALTER SYSTEM DECOMMISSION BACKEND "old_be_ip:9050", "old_be_ip2:9050";

4. Monitor progress:

sql
    SHOW BACKENDS;

*

SystemDecommissioned

will be

true

. * Watch the

TabletNum

column. When it reaches 0, the node will automatically disappear from the list. 5. Clean up Terraform: Once StarRocks has removed the old nodes from its metadata, you can safely use Terraform to destroy the old 12 instances. ────────── 2. Monitoring and Performance Tuning Data rebalancing can take time depending on your data volume. You can monitor and accelerate the process with the following: * Check task details:

sql
    SHOW PROC '/cluster_balance/running_tablets';

* Speed up rebalancing (FE Config): If the migration is too slow, you can increase the number of concurrent scheduling slots (execute on the Leader FE):

sql
    ADMIN SET FRONTEND CONFIG ("schedule_slot_num_per_path" = "8");
    ADMIN SET FRONTEND CONFIG ("max_scheduling_tablets" = "2000");
    ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "1000");

────────── 3. Important Precautions * Replica Count: Ensure your

replication_num

is at least 3 (default). If you have single-replica tables, the

DECOMMISSION

will wait until a new replica is created elsewhere. If there aren't enough nodes to satisfy the replication requirement, the decommission will hang. * Terraform Replacement: If your Terraform plan is set to replace existing instances (e.g.,

create_before_destroy = false

), Terraform might terminate instances before StarRocks has finished migrating data. Always perform the StarRocks-level

DECOMMISSION

and wait for

TabletNum

to reach 0 before letting Terraform destroy the old instances. * Disk Space: Ensure the 8 new nodes have enough disk capacity to hold the combined data of the original 12 nodes. Since you are moving from

4xlarge

to

8xlarge

, ensure your EBS volumes (or local NVMe) are sized accordingly. Summary Checklist 1.

ALTER SYSTEM ADD BACKEND

(8 new nodes). 2.

ALTER SYSTEM DECOMMISSION BACKEND

(12 old nodes). 3. Monitor

SHOW BACKENDS

until

TabletNum

is 0. 4.

terraform destroy

the old 12 old instances. References • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md

@Rocky If my replication_num is 1 for my PK table, then I can't decommission? Is the option left to drop and re-create table after new cluster creation?

You do not need to drop and re-create your tables. Even with

replication_num = 1

, StarRocks can safely decommission nodes. Here is how the process handles single-replica tables and why it is safe: 1. How Decommissioning works with

replication_num = 1

When you issue the

DECOMMISSION

command, StarRocks does not simply delete the data. Instead: 1. Temporary Replication: For every tablet that currently has only 1 replica, StarRocks will create a second replica on one of the new/remaining BE nodes. 2. Health Check: StarRocks waits for the new replica to be fully synchronized and marked as "Healthy." 3. Removal: Only after the new replica is verified will the system drop the old replica on the node being decommissioned. 4. Completion: Once the

TabletNum

on the old node reaches 0, the node is automatically removed from the cluster metadata. Note: For Primary Key (PK) tables, the PK index is migrated along with the data segments. The process is transparent to your queries and loading. 2. Is there a risk? The only risk with

replication_num = 1

occurs if the source node (the one being decommissioned) crashes or is terminated before the clone to the new node is finished. Since you have only one copy, if that copy is lost during the transfer, the data is gone. As long as the old nodes stay online and healthy throughout the decommissioning process, your data is safe. 3. Recommended "Safety First" Strategy Since you are migrating between instance types and reducing the node count, follow this sequence to minimize risk: Step A: Scale Out First Add your 8 new

r6in.8xlarge

nodes to the cluster first. This ensures there is plenty of target capacity for the tablets to move to.

sql
ALTER SYSTEM ADD BACKEND "new_node_ip:9050";

Step B: (Optional) Increase Replication Temporarily If you want 100% peace of mind for your PK tables, you can temporarily increase the replication factor to 2 before decommissioning. This ensures that even if a node crashes during the move, you have a backup.

sql
ALTER TABLE your_pk_table SET ("replication_num" = "2");
-- Wait for health check: SHOW TABLET FROM your_pk_table;

After the entire migration is done and old nodes are gone, you can set it back to 1. Step C: Decommission in Batches Decommission the 12 old nodes. It is better to do this in batches (e.g., 3-4 at a time) to avoid saturating the network bandwidth of your new 8-node cluster.

sql
ALTER SYSTEM DECOMMISSION BACKEND "old_ip_1:9050", "old_ip_2:9050", ... ;

Step D: Monitor and Cleanup Monitor the progress:

sql
SHOW BACKENDS; -- Check TabletNum reaching 0
SHOW PROC '/cluster_balance/running_tablets'; -- View active migration tasks

Only after the nodes have disappeared from

SHOW BACKENDS

should you run

terraform destroy

to terminate the instances. Summary * Can you decommission? Yes. * Is dropping/re-creating required? No. * Key Requirement: The old nodes must remain powered on and "Alive" in StarRocks until the

TabletNum

reaches zero. References • administration/management/resource_management/Replica.md

@Rocky If I want to be as quick as possible, can i perform the following steps: 1. Mark 4 nodes to be removed as decommissioned. 2. Wait for re-balancing to be complete 3. Upgrade existing nodes to better machines, but still tied to same EBS so no data re-distribution required? For additional context, this is the terraform plan

!       instance_type                        = "r6in.4xlarge" -> "r6in.8xlarge"
!       public_dns                           = "ec2-<ip>.<http://compute-1.amazonaws.com|compute-1.amazonaws.com>" -> (known after apply)
!       public_ip                            = "<ip>" -> (known after apply)