@Rocky I have a shared-nothing be cluster that i scaled from 3 to 6 replicas. It's configured using the kubernetes operator. each table is distributed with 10 buckets

ALTER TABLE mytable DISTRIBUTED BY HASH(my_key) BUCKETS 10

. How long does it take for the data to be distributed across all 6 replicas after i deploy the change?

In a StarRocks shared-nothing cluster, data redistribution happens automatically after you scale the BE replicas. Here is how it works and how you can track it: 1. How it works When you increase the number of BEs (e.g., from 3 to 6) via the Kubernetes Operator, the StarRocks FE Tablet Scheduler detects the new, low-load BEs. It will then start migrating tablets from the original 3 nodes to the 3 new nodes to balance the cluster's load and disk usage. Since you have 10 buckets with a typical replication factor of 3, your table consists of 30 tablets (10 buckets × 3 replicas). * Original State (3 BEs): Each BE holds roughly 10 tablets. * Target State (6 BEs): Each BE will hold roughly 5 tablets. * The Task: The cluster needs to move approximately 15 tablets to the new BEs. 2. How long will it take? The duration depends on several factors: * Data Volume: The size of those 15 tablets. Moving 10GB of data is much faster than 10TB. * Scheduling Limits: By default, StarRocks limits the number of concurrent migrations to avoid impacting query performance. * Network/Disk Speed: The speed of your underlying Kubernetes nodes' storage and interconnect. For a typical small-to-medium dataset, this often finishes within minutes. For very large clusters, it can take hours. 3. How to monitor progress You can check the status of the rebalancing tasks using the following SQL commands: * Check overall progress:

sql
    SHOW PROC '/statistic';

Look for the "Unhealthy" or "Missing" replicas; during rebalancing, the system temporarily creates new replicas on the target nodes before deleting the old ones. * Check active migration tasks:

sql
    SHOW PROC '/cluster_balance/running_tablets';

This shows you exactly which tablets are currently being moved. * Check pending tasks:

sql
    SHOW PROC '/cluster_balance/pending_tablets';

4. How to speed it up (Optional) If you find that the redistribution is taking too long and you have spare resources, you can increase the scheduling concurrency by adjusting the FE configuration:

sql
ADMIN SET FRONTEND CONFIG ("max_scheduling_tablets" = "1000");
ADMIN SET FRONTEND CONFIG ("max_balancing_tablets" = "500");
ADMIN SET FRONTEND CONFIG ("schedule_slot_num_per_path" = "8");

Important Note on Bucket Count With

BUCKETS 10

, you have a maximum of 10-way parallelism for writes/scans per partition across the cluster. While scaling to 6 BEs is perfectly fine, if you plan to scale to much larger clusters in the future (e.g., 20+ BEs), you may want to consider increasing the bucket count for your tables to ensure data can be spread across more nodes. Changing the bucket count for existing data requires an

ALTER TABLE

operation which effectively rewrites the table. References • faq/operation_maintenance_faq.md • best_practices/bucketing.md • administration/management/Scale_up_down.md

@Rocky how do i know when it's done?

The redistribution is officially "done" when the cluster reaches a balanced state and all background migration tasks have completed. You can verify this using the following three checks: 1. Check for Active Scheduling Tasks The most direct way to know if StarRocks is still moving data is to check the internal scheduler. Run these two commands: * Pending Tasks:

SHOW PROC '/cluster_balance/pending_tablets';

* Running Tasks:

SHOW PROC '/cluster_balance/running_tablets';

It is done when: Both of these commands return an empty set. This means there are no more tablets waiting to be moved or currently in transit. 2. Check Replica Health During redistribution, StarRocks creates new replicas on the new BEs before removing the old ones. This can temporarily cause "unhealthy" counts in the system statistics. * Run:

SHOW PROC '/statistic';

It is done when: The

UnhealthyTabletNum

and

InconsistentTabletNum

columns are 0. (Note: If you have 3 replicas for every bucket, the

TotalReplicaNum

should eventually settle at

3 * TotalTabletNum

). 3. Verify Backend Balance Finally, you can verify that the load is actually spread across all 6 BEs by looking at the tablet distribution. * Run:

SHOW PROC '/backends';

It is done when: The

TabletNum

column shows roughly equal numbers for all 6 BE nodes. With your 10-bucket table and 3 replicas (30 tablets total), you should see each BE holding approximately 5 tablets (30 / 6 = 5). Summary Checklist Metric | Status when "Done"

pending_tablets

| Empty

running_tablets

| Empty

UnhealthyTabletNum

| 0

TabletNum

(per BE) | Roughly equal across all 6 nodes References • administration/data_migration_tool.md • administration/management/resource_management/Replica.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky all of a sudden the output of

SHOW PROC '/backends'

shows 0.000 B of DataUsedCapacity. queries are running fine

Seeing

0.000 B

for

DataUsedCapacity

in

SHOW PROC '/backends'

while queries are running fine is a common behavior in StarRocks, especially right after scaling or a cluster restart. This occurs because of the way StarRocks reports and aggregates storage statistics: 1. Reporting Lag (The 5-Minute Window) StarRocks does not update storage metrics in real-time to avoid overwhelming the Leader FE. There are two main timers involved: * BE Report Interval: By default, BEs report their disk state to the FE every 60 seconds (

report_disk_state_interval_seconds

). * FE Aggregation Interval: The FE typically updates the aggregated tablet statistics every 5 minutes (controlled by the FE config

tablet_stat_update_interval_second

). Since you just scaled your cluster, the new BEs (and potentially the old ones busy with rebalancing) may not have completed a full reporting cycle yet. 2. Authority of Metadata As noted in the documentation,

SHOW PROC '/backends'

retrieves metadata from the FE you are currently connected to. If you are connected to a Follower FE instead of the Leader, the data may be even further out of sync. * Recommendation: Try running

SHOW BACKENDS;

(which always pulls from the Leader) or ensure you are connected to the Leader FE. 3. Better ways to verify data exists If you want to verify that your data is actually there and taking up space while waiting for the BE metrics to catch up, use these commands: * Check Database/Table level storage:

sql
    SHOW DATA;
    -- Or for your specific table
    SHOW DATA FROM mytable;

SHOW DATA

is generally more responsive than the backend process list for data volume. * Check Partition level storage:

sql
    SHOW PARTITIONS FROM mytable;

Look at the

DataSize

column here. If this shows values greater than 0, your data is safe and recognized by the catalog. Summary Don't worry—as long as your queries are returning results, your data is safe. The

0.000 B

is simply a display lag in the management console. It should populate with the correct values within 5 to 10 minutes once the background reporting tasks stabilize after the scale-out. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md