http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> When I added a location label to my...
# questions-and-troubleshooting
m
@Rocky When I added a location label to my BE nodes and main table containing ~5 billion records, I have a lot of pending tablets with status either location_mismatch, redundant, or version_incomplete. I understand redundant and location_mismatch. However, is version_incomplete expected?
r
When you update location labels for a large table (~5 billion records), the 
version_incomplete
 status is indeed expected and part of the normal migration process. Here is why you are seeing these statuses together: 1. Why 
version_incomplete
happens In StarRocks, a tablet is marked as 
version_incomplete
when the total number of alive replicas is sufficient (i.e., $\ge$ your replication factor), but the number of healthy (fully up-to-date) replicas is less than the replication factor. When you add location labels: * The Workflow: StarRocks identifies that existing replicas are in the wrong place (
location_mismatch
). It begins creating new replicas on BEs that match the new labels. * The Status: While a new replica is being "cloned" to the new BE, it is considered alive but not yet healthy because it hasn't finished copying all data versions from the source. * The Result: During this window, you have the old replicas (which are 
location_mismatch
) and new replicas (which are 
version_incomplete
until the clone finishes). Because you have "too many" replicas during the handoff, you also see 
redundant
. 2. Is it a problem? For a table with 5 billion records, this process will take significant time and IO. 
version_incomplete
simply indicates that the background clone tasks are still synchronizing data to the new locations. * Expected: It should eventually transition to 
healthy
once the clone completes and the `location_mismatch`/`redundant` replicas are purged. * Warning Sign: It is only a concern if the tablets stay in 
version_incomplete
for an extended period without progress. This could mean the clone tasks are failing (e.g., due to disk space, network limits, or BE high load). 3. How to monitor progress You can verify that the system is working through it by checking the Tablet Scheduler's current tasks: * Check scheduling tasks:
Copy code
sql
    SHOW PROC '/cluster_balance/scheduling_tablets';
This will show you the active 
CLONE
tasks. If you see many tasks with a 
RUNNING
state, the system is actively moving your 5 billion records. * Check replica details:
Copy code
sql
    ADMIN SHOW REPLICA STATUS FROM <table_name> WHERE status = 'VERSION_INCOMPLETE';
This will confirm if the replicas are simply lagging in version. Summary: Since you've triggered a massive data movement by changing labels, 
version_incomplete
is just the "work-in-progress" sign for the new replicas catching up to the current data parity with the old ones. Once the data is fully copied, the FE will delete the `redundant`/`mismatch` replicas and the status will return to 
NORMAL
. References • administration/management/resource_management/be_label.md • administration/management/resource_management/Replica.md
m
@Rocky invalid type: scheduling_tablets
r
My apologies for the error in the previous command. The correct path for viewing active tablet repair or balance tasks is 
running_tablets
, not 
scheduling_tablets
. You can monitor the progress of the data movement using these specific paths: 1. Check Active Migration/Clone Tasks To see tablets that are currently being moved or synchronized (the tasks moving your 5 billion records):
Copy code
sql
SHOW PROC '/cluster_balance/running_tablets';
In the output, look at the 
Type
 column (REPAIR or BALANCE) and the 
State
 column. If you see many tasks here, the system is actively working on resolving the 
version_incomplete
and 
location_mismatch
statuses. 2. Check Tasks Waiting to Start If the system is under heavy load, many tasks might be waiting in the queue:
Copy code
sql
SHOW PROC '/cluster_balance/pending_tablets';
3. Check Overall Balance Status To see if the system has recognized that your data is not yet compliant with the new location labels:
Copy code
sql
SHOW PROC '/cluster_balance/balance_stat';
Look for the row where 
BalanceType
is 
label-aware location
. If 
Balanced
is 
false
, it means StarRocks is still aware that replicas are not in the correct locations based on your new BE labels. Why you see 
version_incomplete
again Since you have 5 billion records, the system must copy (clone) large amounts of data to the new nodes. * A tablet is 
version_incomplete
the moment it is created on a new node because it starts with 0 data and must pull all versions from an existing replica. * Once the 
running_tablet
task for that ID finishes, the status for that specific replica will change from 
version_incomplete
to 
OK
. References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/Replica.md
m
@Rocky
Copy code
+----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+---------------+--------------+---------------+
| TabletId | ReplicaId | BackendId | Version | LastFailedVersion | LastSuccessVersion | CommittedVersion | SchemaHash | VersionNum | IsBad | IsSetBadForce | State        | Status        |
+----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+---------------+--------------+---------------+
| 7453456  | 7462071   | 203697    | 1963    | 1964              | 1963               | 1957             | 0          | 71         | false | false         | NORMAL       | VERSION_ERROR |
| 7453618  | 7462450   | 95876     | 1975    | 1976              | 1975               | 1957             | 0          | 58         | false | false         | NORMAL       | VERSION_ERROR |
| 7454640  | 7462453   | 203697    | 1702    | 1703              | 1702               | 1684             | 0          | 59         | false | false         | NORMAL       | VERSION_ERROR |
| 7458424  | 7462082   | 110783    | 809     | 810               | 809                | 801              | 0          | 76         | false | false         | NORMAL       | VERSION_ERROR |
| 7460592  | 7460593   | 171908    | 405     | -1                | 405                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460592  | 7462223   | 203697    | 408     | 409               | 408                | 368              | 0          | 177        | false | false         | NORMAL       | VERSION_ERROR |
| 7460594  | 7460595   | 203701    | 409     | -1                | 409                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460602  | 7460603   | 203700    | 387     | -1                | 387                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460604  | 7460605   | 110783    | 379     | -1                | 379                | 368              | 0          | 10         | true  | false         | DECOMMISSION | DEAD          |
| 7460610  | 7460611   | 203701    | 416     | -1                | 416                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460610  | 7462328   | 171908    | 418     | 420               | 418                | 368              | 0          | 163        | false | false         | NORMAL       | VERSION_ERROR |
| 7460618  | 7460619   | 203700    | 404     | -1                | 404                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460624  | 7460625   | 171908    | 413     | -1                | 413                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460626  | 7460627   | 203701    | 435     | -1                | 435                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460634  | 7460635   | 203700    | 387     | -1                | 387                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460642  | 7460643   | 203701    | 374     | -1                | 374                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460650  | 7460651   | 203700    | 370     | -1                | 370                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460656  | 7460657   | 171908    | 446     | -1                | 446                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460658  | 7460659   | 203701    | 416     | -1                | 416                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460666  | 7460667   | 203700    | 428     | -1                | 428                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460666  | 7462448   | 203699    | 434     | 435               | 434                | 368              | 0          | 151        | false | false         | NORMAL       | VERSION_ERROR |
| 7460674  | 7460675   | 203701    | 409     | -1                | 409                | 368              | 0          | 10         | true  | false         | DECOMMISSION | DEAD          |
| 7460682  | 7460683   | 203700    | 412     | -1                | 412                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460682  | 7462332   | 171908    | 418     | 420               | 418                | 368              | 0          | 163        | false | false         | NORMAL       | VERSION_ERROR |
| 7460688  | 7460689   | 171908    | 453     | -1                | 453                | 368              | 0          | 10         | true  | false         | DECOMMISSION | DEAD          |
| 7460690  | 7460691   | 203701    | 435     | -1                | 435                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460690  | 7462511   | 203699    | 442     | 443               | 442                | 368              | 0          | 143        | false | false         | NORMAL       | VERSION_ERROR |
| 7460698  | 7460699   | 203700    | 376     | -1                | 376                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460706  | 7460707   | 203701    | 409     | -1                | 409                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460714  | 7460715   | 203700    | 412     | -1                | 412                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460722  | 7460723   | 203701    | 392     | -1                | 392                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460730  | 7460731   | 203700    | 428     | -1                | 428                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460738  | 7460739   | 203701    | 416     | -1                | 416                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460754  | 7460755   | 203701    | 409     | -1                | 409                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460762  | 7460763   | 203700    | 363     | 364               | 368                | 368              | 0          | 12         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7460770  | 7460771   | 203701    | 401     | -1                | 401                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460778  | 7460779   | 203700    | 428     | -1                | 428                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460778  | 7462447   | 11001     | 434     | 435               | 434                | 368              | 0          | 149        | false | false         | NORMAL       | VERSION_ERROR |
| 7460786  | 7460787   | 203701    | 363     | 364               | 368                | 368              | 0          | 12         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7460794  | 7460795   | 203700    | 436     | -1                | 436                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460810  | 7460811   | 203700    | 387     | -1                | 387                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7460818  | 7460819   | 203701    | 363     | 364               | 368                | 368              | 0          | 11         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7460826  | 7460827   | 203700    | 370     | -1                | 370                | 368              | 0          | 12         | true  | false         | DECOMMISSION | DEAD          |
| 7460842  | 7460843   | 203700    | 370     | -1                | 370                | 368              | 0          | 11         | true  | false         | DECOMMISSION | DEAD          |
| 7450088  | 7462417   | 95876     | 2737    | 2738              | 2737               | 2720             | 0          | 41         | false | false         | NORMAL       | VERSION_ERROR |
| 7451123  | 7451124   | 11001     | 2476    | 2477              | 2476               | 2477             | 0          | 10         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7437714  | 7437715   | 203697    | 5070    | 5071              | 5070               | 5071             | 0          | 13         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7437714  | 7461723   | 11001     | 5071    | 5072              | 5071               | 5071             | 0          | 15         | false | false         | NORMAL       | VERSION_ERROR |
| 7437778  | 7437779   | 203697    | 5070    | 5071              | 5070               | 5071             | 0          | 13         | false | false         | DECOMMISSION | VERSION_ERROR |
| 7437778  | 7461715   | 203699    | 5071    | 5072              | 5071               | 5071             | 0          | 15         | false | false         | NORMAL       | VERSION_ERROR |
| 7437948  | 7461726   | 203699    | 5071    | 5072              | 5071               | 5071             | 0          | 13         | false | false         | NORMAL       | VERSION_ERROR |
| 7441509  | 7462391   | 110783    | 4500    | 4501              | 4500               | 4495             | 0          | 38         | false | false         | NORMAL       | VERSION_ERROR |
| 7444042  | 7462408   | 95876     | 3989    | 3990              | 3989               | 3981             | 0          | 40         | false | false         | NORMAL       | VERSION_ERROR |
| 7447659  | 7462416   | 11001     | 3284    | 3285              | 3284               | 3272             | 0          | 38         | false | false         | NORMAL       | VERSION_ERROR |
+----------+-----------+-----------+---------+-------------------+--------------------+------------------+------------+------------+-------+---------------+--------------+---------------+
Is simple solution just to set version to committed version for every tablet that has version error?
r
The 
VERSION_ERROR
status you are seeing, alongside 
DECOMMISSION
and 
DEAD
, is expected behavior during a large-scale data migration triggered by location label changes. For a table with ~5 billion records, StarRocks is performing a massive "re-balancing" act. Here is a breakdown of why those specific statuses appear and why you should avoid manual version manipulation. 1. Why 
VERSION_ERROR
is happening When you apply location labels, StarRocks determines that your existing replicas are in the "wrong" place. The process follows this sequence: 1. New Replica Creation: The FE creates new, empty replicas on BEs that match your new location labels. These replicas start at Version 0 (or a very low version). 2. Cloning: The FE triggers a 
CLONE
task to copy data from the old replicas to the new ones. 3. The Error Status: Because these new replicas are currently missing the data versions held by the "old" replicas, they are flagged with 
VERSION_ERROR
. 4. Catching Up: As the background clone tasks progress, the 
Version
on the new BEs will increase. Once the new replica reaches the current 
CommittedVersion
, the status will automatically flip from 
VERSION_ERROR
to 
OK
. 2. Why you see 
DECOMMISSION
and 
DEAD
 The replicas on your "old" backends are marked as 
DECOMMISSION
. * They are kept alive as the source for the clones. * Once a new replica on a "correct" BE becomes healthy (
OK
), the FE will mark the old replica as 
DEAD
and eventually delete it to reclaim space. 3. Should you manually set the version? No, you should not manually set the version to match the committed version. Manual version intervention (e.g., via 
ADMIN SET PARTITION VERSION
or internal metadata overrides) is a high-risk operation intended for disaster recovery, not for standard migrations. * Data Integrity: If you force the version to match the 
CommittedVersion
without the actual data being copied, you are essentially lying to the FE. * Result: Subsequent queries hitting those replicas will fail with "File not found" errors or return inconsistent/partial results because the physical data files don't actually exist on the disk yet. Recommended Actions Since you are dealing with 5 billion records, the migration will take time. Instead of manual versioning, do the following: 1. Monitor Clone Progress: Check how many tablets are actively being copied:
Copy code
sql
    SHOW PROC '/cluster_balance/running_tablets';
If the list is moving and the 
State
is 
RUNNING
, the system is working as intended. 2. Prioritize Important Tables: If you need this specific table to recover faster, you can give it a high priority in the repair queue:
Copy code
sql
    ADMIN REPAIR TABLE <your_table_name>;
3. Check for Stuck Tasks: If you see the same 
TabletId
in 
VERSION_ERROR
for days without the 
Version
number increasing in 
ADMIN SHOW REPLICA STATUS
, check the BE logs (
<http://be.INFO|be.INFO>
) for "clone task failed" to see if there are underlying issues like disk space or network timeouts. Summary: 
VERSION_ERROR
on the new BE nodes is simply a "loading" state. Given the scale of your data, it is normal for this to persist until the background data transfer completes. References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_REPLICA_STATUS.md
m
@Rocky Is the best explanation of this hang state that there is no source of truth of tablet version count during mass rebalancing from 8 to 6 nodes. If version count drifts too far during rebalance, it goes into hang state
r
The "hang state" you are describing is a common challenge during mass rebalancing (like an 8-to-6 node reduction) and your intuition about "source of truth" and "version drift" is essentially correct. While "version count drift" isn't a formal StarRocks term, it accurately describes the catch-up race happening in the background. Here is why the rebalance appears to hang: 1. The "Moving Target" (Version Drift) When you are still loading data into the table during a rebalance, the 
CommittedVersion
 (the cluster-wide source of truth) continues to increase. * The Problem: For a tablet to become 
NORMAL
, the new replica must catch up to the 
CommittedVersion
. If your ingestion rate is high and the "clone" speed (copying data from the old 8 nodes to the new 6) is slower than the rate of new versions being created, the replica will never reach parity. * The "Hang": It looks stuck because the 
Version
on the new BE is increasing, but the gap (the "drift") between it and the 
CommittedVersion
is not closing. 2. The "Source of Truth" Bottleneck In an 8-to-6 reduction, the 2 nodes being removed are likely the only ones holding the "healthy" data (the source of truth). * Limited Source Capacity: You are trying to pull ~5 billion records worth of data from only 2 BEs to populate the other 6. If those 2 BEs are under high CPU/IO load, the cloning process throttles significantly. * DECOMMISSION vs. DEAD: As long as those 2 nodes are in 
DECOMMISSION
status, they are still "alive" and serving as the source. However, there is a known issue (Bug #62942) in some versions where if the only up-to-date replicas are on nodes marked for decommissioning, the migration can get stuck in a loop or progress extremely slowly. 3. How to verify if it's "Drift" vs. "Stuck" Run the following to see if the "drift" is increasing:
Copy code
sql
-- Run this twice, 5 minutes apart
ADMIN SHOW REPLICA STATUS FROM <table_name> WHERE status = 'VERSION_ERROR';
* If 
Version
is increasing but 
Status
stays `VERSION_ERROR`: The clone is working, but it’s losing the race against your current ingestion. * If 
Version
is NOT increasing: The clone is truly stuck. Check the 
ErrMsg
in 
SHOW PROC '/cluster_balance/running_tablets';
. Common reasons include 
Disk full
, 
Too many clone tasks
, or 
Connection refused
. Recommendation 1. Slow down ingestion: If possible, reduce the frequency of 
Stream Load
or 
Insert
tasks temporarily. This "stops the goalposts from moving" and allows the clones to catch up. 2. Increase Clone Threads: If your BEs have spare IO, you can allow more concurrent clones:
Copy code
sql
    -- In fe.conf (requires FE restart or ADMIN SET CONFIG)
    -- Default is usually 10-20; don't set it too high or it will impact query performance
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "100");
3. Check for "Stale" sources: If a tablet is stuck and the source BE is actually down (not just decommissioning), the FE might be looking for a version that no longer exists. If you have at least one 
OK
replica, you can use 
ADMIN REPAIR TABLE
to force a new clone from that healthy source. Summary: You aren't losing the "source of truth," but you are likely starving it (IO bottleneck on the 2 nodes) while the target is moving too fast (continuous ingestion). References • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_deadlock_problems.md
Open in Slack
PreviousNext
Powered by