REPLICA_MISSING tablets stuck at 1/3 replicas, scheduler never queues repair despite intact source replica - no supported recovery path found Cluster: StarRocks 4.1.4, shared-nothing, 3 FE / 6 BE, Kubernetes (StatefulSets starrocks-fe, starrocks-be) Summary: 20 tablets across 9 tables are UnhealthyTabletNum (SHOW PROC "/statistic") with ErrorStateTabletNum=0. 15 of the 20 have exactly 1 of 3 configured replicas physically present anywhere in the cluster (confirmed via information_schema.be_tablets — zero rows on the other backends). The single surviving replica on each is wedged with LastFailedVersion > Version, causing every commit to fail instantly with quorum: 0<1. The FE tablet scheduler has never queued a REPAIR task for any of these 15 tablets, even after ADMIN REPAIR TABLE was run against all 9 affected tables and 12+ hours have elapsed. A BE restart (be-2, the host of the sole replica) did not clear the version wedge and did not cause the scheduler to act either. Evidence: 1. Sample replica detail (SHOW PROC '/dbs/10226/57707081/partitions/57707083/57707082/57707088'): ReplicaId BackendId Version LstSuccessVersion LstFailedVersion LstFailedTime State IsBad 57707089 10002 453 453 456 2026-08-25 142808 NORMAL false 2. Live commit failure (FE log, still recurring at time of writing — 1,304+ occurrences for tablet 55310237 alone): com.starrocks.transaction.TabletQuorumFailedException: Commit failed. txn: 83429506 table: table1 tablet: 55310237 quorum: 0<1 errorReplicas: 71215233{be10002 starrocks-be-2... st:NORMAL V:536254 LFV:536930}, commitBackends: [10002] 3. Scheduler never engages — information_schema.fe_tablet_schedules returns zero rows for any of the 15 tablet IDs, both before and after running ADMIN REPAIR TABLE <table> for all 9 affected tables. Searched all FE log rotations spanning the full day — no scheduler log line mentions any of these tablet IDs except the ADMIN SET REPLICA STATUS audit lines we issued ourselves. 4. Confirmed the scheduler can reason about these tablets, and explicitly declines when the last replica is marked bad — after we set ADMIN SET REPLICA STATUS PROPERTIES("tablet_id"="57708423","backend_id"="10002","status"="bad"), the FE log immediately produced: [TabletChecker.doCheckOnePartition():523] tablet: 57708423 is in unhealthy state: REPLICA_MISSING, but there are no healthy replicas, can not repair This confirms the checker does evaluate the tablet and correctly identifies REPLICA_MISSING — but with the replica in normal (non-bad) NORMAL state (i.e., its actual state for the past 12+ hours), it produces no corresponding log line and no schedule entry at all. We reverted the bad-status change immediately (status="ok") once we saw this, restoring the intact single replica — no data was lost. 5. BE restart does not help. We restarted be-2 (kubectl delete pod starrocks-be-2 under the StatefulSet, clean recreate, 0 restarts, came back Ready in ~60s). Before/after comparison on the same replica: before: Version=453 LstSuccessVersion=453 LstFailedVersion=456 (unchanged after restart) after: Version=453 LstSuccessVersion=453 LstFailedVersion=456 The wedge is evidently persisted in on-disk tablet/rowset meta, not in-memory scheduler state. The restart caused a normal, self-resolving transient spike in unrelated tablet health (other tablets hosted on be-2 briefly fell behind and re-cloned; UnhealthyTabletNum rose 20→102 and CloningTabletNum to 26, both returned to baseline within ~2 minutes with no intervention). Questions: 1. Under what condition does TabletChecker classify a REPLICA_MISSING tablet as eligible for scheduling vs. silently skipping it, when the sole remaining replica is NORMAL/not-bad but has LastFailedVersion > Version? Is a version-gapped replica excluded from being used as a clone source for backfilling the missing replicas, and if so, is that intentional? 2. Is there a supported way to clear a replica's LastFailedVersion in place (short of meta_tool-level manual surgery) so it becomes eligible again as both a write target and a clone source? 3. Given the data on the sole replica is confirmed intact (num_row/data_size stable, state: RUNNING), is there a supported forced-full-clone command that bypasses the checker's apparent no-op path for this specific case? cc: @Rocky

This situation occurs because StarRocks'

TabletChecker

is designed to be safety-first: it refuses to use a replica as a clone source if it is deemed "unhealthy." In your case,

LastFailedVersion > Version

(a "version wedge") signals to the FE that the replica is in an inconsistent state (a publish task failed or is pending). Because this is your sole surviving replica, the scheduler is trapped: it cannot find a healthy source to backfill the missing replicas, and it won't trust the one you have. Here are the answers to your specific questions and the recommended recovery path: 1. Condition for Scheduling Eligibility The

TabletChecker

requires a healthy replica to act as a

clone source

. A replica is considered healthy only if: * Its state is

NORMAL

. * It is not marked as

IsBad

. * Critical: Its

Version

matches the FE's expected

CommittedVersion

, and it has no

LastFailedVersion

(or

Version >= LastFailedVersion

). When your sole replica has a version gap, the FE classifies it as

REPLICA_MISSING

(due to being 1/3) but then immediately disqualifies the only candidate source because of the version mismatch. This is why you see the "no healthy replicas, can not repair" log line even when the status is

NORMAL

. 2. Clearing

LastFailedVersion

in Place There is no "soft" SQL command to specifically nullify

LastFailedVersion

while keeping the replica.

LastFailedVersion

is typically cleared by the BE only after a successful publish of a higher version. Since your transactions are failing with

quorum: 0<1

, the BE never receives a successful publish command to clear that state. Recommended Supported Path (Metadata Alignment): Use the

ADMIN SET PARTITION VERSION

command. This is the primary way to force the FE and BE metadata to align on a specific version, effectively "forgetting" the failed versions.

sql
-- Sync the FE's view of the partition version to the actual physical version on the BE
ADMIN SET PARTITION VERSION PROPERTIES (
    "partition_id" = "57707083", 
    "visible_version" = "453"
);

Note: Replace the ID and version with the specific values for each affected tablet/partition. After running this, the FE will update its metadata. Upon the next report cycle (or after a BE restart), the "failed" state should be cleared, making the replica eligible as a clone source. 3. Forced Clone / Recovery Path There is no "forced" clone command for shared-nothing clusters that bypasses health checks, as doing so could silently propagate data corruption. If

ADMIN SET PARTITION VERSION

does not resolve the wedge, follow this manual recovery sequence: 1. Identify the Row Count: Confirm the row count on the surviving replica via

SHOW PROC '/dbs/...'

. 2. Export/Backup (Optional but Recommended): If the data is critical, try to run a

SELECT INTO OUTFILE

or

BACKUP

for the affected tables. Even if the tablet is in an unhealthy state for writes, it may still be readable for queries if you set

set variable set_replication_num = 1;

(depending on version support) or query it directly. 3. Manual Meta Surgery (Last Resort): If you are comfortable with

meta_tool

, you can manually update the tablet header on the BE disk. This is the "meta-tool level surgery" you mentioned: * Stop the BE. * Use

meta_tool --operation=update_meta

to set the

version

and

max_version

to the same value (e.g., 453) and clear the failed version flag. * Start the BE. * The FE should now see a consistent replica and begin cloning the missing 2/3. Summary of Actions 1. Try

ADMIN SET PARTITION VERSION

for one of the affected partitions first. This is the only "supported" way to reset the version state. 2. Verify via

SHOW PROC

: Check if

LastFailedVersion

returns to

-1

or matches

Version

.