hey Team, i have faced a critical problem in my cluster: Subject: Persistent Index Inconsistency (Duplicate Key Error) on PK Table - v3.5.16 Cluster Environment: Version: StarRocks v3.5.16 (Shared-nothing) Topology: 12 BEs, 3 FEs Table Type: Primary Key Table (40B+ records) Configuration: enable_persistent_index = true, replication_num = 2, write_quorum = ALL Problem Description: An INSERT INTO transaction became stuck in the COMMITTED state for several hours. Investigation revealed that a tablet replica entered an error state due to a primary index rebuild failure. The specific error in the BE logs is: Already exist: _apply_rowset_commit error: load primary index failed: Already exist: FixedMutableIndex<16> insert found duplicate key, new(rssid=5 rowid=...), old(rssid=3 rowid=...). persistent_index.cpp:5333 Analysis: The tablet state shows that rowset 3 (referenced as old in the error) is no longer in the active rowset list (likely compacted into rowset 2), yet the index rebuild process still encounters it. This suggests a stale L0 snapshot or a partial apply (crash/IO error) left the Delete Vectors inconsistent. Because replication_num = 2 and both replicas are inconsistent, automatic repair via cloning fails with make snapshot failed. Proposed Solutions for Verification: We are considering the following manual interventions and would like your confirmation on the safest approach for v3.5.16: Manual Index Recovery via Wren Script: Can we safely trigger the recover_tablet(tablet_id) function defined in be/src/script/script.cpp? If so, what is the recommended way to invoke this in v3.5.16 (e.g., via a specific HTTP endpoint or internal tool)? Manual Deletion of Index Files: If the index rebuild from the existing metadata is failing due to a corrupted L0 snapshot, is it safe to manually delete the index.l0.* and index.l1.* files for the specific tablet on the BE disk? Our understanding is that this would force a full rebuild from rowsets using upsert semantics (via LocalPrimaryKeyRecover), which should resolve the duplicate key conflict. Note: i can not set replication_number = 3 because of lack of storage