@Rocky how to fix 78 inconsistent tablets in my Starrocks cluster?

Sorry, I encountered an error while processing your request: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}

@Rocky yo

Yo! I'm Rocky. How can I help you with StarRocks today?

@Rocky how to fix 78 inconsistent tablets in my Starrocks cluster?

In StarRocks, inconsistent tablets usually mean that the replicas of a single tablet have matching version numbers but different data (detected via checksums). Follow these steps to identify and fix the 78 inconsistent tablets: 1. Identify the affected Tablet IDs First, find which database and tablets are reporting inconsistency: * Locate the Database: Run

SHOW PROC '/statistic';

to see which database has a non-zero value in the

InconsistentTabletNum

column. * Get Tablet IDs: Run

SHOW PROC '/statistic/<db_id>';

(replace

<db_id>

with the ID found in the previous step). The

InconsistentTablets

column will list the IDs of the 78 tablets. 2. Analyze the Replicas For a specific

TabletId

, inspect the state of its replicas to determine which one is "the outlier":

sql
SHOW TABLET <tablet_id>;

Check the following columns for discrepancies among the replicas: * `Version`: All replicas usually have the same version. *

DataSize

and `RowCount`: If one replica has a significantly different size or row count than the others, it is likely the inconsistent one. * `LstConsistencyCheckTime`: Shows when the last check was performed. 3. Fix by Marking the "Bad" Replica Once you identify the replica that is incorrect, you can force StarRocks to delete it and recreate it from a healthy replica by marking it as BAD:

sql
ADMIN SET REPLICA STATUS PROPERTIES (
    "tablet_id" = "12345", 
    "backend_id" = "10001", 
    "status" = "bad"
);

* Warning: Replicas set to

bad

status may be dropped immediately. Ensure at least one healthy replica exists before doing this. * StarRocks will automatically schedule a clone task to restore the replica from the remaining "good" ones. 4. Speed up the Repair If the cluster is large and you want these 78 tablets fixed immediately, you can increase their repair priority:

sql
ADMIN REPAIR TABLE <table_name> [PARTITION (partition_name)];

This gives the repair tasks a

VERY_HIGH

priority in the Tablet Scheduler. 5. Verify the Fix Monitor the progress of the repairs: * Check pending tasks:

SHOW PROC '/cluster_balance/pending_tablets';

* Check running tasks:

SHOW PROC '/cluster_balance/running_tablets';

* Final Check: Once the tasks finish, run

SHOW PROC '/statistic';

again. If the count is still not zero, you may need to trigger a fresh consistency check:

sql
    ADMIN CHECK TABLET (tablet_id1, ...) PROPERTIES("type" = "consistency");

Note: If all replicas for a tablet are inconsistent and you cannot determine which one is correct, you may need to check the BE logs for disk errors or contact StarRocks support/community if data integrity is critical. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_REPLICA_STATUS.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SHOW_REPLICA_STATUS.md

@Rocky All replicas have the same version and row count — there's no "outlier" to mark bad The "inconsistency" is only a checksum difference caused by different compaction states (DataSize differs slightly, but RowCount is identical) Marking a replica bad + cloning creates a fresh copy, but since CDC keeps writing, the new replica diverges again immediately

In StarRocks (especially with Primary Key tables and CDC workloads), seeing inconsistent tablets where the version and row count match but the checksum and data size differ is often a false positive caused by inconsistent compaction states across replicas. Because CDC generates many small versions, replicas may be at different stages of merging rowsets. In some StarRocks versions, the default checksum algorithm is sensitive to the physical layout (how many files/rowsets exist) rather than just the logical data. 1. Confirm it is a False Positive If the

VisibleVersion

and

RowCount

are identical across all replicas, the data is logically consistent. You can verify this by running a manual consistency check that uses the newer, more robust row-by-row algorithm:

sql
-- Trigger a manual consistency check for specific tablet IDs
ADMIN CHECK TABLET (tablet_id1, tablet_id2, ...) 
PROPERTIES("type" = "consistency");

2. Why "Marking Bad" doesn't work As you've observed, marking a replica as

bad

triggers a clone. However, if the CDC stream is heavy, the new replica will immediately begin its own compaction schedule, which will naturally diverge from the other replicas' physical layouts. The background checker will then see a different checksum and flag it as inconsistent again. 3. Recommended Fixes A. Upgrade to a version with Checksum Fixes StarRocks recently improved replica checksum verification for Primary Key tables to prevent these false positives. * Fix: Ensure you are on at least v3.3.18+, v3.2.14+, or v4.0.0+. These versions include enhancements (like issue #56519) that use a more associative checksum (XOR-based) which is layout-independent. B. Adjust Background Consistency Check If you cannot upgrade immediately, you can reduce the frequency of the background consistency check to stop the constant alerts, or ignore the count if row counts match. * BE Configuration: Check the

check_consistency_worker_count

(default is 1). Setting this to 0 on BEs can stop the background checks, but this is a "blindfold" approach and not recommended for long-term health. C. Manual Alignment (Maintenance Window) If the inconsistency count persists and you want to "clear" it for peace of mind: 1. Pause the CDC load briefly. 2. Trigger manual compaction on the affected tablets to ensure all replicas reach the same "flat" state:

sql
    -- Note: Manual compaction is usually done via HTTP API or specialized tools
    -- but you can try to force it by setting high priority
    ADMIN REPAIR TABLE <table_name>;

3. Run the

ADMIN CHECK TABLET

command again. Once the check passes with the same checksum, the

InconsistentTabletNum

in

SHOW PROC '/statistic'

will drop. Summary If

RowCount

and

Version

are identical, your data is safe. The "inconsistency" is a metadata reporting issue due to compaction lag. The most permanent fix is upgrading to a version where the checksum algorithm is layout-agnostic. To verify specific tablets now: Run

SHOW TABLET <id>;

and look at the

CompactionStatus

URL provided in the output. This will show you exactly how many rowsets each replica has. If one has 10 rowsets and another has 2, their checksums will differ in older versions despite having the same rows. References • administration/management/BE_configuration.md • developers/type_checker_xml_configuration.md