@Rocky I keep getting rowset_segment_id XXXXX no exist error and transactions getting wedged.

The error

rowset_segment_id XXXXX no exist

(often logged as

invalid rssid

or

rssid not found

) usually indicates a mismatch between the metadata in the storage engine and the actual data files on disk, or a corruption in the Primary Key Persistent Index. This typically leads to transactions getting "wedged" (stuck in a

COMMITTED

but not

VISIBLE

state) because the Version Publishing process cannot complete. Common Causes 1. Primary Key Index Inconsistency: If you are using Primary Key tables, this error often occurs when the persistent index (

index.l0.*

files) points to a rowset or segment that has been deleted by compaction or garbage collection, or was never successfully written due to disk/network issues. 2. Version Lag & Metadata Mismatch: If a BE (Backend) fails to publish a version correctly, it may fall into a

VERSION_INCOMPLETE

state. The FE (Frontend) might think a version is ready, but the BE is missing the corresponding segment files. 3. Specific Version Bugs: Several bugs related to

rssid

(Rowset Segment ID) and "file not found" errors have been fixed in recent releases: • Fixed in 3.5.14+: A bug causing "Iterator undefined behavior when rssid is not found" (#69617). • Fixed in 3.4.x / 3.3.x: Issues where

invalid rssid

errors occurred during column-mode partial updates or automatic partition creation (#66139, #66217, #67419). Troubleshooting Steps 1. Identify the Affected Tablets and Backends Check the BE logs for the specific

tablet_id

and the "No such file or directory" error. You can also run the following SQL to find inconsistent tablets:

sql
-- Check for tablets where the version lags behind the partition's version
SELECT * FROM information_schema.be_tablets 
WHERE tablet_id = <YOUR_TABLET_ID>;

-- Check the partition status
SELECT * FROM information_schema.partitions_meta 
WHERE partition_name = '<YOUR_PARTITION_NAME>';

2. Check Stuck Transactions Use the following commands to see if transactions are stuck in

COMMITTED

state:

sql
SHOW PROC '/transactions';
-- Find the txn_id and look at the "State" and "ErrorMsg" columns.

3. Resolution Strategies * Mark Replicas as Bad: If only one or two replicas are affected, you can mark them as

BAD

. StarRocks will automatically delete them and clone healthy versions from other replicas.

sql
    ADMIN SET REPLICA STATUS PROPERTIES("tablet_id" = "XXX", "backend_id" = "YYY", "status" = "bad");

* Restart BE/FE: In some cases, a metadata cache inconsistency can be cleared by restarting the problematic BE. If the issue is cluster-wide, a rolling restart of BEs and then the FE may be necessary. * Persistent Index Repair: For Primary Key tables, if the error persists, you may need to clear the persistent index for the affected tablet by deleting the

index

directory inside the tablet's storage path on the BE (ensure you have backups or other healthy replicas first). * Upgrade: If you are on a version earlier than 3.5.14 or 3.3.10, consider upgrading to include critical fixes for

rssid

handling and transaction publishing. If these steps do not resolve the "wedged" state, check if your BE disks are full, as this frequently causes truncated segment files that lead to these ID errors. Sources: * StarRocks FAQ - Loading Issues * StarRocks Release Notes (v3.5.14) References • faq/loading/Loading_faq.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • sql-reference/Error_code.md

Running in Shared-data mode with Primary-Key tables. Running version 4.1.14