@Rocky Our starrocks queries are failing with version mismatch. Can anyone faced similar issue. We think issue caused due to unstable/flapping FE nodes crashing one after another and pushed version lag in FE nodes.

1. Initial Trigger & FE Instability

Heap Spike: The incident began yesterday with high heap utilization alerts on the FE leader node. The root cause of this initial heap surge remains under investigation.
Leader Flapping: Whichever FE node became leader experienced runaway heap usage attempting to replay pending transactions, causing repeated leader crashes and failovers across 3 FE nodes.
FE Adjustments Made:
Increased FE heap from 31GB to 48GB and enabled -XX:ObjectAlignmentInBytes=16 (to retain Compressed OOPs above 32GB).
Reduced iceberg_metadata_memory_cache_expiration_seconds from 1 day to 300s and persisted it in fe.conf.
These changes stabilized the FE processes and stopped the crash loop.

2. Backend & Tablet Recovery

Unhealthy Tablets: 269 unhealthy tablets were detected across the cluster. Rolling restarts of BE nodes progressively reduced this count to 3.
Partition Cleanup: The remaining 3 stuck tablets belonged to partition p2026090200 across 3 tables. After verifying data exists in Iceberg and is not actively queried, these partitions were dropped, bringing the unhealthy tablet count to 0.
3. Current Issues & Technical Findings
Although the FEs are stable and unhealthy tablet count is 0, queries and Routine Loads continue to fail due to version desynchronization between FE catalog metadata and BE storage introduced during the rapid FE leader restarts.
Multiple tables are impacted across the cluster; the exact number of affected tables is still being inventoried. The primary active failure modes observed are:

Query Failures on Tables with Inconsistent Metadata:
Queries fail when FE requests an older catalog version that BE has already compacted away: > SQL Error [1064] [42000]: capture_consistent_versions error: version already been compacted. tablet_id: 388308574, version: 46468 tablet_max_version:46469
Queries also fail during scan planning due to invalid metadata version state: > SQL Error [1064] [42000]: Build Exec OlapScanNode fail, scan info is invalid

The error

capture_consistent_versions error: version already been compacted

typically occurs when the Frontend (FE) attempts to query a specific data version of a tablet that the Backend (BE) has already merged (compacted) and deleted. This is common after FE leader flapping because the FE's metadata (stored in the BDBJE image/journals) can become desynchronized or lag behind the actual state of the data on the BEs. If the BEs continued to perform compactions while the FEs were unstable, the FE might still think an older version is the "latest visible version," but the BE no longer has the metadata for that specific version. Recommended Fix: Metadata Recovery Mode According to StarRocks documentation, the standard way to resolve this specific version mismatch is to use Metadata Recovery Mode. This mode allows the FE to re-synchronize its metadata with the actual state of the tablets on the BEs. 1. Stop all FE nodes. 2. Backup your metadata directory (

meta_dir

) on all FE nodes as a precaution. 3. Add the following configuration to

fe.conf

on all FE nodes:

properties
    metadata_enable_recovery_mode = true

4. Start the FE nodes. _ _Note: In this mode, data loading is typically disabled or restricted.* 5. Monitor the recovery process. Connect to the Leader FE and run:

sql
    SHOW PROC '/meta_recovery';

This will show partitions that are still being recovered. If the result is empty, the recovery is complete. 6. Force a new metadata image. Once recovery is finished, execute:

sql
    ALTER SYSTEM CREATE IMAGE;

Wait for the new image file to be generated and synchronized to the

meta/image

directory of all FE nodes. 7. Exit Recovery Mode. * Stop the FE nodes. * Remove

metadata_enable_recovery_mode = true

from

fe.conf

. * Restart the FE nodes. Alternative Configuration Adjustments If the cluster is a Shared-data (Cloud-native) cluster, you can also try increasing the retention period to prevent BEs from vacuuming old versions too quickly while the FEs are catching up: * Modify the FE configuration:

lake_autovacuum_grace_period_minutes

(default is 5). Increasing this to a higher value (e.g., 60) can give the FE more time to "see" versions before they are deleted. For Routine Loads If Routine Loads continue to fail after metadata recovery, you may need to resume or restart them manually:

sql
RESUME ROUTINE LOAD FOR <job_name>;

If the job's internal state (offsets vs versions) is corrupted, you might need to

STOP

and re-create the Routine Load job. Important Note on Heap Usage Since the root cause was a heap spike, ensure that after recovery you monitor the

Non-Heap

and

Heap

memory closely. The

ObjectAlignmentInBytes=16

change you made is a good step for larger heaps, but keep an eye on the

fe.gc.log

to ensure the "replay pending transactions" phase doesn't trigger another surge during the recovery startup. References • administration/Meta_recovery.md • faq/shared_data_faq.md