http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> After loading data, one of our thre...
# questions-and-troubleshooting
j
@Rocky After loading data, one of our three FE nodes (one of the followers) is returning a different COUNT(*) result than the other two. How should I go about diagnosing this?
Copy code
-- leader
mysql> SELECT COUNT(*) FROM lineitem;
+-----------+
| count(*)  |
+-----------+
| 600037902 |
+-----------+

-- follower 1
mysql> SELECT COUNT(*) FROM lineitem;
+-----------+
| count(*)  |
+-----------+
| 600037902 |
+-----------+

-- follower 2
mysql> SELECT COUNT(*) FROM lineitem;
+-----------+
| count(*)  |
+-----------+
| 592304126 |
+-----------+
All three FE nodes appear healthy (the issued fe node is fe-1)
Copy code
mysql> SHOW FRONTENDS\G
*************************** 1. row ***************************
               Id: 1
             Name: kitty-starrocks-fe-0.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local_9010_1775202123840
               IP: kitty-starrocks-fe-0.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 567682858
             Join: true
            Alive: true
ReplayedJournalId: 1700512
    LastHeartbeat: 2026-04-28 16:34:21
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-24 16:28:23
          Version: 4.0.8-2090bfc
*************************** 2. row ***************************
               Id: 2
             Name: kitty-starrocks-fe-2.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local_9010_1775202137364
               IP: kitty-starrocks-fe-2.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: LEADER
        ClusterId: 567682858
             Join: true
            Alive: true
ReplayedJournalId: 1700521
    LastHeartbeat: 2026-04-28 16:34:21
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-24 16:26:26
          Version: 4.0.8-2090bfc
*************************** 3. row ***************************
               Id: 3
             Name: kitty-starrocks-fe-1.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local_9010_1775202137629
               IP: kitty-starrocks-fe-1.kitty-starrocks-fe-search.kitty-starrocks.svc.cluster.local
      EditLogPort: 9010
         HttpPort: 8030
        QueryPort: 9030
          RpcPort: 9020
             Role: FOLLOWER
        ClusterId: 567682858
             Join: true
            Alive: true
ReplayedJournalId: 1700512
    LastHeartbeat: 2026-04-28 16:34:21
         IsHelper: true
           ErrMsg:
        StartTime: 2026-04-24 16:27:31
          Version: 4.0.8-2090bfc
3 rows in set (0.01 sec)
r
Sorry, I encountered an error while processing your request: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Resource has been exhausted (e.g. check quota).', 'status': 'RESOURCE_EXHAUSTED'}}
j
@Rocky can you answer above question?
r
This behavior typically indicates metadata synchronization lag or a journal replay issue between the Leader FE and the Followers. From your 
SHOW FRONTENDS
output, there is a clear discrepancy in the 
ReplayedJournalId
: * Leader (FE-2): 1700521 * Follower 0 (FE-0): 1700512 * Follower 1 (FE-1): 1700512 The followers are 9 journals behind the leader. While 9 journals may seem small, if those journals contain the 
COMMIT
records for a large data load (like the ~8 million row difference you see), the followers will not "see" that data until those journals are replayed. Recommended Diagnostic Steps 1. Check for Replay Lag and Errors Even if the 
ReplayedJournalId
is moving, it might be stuck or replaying very slowly due to an error. * Action: On the "bad" FE node (
FE-1
), check the 
fe.log
(located in your StarRocks 
log/
directory). * Search for: Keywords like 
catch up
, 
replay
, 
BDBJE
, or 
Exception
. * Verify: If you see 
ReplicaWriteException
or 
RollbackException
, the metadata on that follower might be inconsistent with the leader. 2. Verify Metadata State on Each FE The 
COUNT(*)
result is often derived from the FE's cached metadata about tablet row counts. You can compare how each FE sees the table's version: * Connect to each FE individually and run:
Copy code
sql
    SHOW TABLE STATUS LIKE 'lineitem';
    -- Check if 'Create_time' or version-related info differs.

    SHOW PARTITIONS FROM lineitem;
    -- Compare the 'VisibleVersion' and 'RowCount' columns across FEs.
If the 
VisibleVersion
on the lagging follower is lower than the leader's, it confirms that the follower hasn't processed the latest transaction. 3. Monitor the Replay Thread If the 
ReplayedJournalId
(1700512) stays the same for several minutes while the Leader's ID increases, the replay thread is likely stuck. * Check 
fe.conf
for the parameter 
meta_delay_toleration_second
(default is 300s). If the lag exceeds this, the follower should eventually stop serving queries or report as 
Alive: false
. However, if the lag is small but includes a critical transaction, you get the result discrepancy you're seeing. 4. Force a Metadata Refresh (Safest First Step) If the node is healthy but just lagging, restarting the FE service on the lagging node usually forces it to re-sync with the leader: 1. Stop the "bad" FE node. 2. Clear the 
meta/bdb
directory if you suspect corruption (only as a last resort, and back it up first!). 3. Restart the FE. It will catch up with the Leader's journals upon startup. Why does one follower match while the other doesn't? In your 
SHOW FRONTENDS
, both followers show the same 
ReplayedJournalId
(1700512). If "Follower 1" matches the leader's count but "Follower 2" doesn't, yet they have the same ID, it suggests that FE-1 might have failed to apply a specific journal entry correctly or its internal row-count cache is stale. Key Command to run on the Leader:
Copy code
sql
ADMIN CHECK TABLET (lineitem);
This will verify if the BEs and FEs agree on the state of the data replicas. Summary: The 9-journal gap is the smoking gun. Focus on why 
FE-1
is trailing the Leader and whether its 
ReplayedJournalId
is advancing over time. If it's not moving, restart that FE-1. References • administration/management/FE_configuration.md • faq/Deploy_faq.md • deployment/deploy_manually.md • administration/Meta_recovery.md
j
@Rocky I don't think so. The same behavior was observed even when the JOURNAL_ID was in sync. It wasn't even resolved after the problematic Follower was promoted to Leader. In other words, only the Leader ended up returning the abnormal values.
r
Since your nodes appear healthy and the issue persists even after promoting the problematic Follower to Leader, this is almost certainly an inconsistency in the FE metadata cache or a bug related to the Meta Scan optimization rather than a simple journal replay lag. In StarRocks 4.x, 
COUNT(*)
queries are often optimized to return results directly from the FE's metadata (without scanning data on BEs). If one FE's metadata is "stale" or calculated incorrectly, it will return a different result than the others. 1. Verify if it's a "Meta Scan" issue StarRocks uses a rewrite rule called 
enable_rewrite_simple_agg_to_meta_scan
. When this is enabled (default), 
COUNT(*)
reads the 
ROW_COUNT
stored in the FE's memory. Test: Run the query on the "bad" node with the optimization disabled:
Copy code
sql
SELECT /*+ SET_VAR(enable_rewrite_simple_agg_to_meta_scan=false) */ COUNT(*) FROM lineitem;
* If the result is now correct: The data on the BEs is fine, but the FE's internal metadata for this table is wrong. * If the result is still wrong: There is a deeper data consistency issue between the replicas on the BEs. 2. Compare Partition Metadata Check exactly what each FE "thinks" the row count is by querying the system schema. Run this on all three FE nodes and compare the output:
Copy code
sql
SELECT PARTITION_NAME, VISIBLE_VERSION, ROW_COUNT 
FROM information_schema.partitions_meta 
WHERE TABLE_NAME = 'lineitem';
Look for discrepancies in 
VISIBLE_VERSION
or 
ROW_COUNT
. If the 
VISIBLE_VERSION
is the same across all nodes but 
ROW_COUNT
differs, the FE's cached metadata for those partitions has diverged. 3. Check for Known Bugs in 4.0.8 Your version (
4.0.8-2090bfc
) has a few known issues regarding metadata consistency: * Issue #66852: Incorrect row count calculation in the Meta Scan rewrite rule. * Issue #69428: Materialized view and tablet metadata inconsistency between Leader and Follower. These were specifically addressed in v4.0.9. If you are hitting these, the FE calculates the row count during the image-loading or journal-replay phase and caches a "bad" value. 4. Diagnostic & Repair Steps Step A: Tablet Consistency Check Run this on the Leader to ensure the Backend replicas are in sync:
Copy code
sql
ADMIN CHECK TABLET (lineitem) PROPERTIES ("type" = "consistency");
-- Then check the result:
SHOW PROC "/statistic";
If 
InconsistentTabletNum
is greater than 0, some BE replicas have different data. Step B: Force a Metadata Refresh If the BEs are fine but the FE metadata is wrong, you can try to "nudge" the FE to update its counts: 1. Analyze the table: This can sometimes force a refresh of the statistics and metadata.
Copy code
sql
    ANALYZE TABLE lineitem;
2. Create a new Metadata Image: On the Leader, force the system to write a fresh snapshot of the metadata:
Copy code
sql
    ALTER SYSTEM CREATE IMAGE;
Wait a few minutes for the image to be generated and synced to followers. Step C: The "Clean" Re-sync (If persistent) If the bad metadata is "stuck" in the BDBJE logs of the problematic FE: 1. Stop the problematic FE node. 2. Back up and then clear its 
meta/
directory. 3. Restart it using the Leader as a helper:
Copy code
bash
    ./bin/start_fe.sh --helper <leader_ip>:9010 --daemon
This forces the node to pull a fresh, full metadata image from the Leader, bypassing any local "dirty" state. Summary Recommendation: First, disable 
enable_rewrite_simple_agg_to_meta_scan
to confirm the FE metadata is the culprit. If so, plan an upgrade to v4.0.9+ to resolve the underlying bug. References • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • knowledge/usage/best-practice.md
j
@Rocky This looks like a Meta Scan issue. After disabling it with SET_VAR(enable_rewrite_simple_agg_to_meta_scan=false), the query returns the correct result. Also, the partition metadata is identical across FEs, so the discrepancy does not seem to come from partition-level metadata.
r
This is a classic symptom of metadata inconsistency in the FE's in-memory representation, specifically affecting the "Meta Scan" optimization. In StarRocks 4.x, the optimizer defaults to 
enable_rewrite_simple_agg_to_meta_scan=true
. When this is on, a 
SELECT COUNT(*)
doesn't scan the data on the BEs; instead, it sums up the 
rowCount
values stored in the FE's memory. Since your query returns the correct result with this setting disabled, we know the data on the BEs is correct, but the FE's internal cache is stale or corrupted. The fact that this persisted even after the follower was promoted to Leader suggests that the "bad" state is either persisted in that node's local image/journals or there is a logic bug in how that specific node is aggregating the metadata. Why does this happen? Even if 
information_schema.partitions_meta
looks identical (because it's a view that calculates values on the fly), the Meta Scan rewrite uses an internal aggregate of the table's "Materialized Index" row counts. If the FE failed to update the row count for a specific index or partition during a load (a known issue in early 4.0.x versions), the Meta Scan will be off. Recommended Steps to Resolve 1. Force a Statistics Update Sometimes the internal metadata counters aren't updated until a full analysis is triggered. Run this on the Leader:
Copy code
sql
ANALYZE TABLE lineitem;
After the analysis completes, check if the 
COUNT(*)
(with Meta Scan enabled) is corrected. 2. Create a Fresh Metadata Image Force the cluster to write a new checkpoint image. This forces all nodes to serialize their current state to disk and can sometimes "clean up" transient in-memory inconsistencies:
Copy code
sql
-- Run on the Leader
ALTER SYSTEM CREATE IMAGE;
Wait a few minutes for the image to sync to the other FEs, then check the count again. 3. The "Clean Resync" (The most reliable fix) If the metadata on the "bad" node is truly out of sync and survived promotion, you need to wipe its local state and force it to inherit the state from a "good" node: 1. Stop the problematic FE node. 2. Back up and then delete the contents of its 
meta/
directory (specifically 
meta/bdb
and 
meta/image
). 3. Restart the FE node using a healthy node as a helper:
Copy code
bash
    ./bin/start_fe.sh --helper <healthy_fe_ip>:9010 --daemon
This forces the node to pull a fresh, full metadata image from the healthy node, bypassing any local corruption. 4. Upgrade to v4.0.9+ Your version (
4.0.8
) is susceptible to a few documented Meta Scan and partition metadata bugs (such as #66967 and #69428). These were addressed in v4.0.9. If this issue recurs after a fresh resync, an upgrade is strongly recommended. Summary Checklist * Is it just Meta Scan? Yes (confirmed by 
SET_VAR
). * Is the BE data safe? Yes (confirmed by the correct count when disabling Meta Scan). * Immediate Fix: Perform a Clean Resync (Step 3) on the node that originally showed the bad data. * Permanent Fix: Upgrade to v4.0.9 or later. References • release_notes/release-4.0.md
j
@Rocky Running 
ANALYZE
and 
ALTER SYSTEM CREATE IMAGE
both failed, but the issue was resolved after restarting the FE nodes. Note that the 
meta/
directory was not deleted. In this situation, what would be the most likely root cause of the issue?
r
Based on your observation that the issue was resolved by a simple restart (without deleting the 
meta/
directory) and that maintenance commands like 
ANALYZE
and 
CREATE IMAGE
were failing, the most likely root cause is a transient in-memory metadata inconsistency combined with a lock contention/deadlock in the FE's internal state. Here is the breakdown of why this occurred: 1. In-Memory "Catalog" Divergence StarRocks FEs maintain an in-memory "Catalog" that caches table, partition, and tablet metadata (including row counts). The Meta Scan optimization relies entirely on this in-memory cache. * The Bug: In version 4.0.8, there are known edge cases (e.g., Issue #66967) where the FE fails to correctly update these in-memory row count counters during specific load scenarios (like partial updates or frequent small loads). * Why a restart fixed it: When you restart the FE, it wipes its memory and re-constructs the entire Catalog by replaying the BDBJE journals from scratch. Since the journals themselves contained the correct transaction records, the fresh in-memory state became accurate again. 2. Why 
ANALYZE
and 
ALTER SYSTEM CREATE IMAGE
Failed The failure of these two specific commands is the "smoking gun" for a Metadata Lock issue. * Lock Contention: Both 
ANALYZE
(which updates statistics) and 
CREATE IMAGE
(which serializes the entire metadata state to a file) require a Global Metadata Read Lock. * The Conflict: If a background thread (like a 
Publish Version
task or a 
Tablet Report
processor) was hung or stuck in a loop while holding a write lock, any subsequent metadata operations would fail with a timeout or an internal exception. This explains why you couldn't fix the metadata via `ANALYZE`—the command couldn't get the lock it needed to update the very cache that was wrong. 3. The "Meta Scan" Optimization Flaw The reason the 
COUNT(_)
was wrong while 
SELECT _
(or disabling Meta Scan) was right is that they use different code paths: * Normal Query: The FE generates a plan that sends a request to the BEs to scan actual data rows. * Meta Scan: The FE short-circuits the query and simply sums up the 
rowCount
variables it has in memory. If the memory variable is corrupted due to a software bug but the journal entry is correct, the "Meta Scan" will show the error while the actual data remains perfectly safe. Most Likely Root Cause Summary A race condition or logic bug in the metadata update thread (likely related to high-concurrency loads) caused the FE's in-memory row-count cache to drift from reality. This same bug likely caused a "stuck" metadata lock, preventing maintenance commands from running. Because the BDBJE journals on disk were still correct, a simple restart was sufficient to "re-sync" the memory to the disk state. Recommendation Since you are on v4.0.8, you are hitting a known stability window for the new Meta Scan logic. * Immediate Fix: You've already done it (FE restart). * Long-term Fix: Upgrade to v4.0.9 or later. This version includes specific fixes for: * `#66967`: Incorrect row count calculation in the Meta Scan rewrite rule. * `#69428`: Metadata inconsistency between Leader and Follower during image generation. * Improved locking mechanisms for 
ALTER SYSTEM CREATE IMAGE
. References • release_notes/release-4.0.md • release_notes/release-3.4.md
Open in Slack
PreviousNext
Powered by