http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I continue to get ```"1064 (HY000):...
# questions-and-troubleshooting
g
@Rocky I continue to get
Copy code
"1064 (HY000): starlet err Object <path to .dat> does not exist: BE:2159792",
fairly frequently. I upgraded from 4.0.9 from 4.0.2 at your recommendation as a fix but the problem remains. What additional information do you need to help me diagnose and resolve?
r
To help you diagnose and resolve the 
starlet err Object <path> does not exist
 error, we need to pinpoint whether this is a metadata inconsistency, a compaction issue, or an underlying storage problem. This error typically occurs in StarRocks Shared-data (Cloud-native) clusters when the storage engine (Starlet) attempts to access a data file (
.dat
) or an index file in the remote object storage (e.g., S3, OSS) that is no longer there, or the metadata is pointing to an incorrect location. Information Needed for Diagnosis Please provide the following details to help us investigate: 1. Cluster & Table Details: * Storage Backend: What object storage are you using (S3, Minio, OSS, etc.)? * Table Model: Does this happen on a Primary Key table? * Index Configuration: Is 
enable_persistent_index
enabled for this table? (Persistent index files are often stored as objects and can trigger this if they go missing). 2. Scenario Context: * Trigger: Does this error occur during a Query, an INSERT/Load, or during Compaction (check 
fe.log
for publish version failures)? * Frequency: Is it always the same table/partition, or does it happen randomly across the cluster? 3. Logs (Crucial): * BE Logs: Search the 
be.WARNING
or 
<http://be.INFO|be.INFO>
logs on the BE node mentioned in the error (e.g., the node where the error originated) for the string 
Object does not exist
or 
fail to get object
. Look for messages immediately preceding the error to see if there was a failed compaction or a "publish version" timeout. * FE Logs: Check 
fe.log
and 
fe.warn.log
for any "Publish version failed" errors related to the table ID. 4. Metadata vs. Physical Storage Check: * Verify File Existence: Manually check your object storage (S3/OSS) for the specific path mentioned in the error. Does the file actually exist? * Audit External Cleanup: Are there any external Lifecycle Policies (TTL) on your S3 bucket that might be deleting files StarRocks still needs? 5. Cluster Status: * Output of 
SHOW PROC '/backends'\G
to see if any BEs are in an unstable state. Potential Immediate Steps to Try * Clear Data Cache: Sometimes the StarRocks Data Cache (StarCache) might have an entry for a file that was deleted or moved. You can try clearing the cache or disabling it temporarily (
SET GLOBAL enable_scan_datacache = false;
) to see if queries still fail. * Persistent Index Check: If you are using Primary Key tables with persistent indexes, the index files can sometimes become corrupted or lost. In some cases, recreating the index or the table is a workaround, but we should identify the root cause first. Note: Since you've upgraded to a very recent version (4.0.x in your context), it's possible this is a regression or a specific interaction with your storage provider's consistency model. Providing the logs mentioned above is the fastest way to narrow this down. References • administration/management/BE_configuration.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md
g
@Rocky • storage backend azure blob • primary key table yes • "persistent_index_type" = "CLOUD_NATIVE" • happens during a query, will be resolved after partition is dropped. only happens on specific table though. Gonna dig into the logs now
r
Since you have already upgraded to 4.0.9 (which includes several critical fixes for cloud-native cleanup and metadata handling) but are still encountering the 
starlet err Object does not exist
error, the issue is likely related to Garbage Collection (AutoVacuum) timing or a metadata-to-storage inconsistency specific to your Azure Blob environment. In StarRocks shared-data (cloud-native) architecture, this error occurs when the Starlet storage engine attempts to read a data file (
.dat
) that the tablet metadata says should exist, but the physical file has been deleted from Azure Blob. To help us pinpoint the root cause, please provide or check the following: 1. Key Configuration Check: 
lake_autovacuum_grace_period_minutes
 The most common cause for this error is that the AutoVacuum process is too aggressive. When StarRocks compacts data, it creates new rowsets and marks old ones for deletion. If a query is still running on an old version, or if a BE node has stale metadata, and AutoVacuum deletes the physical files before they are no longer needed, you get this error. * Check: Run 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum_grace_period_minutes';
* Recommendation: If it is set to a low value (e.g., 5 or 10), try increasing it to 60 or 120 to see if the frequency drops. * 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
2. Identify the Missing Object When the error occurs, please capture the exact path of the missing object from the error message. * Verification: Manually check your Azure Blob container for that specific path. Does the file exist? * If it DOES NOT exist: StarRocks GC/Vacuum deleted it prematurely. This points to 
lake_autovacuum_grace_period_minutes
or a bug in the versioning logic. * If it DOES exist: This indicates a connectivity, permission, or protocol issue (e.g., StarRocks is looking for the file but Azure is returning a 404/403 due to path encoding or configuration). 3. Tablet & Metadata Status Since dropping the partition resolves the issue, it suggests the partition's metadata was stuck pointing to a "stale" file. * When the error occurs again (before dropping the partition), please run: * 
SHOW TABLET FROM <table_name> WHERE TabletId = <tablet_id_from_logs>;
* 
SHOW PROC '/dbs/<db_id>/<table_id>/partitions/<partition_id>/<index_id>/<tablet_id>';
* This will help us see if there are an excessive number of versions or if a specific rowset is orphaned. 4. BE/CN Logs for Vacuum Tasks Search the 
<http://be.INFO|be.INFO>
or 
be.WARNING
logs on the BE/CN node (especially the one mentioned in the error, e.g., 
BE:2159792
) for the following keywords around the time of the error: * 
Vacuum
* 
GC
* 
delete_tablet
* 
fail to get object
* Look for: Any logs indicating that a vacuum task failed or that it successfully deleted files that were still in use. 5. Azure Storage Specifics * Lifecycle Policies: Do you have any Azure Storage Lifecycle Management policies (TTL) enabled on the container that might be automatically deleting or moving files to "Cool" or "Archive" storage? * Storage Volume Config: How is your storage volume defined? (e.g., 
azblob://
vs 
wasbs://
). Please provide the 
CREATE STORAGE VOLUME
statement (redact the keys). Summary of what to provide next: 1. The value of 
lake_autovacuum_grace_period_minutes
. 2. The full path of a missing 
.dat
file from the error message. 3. Confirmation if that file actually exists in Azure Blob. 4. BE logs snippets showing 
Vacuum
or 
Starlet
errors. 5. Is the table under high concurrent write/compaction load? Note: If you are using Persistent Index (
CLOUD_NATIVE
), the SST files are also stored in the cloud. However, since your error specifically mentions a 
.dat
file, we are looking at data rowsets rather than index files. References
g
@Rocky what helpful information should I be trying to get from 
SHOW PROC '/dbs/<db_id>/<table_id>/partitions/<partition_id>/<index_id>/<tablet_id>';
r
The 
SHOW PROC
command at the tablet level provides a granular look at the Rowsets that currently make up that specific tablet's data. When you are troubleshooting a "starlet err Object does not exist" error, this output is your primary tool to determine if there is a mismatch between what StarRocks metadata thinks exists and what is actually in your Azure Blob storage. When you run: 
SHOW PROC '/dbs/<db_id>/<table_id>/partitions/<partition_id>/<index_id>/<tablet_id>';
You should focus on the following columns and information: 1. Rowset ID Correlation * What to look for: Extract the Rowset ID from the error message. The path in your error likely looks like 
.../data/<tablet_id>/<rowset_id>_<segment_index>.dat
. * The Check: Look for that exact 
RowsetId
in the 
SHOW PROC
output. * If the RowsetId is present: StarRocks metadata believes this rowset is part of the current active version of the tablet. If the file is missing from Azure, your AutoVacuum or an external process deleted it prematurely. * If the RowsetId is NOT present: This means the rowset has already been compacted away or deleted from metadata, but your query is still trying to read it. This usually points to a stale metadata cache on the BE/CN node or a query holding a very old version. 2. Version and CreationTime * The Check: Look at the 
Version
and 
CreationTime
columns for the rowsets. * Significance: * If the "missing" rowset has a very old 
CreationTime
, it should have been cleaned up, but a long-running query or a stuck metadata update might be keeping it "alive" in the query plan. * If it has a very recent 
CreationTime
(e.g., minutes ago), and it's already missing, there may be a race condition between the Publish Version phase and the storage backend's consistency (Azure Blob). 3. Number of Rowsets (Compaction Health) * What to look for: The total number of rows in the output. * Significance: If you see hundreds of rowsets for a single tablet, it indicates that Compaction is lagging. High rowset counts increase the chance of metadata inconsistencies and make the "Object does not exist" error more likely during frequent updates. 4. Segment Count * What to look for: The 
SegmentNum
or 
NumSegments
column. * The Check: If the error says 
segment_2.dat
does not exist, but 
SHOW PROC
says the rowset only has 1 segment, there is a serious metadata corruption. If it says it has 3 segments, but only 2 are in Azure, then physical data loss has occurred. ────────── Diagnosis Summary Table If Rowset ID in 
SHOW PROC
... | And File in Azure... | Diagnosis Is Present | Is Missing | Physical Deletion/GC Issue: AutoVacuum deleted the file too early. Increase 
lake_autovacuum_grace_period_minutes
. Is Missing | Is Missing | Metadata Lag: The query is using an old version of the tablet. Try 
SET GLOBAL enable_tablet_report_worker = true
or check for FE-to-BE communication lag. Is Present | Is Present | Access/Network Issue: The file exists, but the BE node cannot reach it (Check Azure credentials/permissions or network timeouts). Next Step Once you have the 
SHOW PROC
output, compare the Rowset IDs listed there against the file path mentioned in your error. If you find the Rowset ID is listed in StarRocks but missing in Azure, please check your 
lake_autovacuum_grace_period_minutes
as mentioned previously. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md
g
@Rocky I don't see anything about rowset id in SHOW PROC
r
Apologies for the confusion. You are correct—in StarRocks Shared-data (Cloud-native) clusters, the 
SHOW PROC
output at the tablet level differs from the traditional shared-nothing (OLAP) model. It does not show a flat list of "Rowsets" because data management is handled through tablet metadata files stored in your object storage. To see which rowsets and segments (the 
.dat
files) are currently active or referenced for a specific tablet, you need to inspect the tablet metadata. How to see Rowset/Segment details for Shared-data Since 
SHOW PROC
doesn't provide it, you can use the following administrative command to dump the tablet's metadata in JSON format. This will show you exactly which files the BE expects to find in Azure Blob. 1. Identify the Tablet and Version: First, get the visible version of your partition:
Copy code
sql
    SHOW PARTITIONS FROM <table_name>;
2. Execute the Metadata Dump: Run this on the BE/CN node where you are seeing the error (use the 
backend_id
from the error or from 
SHOW PROC '/backends'
):
Copy code
sql
    ADMIN EXECUTE ON <backend_id>
    'System.print(StorageEngine.get_lake_tablet_metadata_json(<tablet_id>, <version>))';
_Replace 
<tablet_id>
with the ID from your error and 
<version>
with the version found in step 1._ What to look for in the JSON Output: Inside the returned JSON, look for the 
rowsets
section. It will list the segments that StarRocks thinks should exist: * `segments`: This array contains the names/IDs of the 
.dat
files. * Correlation: Match the "Object does not exist" path from your error to the segments listed in this JSON. * If the file is in the JSON but missing in Azure: This is a Premature Deletion issue. The StarRocks Vacuum process deleted the file because it thought it was no longer needed, but the metadata (or a running query) still points to it. * If the file is NOT in the JSON: This is a Metadata Inconsistency. The query plan is likely using a stale version of the metadata. ────────── Why this happens and how to fix it (Azure Specifics) Given you are on Azure Blob and using Primary Key tables with Cloud-native Persistent Indexes, there are two likely causes for this persistent "Object does not exist" error: 1. AutoVacuum is too aggressive (Most Likely) When StarRocks compacts data, it marks old files for deletion. If a query is slow or a BE hasn't refreshed its metadata cache, it may try to read a file that the Vacuum process just deleted. * Fix: Increase the "grace period" for garbage collection. This gives the cluster more time to finish queries before files are physically removed from Azure.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
(Default is often 5-20 minutes; 120 is safer for debugging). 2. Azure Blob Consistency / Latency Azure Blob Storage is strongly consistent, but StarRocks' Starlet engine sometimes encounters "404 Not Found" if there's a race condition between a compaction "Publish" and the metadata update. * Check: When you get the error, immediately check your Azure Portal or use 
az storage blob exists
for that path. If the file is there, the issue is a transient network/permission error or a path encoding issue. 3. Persistent Index Mutation (Primary Key Tables) Since you use 
CLOUD_NATIVE
persistent indexes, StarRocks also stores index files in Azure. If the error mentions a path containing 
index/
instead of 
data/
, the issue is with the index cleanup logic. Summary for your next diagnostic step: Run the 
ADMIN EXECUTE... get_lake_tablet_metadata_json
command for the failing tablet. If the missing file is listed in that JSON, increase your 
lake_autovacuum_grace_period_minutes
 immediately, as that is the primary knob for resolving "Object does not exist" errors in version 4.0.x. References
g
@Rocky This is the actual object string (with some fields redacted), can you help me break it into its parts (tablet_id, partition_id, etc)
Copy code
https://<my_storage_account>.dfs.core.windows.net/<my_blob>/starrocks/<some_uuid>/db49953/925631/2169435/data/0000000000188f3e_fddc941b-ed82-435b-afd9-741169065fb5.dat does not exist: BE:2159792",
r
Based on the standard StarRocks shared-data (cloud-native) storage layout for Azure, here is the breakdown of your object string. The path follows this pattern: 
.../starrocks/<cluster_uuid>/db<db_id>/<table_id>/<partition_id>/data/<tablet_id_hex>_<rowset_uuid>.dat
Breakdown of your specific path: * Database ID: 
49953
(from 
db49953
) * Table ID: 
925631
* Partition ID: 
2169435
* Tablet ID (Hex): 
0000000000188f3e
_ _Decimal conversion:* 1,609,534 * Rowset UUID: 
fddc941b-ed82-435b-afd9-741169065fb5
* Reporting Backend ID: 
2159792
(This is the node that tried to read the file). ────────── How to use this information for debugging: 1. Verify the Tablet and Table names You can use the IDs to find exactly which table/partition is affected in your SQL console:
Copy code
sql
-- Find Table and DB name
SHOW TABLE FROM <db_name> WHERE TableId = 925631;

-- Find specific Partition name
SHOW PARTITIONS FROM <table_name> WHERE PartitionId = 2169435;

-- Get detailed info about the specific tablet
SHOW TABLET 1609534;
2. Check the Tablet Metadata Since you have the Tablet ID and the Reporting BE ID, you can dump the metadata that the BE is currently using to see why it thinks this specific 
.dat
file should exist:
Copy code
sql
ADMIN EXECUTE ON 2159792 
'System.print(StorageEngine.get_lake_tablet_metadata_json(1609534))';
*Look for the 
rowsets
array in the JSON output. If the Rowset UUID 
fddc...
is listed there but the file is missing from Azure, it confirms a Premature Deletion (AutoVacuum) issue.* 3. Search BE Logs for the Rowset On Backend 
2159792
, search the 
<http://be.INFO|be.INFO>
logs for that specific Rowset UUID to see when it was created or if there were any errors during its "Publish" phase:
Copy code
bash
grep "fddc941b-ed82-435b-afd9-741169065fb5" <http://be.INFO|be.INFO>
Immediate Mitigation Reminder As mentioned earlier, if you find that StarRocks is looking for files that were recently deleted, the most effective fix is to increase the vacuum grace period: 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
References • table_design/table_types/primary_key_table.md • faq/shared_data_faq.md • best_practices/table_clustering.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md
g
@Rocky that ADMIN EXECUTE gives me
Copy code
Runtime error: StorageEngine metaclass does not implement 'get_lake_tablet_metadata_json(_)'.
r
The error you encountered, 
StorageEngine metaclass does not implement 'get_lake_tablet_metadata_json(_)'
, is likely occurring because the command requires exactly two arguments: the 
tablet_id
and the 
version
. The 
(_)
in the error is a hint from the internal scripting engine that it couldn't find a version of that function that takes only one argument. In StarRocks Shared-data (Cloud-native), metadata is versioned. To see which files a tablet is tracking, you must specify which version of the metadata you want to inspect. Step 1: Get the Visible Version Run this to find the current 
VisibleVersion
for your partition:
Copy code
sql
-- Use the Table ID 925631 or Table Name
SHOW PARTITIONS FROM <table_name> WHERE PartitionId = 2169435;
Look at the VisibleVersion column (e.g., 
105
). Step 2: Run the Corrected Admin Command Now, use both the Tablet ID (decimal: 1609534) and the version you just found:
Copy code
sql
ADMIN EXECUTE ON 2159792 
'System.print(StorageEngine.get_lake_tablet_metadata_json(1609534, <visible_version>))';
_(Replace 
<visible_version>
with the number from Step 1)_. ────────── What to Analyze in the JSON Once you get the JSON output, look for the 
rowsets
array. For each rowset, you will see a list of 
segments
. * Case A: The Rowset UUID from your error (
fddc941b...
) is in the JSON. * Meaning: StarRocks metadata correctly points to this file, but the file is physically missing from Azure Blob. * Root Cause: The AutoVacuum process deleted the file before all queries finished or before the BE metadata cache was updated. * Fix: Increase the grace period to prevent premature deletion:
Copy code
sql
        ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
* Case B: The Rowset UUID is NOT in the JSON. * Meaning: The BE node (2159792) is using a stale version of the metadata. It thinks it needs an old file that has already been legally compacted and vacuumed. * Root Cause: Metadata synchronization lag between the FE (Leader) and this specific BE. * Fix: Check the BE's network connectivity to the FE or check for high CPU/IO on the BE that might be delaying metadata heartbeats. Manual Path Check in Azure Using the IDs from your error string, you can also check if the metadata file itself exists in your Azure Blob storage. This file tells StarRocks what the tablet looks like at a specific version: * Path Pattern: 
.../starrocks/<cluster_uuid>/db49953/925631/2169435/metadata/1609534_<version>.meta
If you find that many 
.meta
files are missing for current versions, it points to a broader storage permissions or Azure connectivity issue. References • faq/shared_data_faq.md • deployment/shared_data/azure.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • administration/management/FE_configuration.md References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • deployment/shared_data/azure.md
g
@Rocky the output
Copy code
result                                                                               |
-------------------------------------------------------------------------------------+
Not found: starlet err grpc.GetShard(shardId=1609534) error: shard 1609534 not exist.|
be/src/storage/lake/tablet_manager.cpp:616 value_or_err_L616                         |
r
This "shard not exist" error is a very specific Starlet (the storage engine layer for StarRocks Cloud-native) error. It means that while you are asking the Backend (BE) to dump metadata for Tablet 
1609534
, the Starlet service on that BE has no record of this tablet in its local registry. This is unexpected because your original error message explicitly mentioned 
BE:2159792
was trying to read a file for that tablet. Here is how to interpret this and the next steps to fix it: 1. Why "Shard Not Exist"? In StarRocks Shared-data clusters, tablets are referred to as "shards" at the storage layer. For a BE to provide metadata or read data, it must first "load" the shard information dispatched by the Frontend (FE). * The tablet was dropped: If the table or partition was dropped/truncated between the time of the error and your command, the shard is removed from the BE. * BE-FE Desync: The FE is the source of truth. If the FE has re-assigned this tablet to a different BE (e.g., during a node restart or load balance), BE 
2159792
will have purged its local "shard" info. * Incorrect Backend: Even if BE 
2159792
reported the initial error, the tablet's metadata might now be managed by a different node. 2. Verify the Tablet's Current "Home" Run this command to see which Backend the FE currently thinks is responsible for this tablet:
Copy code
sql
SHOW TABLET 1609534;
Look for the 
BackendId
 column in the result. * If the BackendId is different from 
2159792
, run your 
ADMIN EXECUTE
command on that new ID instead. * If the result is empty, the tablet no longer exists (it was dropped). 3. Check for "Stale" Metadata on the FE If 
SHOW TABLET
confirms the Backend is 
2159792
, but the BE still says "shard not exist," it means the BE's internal Starlet state is corrupted or stuck. Try to "Force" a metadata reload: You can often force the BE to reload the shard by simply attempting to query a small amount of data from that specific partition:
Copy code
sql
SELECT * FROM <table_name> PARTITION (<partition_name>) LIMIT 1;
After running this query, try the 
ADMIN EXECUTE
command again. The query forces the BE to talk to the FE, fetch the shard info, and register it with Starlet. 4. Alternative: Use the FE-level Metadata Check If the BE-level command keeps failing, you can check if the metadata file exists directly in your Azure storage. Based on the path you provided earlier, the metadata for a specific version should be at: 
https://<account>.<http://dfs.core.windows.net/<blob>/starrocks/<uuid>/db49953/925631/2169435/meta/1609534_<version>.meta|dfs.core.windows.net/<blob>/starrocks/<uuid>/db49953/925631/2169435/meta/1609534_<version>.meta>
* If the 
.meta
file is missing in Azure: You have a critical metadata loss. This usually happens if a 
DROP TABLE...FORCE
or a manual Azure deletion occurred. * If the 
.meta
file exists but BE says "shard not exist": This is a communication/caching issue between FE and BE. Restarting the BE node 
2159792
will force it to clear its Starlet cache and re-sync with the FE. Summary Recommendation 1. Run 
SHOW TABLET 1609534
to confirm the BackendId. 2. If the BackendId matches, restart that BE node. 3. Once the BE is back online, immediately run:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
This will stop the "Object does not exist" errors from getting worse while you stabilize the cluster. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • faq/Sql_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • administration/management/BE_configuration.md
g
@Rocky 
SHOW TABLET 1609534
does not show any backend info, just
Copy code
Name         |Value                                           |
-------------+------------------------------------------------+
DbName       |                                                |
TableName    |                                                |
PartitionName|                                                |
IndexName    |                                                |
DbId         |-1                                              |
TableId      |-1                                              |
PartitionId  |-1                                              |
IndexId      |-1                                              |
IsSync       |false                                           |
DetailCmd    |SHOW PROC '/dbs/-1/-1/partitions/-1/-1/1609534';|
r
The fact that 
SHOW TABLET 1609534
returns 
-1
for all IDs (DbId, TableId, etc.) is a definitive signal: the Frontend (FE) no longer has this tablet in its metadata. This usually happens because the table or partition was dropped, truncated, or the database was deleted. The error you see on the Backend (BE) occurs because that specific BE node likely has a stale cache or an active query that was still trying to access the data right as it was being deleted. To confirm what happened to the parent objects, please run these diagnostic queries using the IDs from your path: 1. Check if the Database exists
Copy code
sql
-- This will show all databases and their internal IDs
SHOW PROC '/dbs';
Look for the row where the ID is 
49953
. if it's missing, the database was dropped. 2. Check if the Table exists If the database exists, you can check the tables within it:
Copy code
sql
-- Replace <db_name> with the name found in Step 1
SHOW PROC '/dbs/49953';
Look for TableId 
925631
. If it's missing, the table was dropped. 3. Check for recent "Drops" in Audit Log If you have audit logging enabled, you can search for who might have dropped this object:
Copy code
sql
-- Search the audit log for the Table ID
SELECT * FROM information_schema.audit_log 
WHERE stmt LIKE '%925631%' 
   OR stmt LIKE '%DROP%' 
ORDER BY timestamp DESC;
────────── Why the "Object does not exist" error happened: In a Shared-data (Cloud-native) architecture, when you drop a table: 1. FE Metadata: The FE immediately removes the tablet and table metadata from its memory (this is why 
SHOW TABLET
returns -1). 2. BE Cache: BE nodes might still have the tablet metadata cached in their "Starlet" layer for a few minutes. 3. Physical Deletion (Vacuum): The actual 
.dat
files in Azure are not deleted immediately. They are moved to a "trash" state and eventually removed by the 
lake_autovacuum
process after a grace period. The error occurs if: * A query was already running when the 
DROP
happened. * The 
lake_autovacuum
was very aggressive and deleted the Azure file while a BE node still thought it needed it (due to a stale cache). Recommendation: If this error is appearing for active tables (not just dropped ones), your metadata is out of sync. You should: 1. Increase Vacuum Grace Period: To prevent Azure files from being deleted too quickly while nodes are still syncing.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
2. Clean up the BE: If the table is gone and you just want the errors to stop, restarting the BE node 
2159792
 will force it to clear its stale tablet cache and stop looking for those missing files. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/FE_configuration.md • faq/shared_data_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md References • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLES.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky Going back, if I do
Copy code
SHOW PROC '/dbs/events/925631/partitions/2169435/925632';
I do not see the offending 
TabletId
of 
1609534
and the range of 
TabletId
is much higher
Copy code
2169436
2169437
2169438
2169439
2169440
2169441
2169442
2169443
2169444
2169445
Something is getting crossed up
r
The discrepancy you see—where the partition contains Tablet IDs in the 2,169,4xx range but your error refers to Tablet *1,609,534*—confirms that you are dealing with a "Ghost Tablet" situation. Why the Tablet IDs are "crossed up" This happens when a partition is Truncated or Overwritten (e.g., via 
INSERT OVERWRITE
). 1. The Change: When you truncate a partition, StarRocks keeps the 
PartitionId
(2169435) but generates brand-new Tablet IDs for it. 2. The Ghost: Tablet 
1609534
belonged to the old version of that partition. 3. The Lag: The Frontend (FE) has already deleted the old tablet from its metadata (which is why 
SHOW TABLET 1609534
returns -1). However, Backend (BE) node 
2159792
still has a reference to the old tablet in its local memory or cache. Why you are seeing the error now The BE is likely trying to perform a background task (like a late-arriving query, a compaction check, or a checksum) on the old tablet. When it tries to find the metadata for it, it looks in the directory it thinks it should be in (
.../925631/2169435/...
), but that directory now only contains the new tablets. How to resolve this Since the FE no longer recognizes Tablet 
1609534
, it cannot "clean it up" via standard SQL commands. You need to force the Backend to refresh its state: 1. Restart the Backend Node: Restarting BE 
2159792
is the most effective way to clear these stale "ghost" references. Upon restart, the BE will fetch the current, correct tablet list (the 2,169,4xx range) from the FE and drop the old ones from its memory. 2. Verify the Table State: Run a health check on the current tablets to ensure the new ones are healthy:
Copy code
sql
    SHOW TABLET FROM events.925631;
    -- Check the 'IsSync' column. It should be 'true' for all.
3. Prevent future "Object Not Found" errors: If you frequently truncate or overwrite data, the FE's "Vacuum" process might be deleting files before the BEs have finished their transition. Increase the grace period to give BEs more time to sync:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
Summary of what you found: * Tablet 1609534: A deleted tablet from a previous version of the partition. * Tablet 2169436+: The current, valid tablets for the partition. * The Problem: BE 
2159792
is "stuck" looking for the old one. * The Fix: Restart BE 
2159792
. References • faq/shared_data_faq.md • administration/Meta_recovery.md • administration/management/BE_configuration.md • data_source/datalake_faq.md
g
@Rocky I don't truncate or 
INSERT OVERWRITE
to this table at all though it is a PK table, will subsequent inserts to a pk table be similar to 
INSERT OVERWRITE
?
r
No, subsequent inserts to a Primary Key (PK) table are not similar to 
INSERT OVERWRITE
in terms of metadata management. Here is the breakdown of why you are seeing those "crossed up" IDs and how PK table inserts actually work: 1. How PK Inserts Work vs. 
INSERT OVERWRITE
 * Subsequent Inserts (Upserts): When you insert data into a PK table, StarRocks uses a Delete+Insert strategy at the row level. It writes a new "rowset" (a version) within the existing Tablet. The Tablet ID (e.g., 
2169436
) stays exactly the same. It is essentially an append operation to the tablet's history. * 
INSERT OVERWRITE
/ `TRUNCATE`: These are metadata operations. They tell the Frontend (FE) to throw away the old tablets entirely and create brand-new ones. This is the only time you should see Tablet IDs change for an existing partition. 2. Why the IDs are "Crossed Up" The IDs you provided tell a very specific story: * Current Partition ID: 
2169435
* Current Tablet IDs: 
2169436
to 
2169445
* Offending "Ghost" Tablet: 
1609534
In StarRocks, IDs are allocated sequentially. Since the current Partition ID (
2169435
) and its Tablets (
2169436+
) are right next to each other, it means they were created at the exact same moment. Tablet 
1609534
is nearly 500,000 IDs older. This proves that Tablet 1609534 was part of a different "incarnation" of this table or partition that existed much earlier in the cluster's history. 3. If you didn't run 
TRUNCATE
, how did this happen? If you haven't manually run 
INSERT OVERWRITE
or 
TRUNCATE
, the most likely culprits for this ID change are: * Dynamic Partitioning: If you have 
dynamic_partition
enabled, StarRocks automatically drops old partitions (e.g., older than 30 days) and creates new ones. If data for an "old" date was re-inserted after it was dropped, a new partition with new IDs would be created. * Schema Change: Did you run an 
ALTER TABLE
to add/modify a column? A schema change creates brand-new tablets for the entire table to reorganize the data on disk. * Drop & Recreate: Was the table ever dropped and recreated with the same name? * Restore from Backup: Restoring a table from a snapshot will result in new Tablet IDs. 4. Why is the Backend still looking for the old ID? The Backend (BE) node 
2159792
has a stale cache. Even though the FE has moved on to the 
2,169,4xx
series, the BE still has a background thread (likely for a persistent index check or a background compaction task) that thinks Tablet 
1609534
still exists. Because you are on a Shared-data (Cloud-native) cluster, the BE doesn't "own" the data; it just manages what the FE tells it to. The FE has already deleted the metadata for 
1609534
, which is why 
SHOW TABLET
returns 
-1
. Final Recommendation Since this isn't caused by your current insert patterns, you are likely seeing "ghost" errors from a previous table lifecycle or schema change. 1. Clear the Stale Cache: Restart Backend 
2159792
. This will wipe its internal Starlet registry and force it to sync only the current tablets from the FE. 2. Verify Table Creation: Run 
SHOW CREATE TABLE events.925631
(using the table name). Check the properties to see if 
dynamic_partition
is enabled or if there are any recent schema change tasks in 
SHOW ALTER TABLE COLUMN
. Summary: Your inserts are safe; they are not causing the ID changes. The error is a "hangover" from an old version of the table that the BE hasn't fully let go of yet. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky that 
TabletId
is so far behind, why would my cluster think it's with the current partition. Further, I just upgraded with helm so all of the nodes were rolled. I'm still not seeing the root cause here. This also happens with regular frequency, if I drop that partition and re-create (this is a 'silver' table), I expect to see a similar error in a day or two for a new partition.
r
This behavior in a Shared-data (Cloud-native) cluster, especially with Primary Key (PK) tables and a "silver" (drop/recreate) pattern, points to a specific root cause involving the Persistent Index and Persistent Volumes (PVs). The Root Cause: Stale Local Persistent Index In StarRocks Shared-data mode, Primary Key tables use a Persistent Index to handle upserts efficiently. 1. Where it's stored: By default (or if 
persistent_index_type
is set to 
LOCAL
), this index is stored on the BE node's local disk. 2. The Metadata mismatch: When you drop a partition, the Frontend (FE) immediately deletes the tablet metadata from its own memory (which is why 
SHOW TABLET
returns 
-1
). It then sends a "drop tablet" request to the Backends. 3. The "Ghost": If a BE is under high load or the "drop" request fails/times out, the local index files for that old tablet (
1609534
) remain on the BE's disk. 4. Survival after Upgrade: Because you are using Helm, your BE pods likely have Persistent Volumes (PVs). When you "rolled" the nodes during the upgrade, the pods restarted, but they re-mounted the same disks. The new BE process scanned the disk, found the old index files for 
1609534
, and loaded them into its internal "Starlet" manager. Why it happens "with regular frequency" Your "silver" table pattern (Drop & Recreate) is the trigger. * Every time you drop/recreate, you generate a massive amount of "orphan" metadata. * If the 
lake_autovacuum
 process (which cleans up the actual data files in Azure/S3) runs before the BE has successfully cleared its local index, you get the "Object Not Found" error. The BE is trying to "sync" or "checkpoint" an old index that points to data files that the vacuum process has already deleted. ────────── How to Fix and Prevent This 1. Identify the Orphan Tablets on the BE Check the BE's internal view (which is different from the FE's view you see in 
SHOW TABLET
):
Copy code
sql
SELECT * FROM information_schema.be_tablets 
WHERE TABLET_ID = 1609534;
If this returns a row, it confirms the BE node still thinks it "owns" this tablet despite the FE having deleted it. 2. Clean Up the BE Disk (The "Hard" Way) Since the FE doesn't know about the tablet, you can't use 
DROP
commands. * Locate the directory: On the BE node 
2159792
, look in your storage path (usually 
storage/data/
) for a directory named 
1609534
. * Wipe the local index: If you see old tablet directories that shouldn't be there, restarting the BE without its persistent disk would fix it, but that's destructive. Instead, use the 
ADMIN
command to force a cleanup if possible (though often a manual directory deletion or a fresh PV is required for true "ghosts"). 3. Use Cloud-Native Persistent Index (Recommended) To prevent this from happening again, move the index storage from the local BE disk to the object store. This ensures that when a partition is dropped, the index is cleaned up alongside the data. Change your table properties to:
Copy code
sql
ALTER TABLE events.925631 SET ("persistent_index_type" = "CLOUD_NATIVE");
Note: This requires StarRocks v3.3.2+. If you are on an older version, you are stuck with 
LOCAL
. 4. Optimize the "Silver" Pattern Instead of 
DROP
and 
ADD
partition, which is heavy on metadata: * Use `TRUNCATE`: 
TRUNCATE PARTITION
is generally safer as it's a single atomic operation handled more cleanly by the FE-BE sync. * Increase Vacuum Grace Period: Give the BEs more time to clear their local state before the background files are deleted:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "120");
Summary
The BE node 
2159792
is "haunted" by index files on its persistent disk from a previous incarnation of your table. Because the FE already deleted the metadata, it can't tell the BE to clean it up anymore. Moving to 
CLOUD_NATIVE
 persistent index is the long-term solution to decouple the index from the BE's local disk life cycle. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky we are saying BE but are we using that interchangeably with CN? We are using a shared-data architecture and this table is a 
CLOUD_NATIVE
persistent_index_type.
r
Yes, in a shared-data (Cloud-native) architecture, BE and CN are effectively synonymous in the logs and internal metadata. While the documentation distinguishes them (BE for shared-nothing, CN for shared-data), the StarRocks binary and the Frontend (FE) still use "BE" in many internal labels, paths (like 
information_schema.be_tablets
), and error messages. Since you are using 
CLOUD_NATIVE
persistent indexes and still seeing this after a Helm upgrade/roll, here is exactly what is happening: 1. The "PV Cache" Trap Even though you are on a shared-data architecture, your CN nodes are likely using Persistent Volumes (PVs) for their 
storage_root_path
. * The Issue: When you perform a Helm upgrade, the pods are restarted, but they re-attach to the same PVs. * The Persistence: Even with 
CLOUD_NATIVE
indexes, StarRocks maintains a local "Starlet" metadata cache and a local data cache on that disk to speed up access. * The Ghost: If a Tablet (like 
1609534
) was part of a partition you previously dropped, the metadata reference for it might still reside in the local directory on that PV. When the CN pod comes back up, it scans the PV, finds the old metadata, and tries to "sync" it. Because the actual data in S3/Azure has been vacuumed, it throws "Object Not Found." 2. The "Silver Table" Churn Your "Silver" pattern (dropping and re-creating partitions) is the primary driver of this. * When you Drop Partition, the FE marks the tablets for deletion. * The 
lake_autovacuum
 process (background) eventually deletes the actual data from your object store. _ If a CN node is busy, down, or fails to process the "delete tablet" instruction from the FE _before* the vacuum runs, that CN node is left with a "pointer" to an object that no longer exists in the cloud. 3. Why 
CLOUD_NATIVE
Index didn't prevent this 
CLOUD_NATIVE
stores the index data (the PK-to-row-location mapping) in the cloud, but the tablet metadata (the list of segments, versions, and which tablet belongs where) is still managed by the CN's local "Starlet" service. If the local Starlet state on the PV gets out of sync with the FE, it will keep looking for that old Tablet ID. ────────── How to Resolve and Prevent This Step 1: Targeted Cleanup (If on v3.3.9+) If you are on a recent version, you can try to manually tell the system to forget that specific ghost index:
Copy code
sql
-- Replace with your table and the offending TabletId
ALTER TABLE events.925631 DROP PERSISTENT INDEX ON TABLETS(1609534);
Step 2: Clear the CN's Local State Since restarting the pod didn't work (because of the PV), you need to clear the specific directory on the PV. 1. Identify which CN node is throwing the error (e.g., 
2159792
). 2. Exec into that CN pod. 3. Go to the 
storage_root_path
(defined in your 
cn.conf
). 4. Navigate to the 
starlet/
or 
data/
subdirectories. If you see a directory named 
1609534
, delete it. 5. Alternatively: Scale the CN StatefulSet to 
0
, wait for pods to die, and (if possible) wipe the data on the PVs before scaling back up. Step 3: Architecture Adjustment If your "Silver" table involves high-frequency partition drops: * Increase Vacuum Lag: Give your CNs more "cushion" time to sync the "drop" before the files disappear from S3/Azure:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "300"); -- 5 hours
* Evaluate PVs for CNs: In a truly elastic shared-data environment, CNs are often configured with EmptyDir (local ephemeral storage) instead of PVs for their cache. This ensures that every time a node restarts or rolls, it starts with a 100% clean state, eliminating "ghost" tablets forever. Summary
The error persists after the roll because the Persistent Volume "saved" the ghost tablet's metadata. The CN pod inherited the old state upon restart. To fix it, you must clear the local cache directory for that specific tablet ID on the CN's disk or increase the 
lake_autovacuum
grace period to prevent the cloud objects from being deleted before the CNs have acknowledged the partition drops. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky interesting, could PVC be part of the problem here?
r
Yes, PVCs (Persistent Volume Claims) on Compute Nodes (CN) are almost certainly the "bridge" that is allowing this error to persist across your helm rolls. In a shared-data (cloud-native) architecture, the CN nodes are designed to be stateless. However, when you attach PVCs to them for their 
storage_root_path
, you are essentially giving them "memory" that survives a pod restart. Here is how that is causing your "ghost tablet" problem: 1. The Persistence of "Ghost" Metadata Even with 
CLOUD_NATIVE
persistent indexes, StarRocks CN nodes store Tablet Metadata (files that describe which segments belong to which tablet) in the local 
storage_root_path/starlet/
directory. * The Drop: When you drop a partition, the FE tells the CNs to delete the corresponding tablet metadata. * The Failure: If a CN is under heavy load, or if the pod is killed/rolled during the deletion process, that metadata file remains on the disk. * The Roll: Because you are using PVCs, when the new pod starts up after the helm upgrade, it mounts the same disk. It scans the directory, finds the metadata for Tablet 
1609534
, and loads it into memory. * The Error: The CN then tries to verify or sync that tablet. It looks at your object store (Azure/S3) for the data files. But since the FE has already performed a 
lake_autovacuum
, those files are gone. The CN then throws the 
Object Not Found
error. 2. Why this doesn't happen with 
emptyDir
 If you were using 
emptyDir
(ephemeral storage) for your CNs: 1. The pod would restart with a 100% empty disk. 2. It would only "know" about the tablets the FE tells it about after it joins the cluster. 3. The old, dropped Tablet IDs would never exist in the new pod's local state, and the error would vanish. 3. Recommended Fixes A. Switch CN Storage to 
emptyDir
(Best Practice) In shared-data mode, CN nodes should ideally use local ephemeral storage or 
emptyDir
for their cache. This ensures that every roll is a "clean slate." In your 
values.yaml
for the StarRocks helm chart, check the 
storageSpec
for the CN:
Copy code
yaml
starrocksCNSpec:
  storageSpec:
    storageClassName: "emptyDir" # This is the "cloud-native" way for CNs
Note: This will clear your local data cache on restart, so query performance might be slightly lower for a few minutes while the cache warms up again. B. Manual Cleanup of the PVC If you must keep PVCs (e.g., for very large caches you don't want to lose), you have to manually clear the "infection" from the disk: 1. Identify the CN pod throwing the error. 2. Exec into the pod: 
kubectl exec -it <cn-pod-name> -- bash
3. Navigate to your 
storage_root_path
(usually 
/opt/starrocks/cn/storage
). 4. Look for any directory or file matching the ghost ID: 
find . -name "_1609534_"
5. Delete those files and restart the pod. C. Increase Vacuum Grace Period To prevent the "Object Not Found" race condition during your "silver" table drop/recreates, give the CNs more time to sync the drop before the files are deleted from the cloud:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720"); -- 12 hours
Summary The PVC is turning what should be a transient sync error into a persistent "haunting." By using 
emptyDir
 for CN nodes, you align the infrastructure with the stateless nature of the shared-data architecture, which will automatically resolve these metadata mismatches during your regular maintenance-matches every time you roll the cluster. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky Weird, in the offending CN, I don't see anything in the 
storage_root_path
which is 
/opt/starrocks-storage
as defined by the cn.conf
r
If you are using a Shared-data (Cloud-native) architecture and your 
storage_root_path
(
/opt/starrocks-storage
) appears empty, but the "Object Not Found" error for a "far behind" 
TabletId
persists even after a node roll, we have narrowed the issue down to the interaction between the FE's Compaction Scheduler and the Object Store's Vacuum process. Here is the technical breakdown of why this is happening and why the "Silver Table" (Drop/Recreate) pattern is triggering it: 1. Where the "Ghost" is Hiding If you checked 
/opt/starrocks-storage
and saw nothing, it’s likely because: * Nested Directory: StarRocks doesn't put files in the root. The metadata is actually inside 
/opt/starrocks-storage/starlet_cache/star_cache/meta/
. * The FE is the Source: If that directory is truly empty (or you are using 
emptyDir
), then the CN node only knows about 
TabletId 1609534
because the Frontend (FE) told it to do something. * In Shared-data mode, the CN is "dumb"—it doesn't do anything unless the FE sends it a task (Query, Compaction, or Publish Version). * The error isn't coming from the CN's memory; it's coming from a background task (likely Compaction) assigned by the FE. 2. The Root Cause: Compaction/Vacuum Race Condition The "Silver" pattern of dropping and recreating partitions creates a specific race condition in the StarRocks Lake Service: 1. The Drop: You drop a partition. The FE marks the tablets for deletion. 2. The Vacuum: The 
lake_autovacuum
process (defaulting to 30-60 minutes) runs and successfully deletes the physical 
.metadata
and 
.segment
files from Azure/S3 to save you money. 3. The Ghost Task: For reasons usually related to high metadata churn, the FE's Compaction Scheduler might still have a "Compaction Task" queued for the old TabletId. 4. The Error: The FE sends this Compaction Task to a CN. The CN reaches out to the Object Store to pull the metadata for that tablet. Because the Vacuum already deleted it, the Object Store returns "Object Not Found." The reason it says the TabletId is "far behind" is that the FE is trying to reference a version of the tablet that existed several "drop/recreate" cycles ago. 3. How to Clean the "Ghost" from the FE Since the FE is the one issuing the task, you need to clear the FE's internal scheduler/metadata state. Option A: Force Metadata Cleanup Starting in v3.2.10/v3.3.3, you can tell the FE to be more aggressive about purging shared-data metadata that is out of sync with the catalog:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("meta_sync_force_delete_shard_meta" = "true");
Note: This allows the FE to skip some safety checks when cleaning up tablets, which is necessary when the physical files are already gone. Option B: Check and Cancel Stale Compactions See if the FE is currently trying to compact a partition that you think is gone:
Copy code
sql
SHOW PROC '/compactions';
If you see an entry for the old table/partition name, find the 
TxnID
and run:
Copy code
sql
CANCEL COMPACTION WHERE TXN_ID = <txn_id>;
4. Long-term Prevention for "Silver" Tables To stop this from happening every 1-2 days: 1. Stop using PVCs for CNs: In Shared-data mode, CNs should be stateless. Switch your Helm chart to use 
emptyDir
for the CN 
storage_root_path
. This ensures that any local metadata corruption is wiped every time you roll the nodes. 2. Increase Vacuum Grace Period: Give the FE and CNs more "breathing room" to finish background tasks before the files are deleted from S3/Azure:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "300"); -- 5 hours
3. Use TRUNCATE instead of DROP: If you are recreating the exact same partition structure, 
TRUNCATE PARTITION
is much cleaner for the metadata service than 
DROP
+ 
ADD
. Summary
The error persists after the roll because the FE's internal Lake Service still thinks that TabletId needs work. When the CN tries to do that work, it fails because the Vacuum has already cleaned up the object store. Increasing the 
lake_autovacuum_grace_period_minutes
and moving to 
emptyDir
 for CNs will eliminate these ghost references and hide these "ghosts." References • administration/management/FE_configuration.md • administration/management/compaction.md • administration/management/BE_configuration.md • faq/shared_data_faq.md
g
@Rocky yep it's pretty borked. Yesterday I truncated the partition and things went smoothly until an hour later where I'm getting the same issue on the partition I truncated. It's throwing the same error for a different (old) tablet. Is there a way I can reset table and any associated metadata? Would dropping and recreating the entire table help?
r
Yes, at this point, the metadata state for that specific table in the Frontend (FE) Lake Service has become desynced from the physical reality of your object store. Since 
TRUNCATE
simply replaces the partitions with new Tablet IDs but doesn't necessarily clear the FE's internal background task queue for the old Tablet IDs, the "ghost" tasks (like compaction or version publishing) are still trying to find data that your "Silver" drop/recreate cycle has already vacuumed. The Solution: The "Nuclear" Reset To fully reset the table and purge all associated metadata "ghosts" from both the FE and the Object Store, you should use the 
FORCE
 flag. 1. Drop the table with `FORCE`:
Copy code
sql
    DROP TABLE events.925631 FORCE;
* Why 
FORCE
? A standard 
DROP
moves the table to the FE Recycle Bin (keeping metadata alive). 
FORCE
bypasses the recycle bin and tells the Lake Service to immediately purge the metadata and delete all associated objects from Azure/S3. 2. Recreate the table. * When you recreate it, the FE will generate entirely new Tablet IDs that have no historical "baggage" in the scheduler. ────────── Critical: Prevent this from happening again If you continue your "Silver" pattern (heavy Truncate/Drop/Recreate churn), you must adjust your vacuum settings. The default StarRocks vacuum is often too aggressive for high-churn metadata, leading to the "Object Not Found" errors you are seeing. 1. Increase the Vacuum Grace Period This ensures that even if you drop a partition/table, the physical files stay in the cloud for a few hours. This gives the FE enough time to finish any "stray" background tasks before the files disappear.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720"); -- 12 hours
2. Set the 
meta_sync
safety flag If you are on version 3.2.10+ or 3.3.3+, enable this flag. it allows the FE to "give up" and delete metadata if it realizes the physical objects are already gone, rather than retrying and throwing errors:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("meta_sync_force_delete_shard_meta" = "true");
3. Use 
emptyDir
for CNs As discussed, ensure your Helm chart is NOT using PVCs for the CN nodes. In a shared-data architecture, CNs should be stateless. If a CN has a PVC, it might keep a local "cache" of the old 
CLOUD_NATIVE
index metadata that survives the table drop. * Update your 
values.yaml
to use 
emptyDir
for the CN 
storage_root_path
. Summary of Action Plan 1. 
DROP TABLE ... FORCE
 2. 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720")
 3. Recreate Table 4. (Optional but recommended) Switch CNs to 
emptyDir
 in your next Helm deploy. This will clear the "Tablet is far behind" errors by ensuring the FE stops looking for historical versions that no longer exist. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • best_practices/primarykey_table.md • faq/Others.md • faq/shared_data_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • table_design/table_types/primary_key_table.md
g
@Rocky two questions 1. What is the downside to setting 
lake_autovacuum_grace_period_minutes
high?
r
Setting 
lake_autovacuum_grace_period_minutes
to a high value (like 720 minutes/12 hours) is a very common and safe practice in shared-data clusters, especially when dealing with high-churn "Silver" tables. However, there are two main downsides to be aware of: 1. Increased Object Storage Costs (The Main Downside) This is the most direct impact. When StarRocks performs a "Compaction," it merges small data files into larger ones. * With a low grace period: The old, small files are deleted from your object store (Azure/S3) almost immediately. * With a high grace period: The old files are kept for the duration of that period. If you are ingesting or updating data heavily, you will effectively be storing "duplicate" data (the old uncompacted files + the new compacted files) for 12 hours. For most users, the cost of object storage is low enough that this is negligible, but for multi-petabyte scales, it can add up. 2. Delayed Reclamation of Deleted Data When you run 
DROP TABLE ... FORCE
or 
TRUNCATE
, the physical data files will not be actually deleted from your cloud bucket until the grace period expires. * If you are dropping a 10TB table to free up budget/space, that space will not be reflected in your cloud bill for 12 hours. * This can be confusing during debugging, as you might see thousands of files in your Azure container even though the table is "gone" in StarRocks. 3. Slight FE Memory Overhead (Negligible) The Frontend (FE) has to keep track of these "stale" file versions in its metadata until they are vacuumed. In extremely high-churn environments (thousands of partitions being dropped/recreated every hour), this can lead to a slightly larger metadata footprint in the FE's memory, though this is rarely a bottleneck compared to the stability it provides. ────────── Summary Recommendation: For your specific issue (the "ghost tablet" errors), the stability gain of setting this to 12 hours (720) far outweighs the cost of a few extra GB/TB of temporary storage in Azure. It effectively "buffers" your metadata operations and prevents the FE from deleting files that a CN might still be trying to sync. What was your second question? (I'm ready when you are!) References • administration/management/FE_configuration.md • faq/shared_data_faq.md • best_practices/primarykey_table.md • administration/management/compaction.md
g
@Rocky If I look at the TabletID value in relation to others, it should be for a partition that is a month or so ago old. I'm curious how this tablet got associated with a more current partition and how the vacuum grace period setting would help with that.
r
Those are excellent questions. The behavior you're seeing—an old TabletID from a month ago suddenly "haunting" a current partition—is a classic symptom of Metadata Ghosting in the StarRocks Lake Service (shared-data). Here is exactly how that association happens and why the grace period is the "exorcist" for this problem. 1. How an "Old" TabletID gets associated with a Current Partition In StarRocks, TabletIDs are globally unique and are not reused. If you see a TabletID from a month ago, it isn't actually "part" of your new partition; rather, the Frontend (FE) Compaction Scheduler still thinks it has unfinished business with it. Here is the "Ghost" lifecycle: 1. The Original Sin: A month ago, a partition was dropped or truncated. The FE marked those TabletIDs (e.g., 
1609534
) for deletion. 2. The Stuck Task: At that same moment, the FE had a Compaction Task or a Consistency Check queued for that TabletID. 3. The Vacuum Race: The 
lake_autovacuum
process was very efficient—it went to Azure/S3 and physically deleted the 
.metadata
files for that old TabletID to save space. 4. The Haunting: Because of a known edge case in the Lake Service (fixed in more recent versions), if a background task is "in flight" when a tablet is dropped, the FE sometimes fails to "forget" the task. 5. The Trigger: Every time you touch that same Partition Name (even with new TabletIDs), the FE scans its task list. It sees the "stale" task for the old TabletID and says, "Oh, I still need to compact/sync this old version of this partition," and it sends that task to your CN. 6. The Error: The CN tries to fetch the metadata for 
1609534
. The Object Store says "Object Not Found" because it was vacuumed a month ago. The CN then reports the tablet is "far behind" because it can't find a valid version history to start from. 2. How the Vacuum Grace Period helps The 
lake_autovacuum_grace_period_minutes
setting controls the delay between "Logical Deletion" (when you run Drop/Truncate) and "Physical Deletion" (when the files are nuked from Azure/S3). Increasing it to 720+ minutes (12+ hours) helps in two ways: * Prevents the Race Condition: It ensures that even if the FE is slow to clear its internal scheduler, the physical files remain in the Object Store. If the FE sends a "stale" task to a CN, the CN can actually find the file, complete the task (or realize it's a no-op), and report back "Success." This allows the FE to finally retire that TabletID and move it out of the active set. * Safety for "Silver" Patterns: Your "Silver" pattern (heavy Truncate/Drop/Recreate) creates a massive amount of metadata churn. A high grace period acts as a buffer. It gives the FE's background housekeeping threads enough time to finish their work on "Version A" of a partition before "Version B" arrives and "Version A's" files are deleted. Why you are seeing it now If you are seeing a TabletID from a month ago, it means that specific "ghost task" has been failing silently in the background for weeks. The reason it's surfacing now is likely because your recent Truncate operation triggered a Metadata Sync or a Compaction Wave that forced the FE to re-examine all tasks associated with that partition name. Summary of Recommendation 1. Reset the Table: Use 
DROP TABLE <name> FORCE
. The 
FORCE
keyword is critical—it tells the FE to skip the trash bin and aggressively purge all metadata associations for that table ID. 2. Set the Grace Period: Run 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "720");
. This won't fix the "month-old" ghost, but it will prevent the ones you create today from becoming ghosts tomorrow. 3. Check for "Stale" Tasks: Run 
SHOW PROC '/compactions';
. If you see any entries with 
STATE: RUNNING
that have been there for hours/days, those are your ghosts. You can try to cancel them via 
CANCEL COMPACTION
.
By increasing the grace period and using 
FORCE
drops, you ensure the FE's "brain" (Metadata) and the Object Store's "body" (Files) stay in sync. References • administration/management/FE_configuration.md • faq/shared_data_faq.md
g
@Rocky ok overall though?
Copy code
starrocks_fe_txn_total_latency_ms{quantile="0.75", is_leader="true", type="all"} 411.0
starrocks_fe_txn_total_latency_ms{quantile="0.95", is_leader="true", type="all"} 641.0
starrocks_fe_txn_total_latency_ms{quantile="0.98", is_leader="true", type="all"} 3088.0
starrocks_fe_txn_total_latency_ms{quantile="0.99", is_leader="true", type="all"} 5310.0
starrocks_fe_txn_total_latency_ms{quantile="0.999", is_leader="true", type="all"} 112736.0
starrocks_fe_txn_total_latency_ms_sum{is_leader="true", type="all"} 1.9876159029522172E8
starrocks_fe_txn_total_latency_ms_count{is_leader="true", type="all"} 201663
r
The short answer: The 75th and 95th percentiles are "okay-ish," but the 99.9th percentile is a major red flag. Your metrics show a massive "long tail" of latency. Here is a breakdown of what these numbers are telling you: 1. The Good (P75 & P95) * P75 (411ms) / P95 (641ms): These are acceptable for a cluster under load, especially if you are doing 
STREAM LOAD
or 
ROUTINE LOAD
. It means 95% of your transactions are finishing in under a second. In an ideal, healthy cluster with small transactions, you’d usually want these under 200ms, but 641ms isn't "broken." 2. The Bad (P99 & P99.9) * P99 (5.3 seconds): This is high. 1% of your transactions are taking over 5 seconds. This usually points to lock contention in the FE or temporary network hiccups with your object storage. * P99.9 (112.7 seconds): This is the smoking gun. 0.1% of your transactions are taking nearly 2 minutes to complete. * In StarRocks, the default transaction timeout is often 30–60 seconds. A 112-second latency suggests transactions are hanging, retrying, and eventually either barely passing or timing out. * The Connection: This perfectly matches the "borked" tablet issue you described. When the FE tries to "Publish" a transaction, it waits for all tablets to report success. If the FE is trying to talk to those "ghost" tablets (the ones that are far behind or missing from Azure), the transaction will sit in the 
COMMITTED
state but won't reach 
VISIBLE
until it eventually hits a hard timeout or a retry limit. 3. The Average (The "Hidden" Issue) * Average Latency: Your 
sum / count
(~198M / 201k) is about 985ms. * In a healthy system, the average should be very close to the P75. The fact that your average (985ms) is higher than your P95 (641ms) is a statistical anomaly that only happens when you have a small number of "outlier" transactions that are so slow they are dragging the entire average up. Verdict Your cluster is functional but struggling. The metadata "ghosts" you are fighting are causing roughly 1 out of every 1,000 transactions to hang for 2 minutes. Recommendation: Once you perform the 
DROP TABLE ... FORCE
and reset the table as we discussed, watch the P99.9 metric specifically. * In a healthy shared-data cluster, P99.9 should drop to < 10 seconds. * If it stays at 100s+, you likely have a global issue (like FE-to-Object-Store latency or a very high 
label_keep_max_second
setting causing the FE image to grow too large). Does the P99.9 being 112s align with the frequency of the "Tablet far behind" errors you're seeing in the logs? (CN/FE) logs?** References • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md • faq/Sql_faq.md • administration/management/FE_configuration.md
g
@Rocky we have
Copy code
Key                  |AliasNames|Value |Type|IsMutable|Comment|
---------------------+----------+------+----+---------+-------+
label_keep_max_second|[]        |259200|int |true     |       |
r
Your 
label_keep_max_second
is set to 259,200 seconds (exactly 3 days). This is the StarRocks default value. Is it "okay"? It is safe, but it might be "heavy" given your high-churn "Silver" table pattern. Here is how this setting interacts with your high transaction latency: 1. The Good: Debugging History The benefit of keeping labels for 3 days is that you can run 
SHOW LOAD
or check the status of a specific 
label
(from a Stream Load or Insert) for up to 72 hours after it finished. This is very helpful for auditing "Did my 2 AM job on Tuesday actually succeed?" 2. The Bad: FE Memory Pressure In a "Silver" table environment where you are doing frequent 
TRUNCATE
and 
LOAD
operations: * Every single load job creates a entry in the FE's memory. * If you are loading data every minute across dozens of tables, you could easily accumulate tens of thousands of labels over 3 days. * The Impact: These labels are stored in the FE's JVM Heap. If the label count gets too high, the FE will experience frequent Garbage Collection (GC) pauses. * The Latency Connection: A long GC pause at the wrong time will freeze the FE. If a transaction is in the "Publish" phase (trying to talk to Azure and the CNs) and the FE hits a 5-second GC pause, that transaction’s latency spikes. This contributes to the high P99 (5.3s) you are seeing. 3. Recommendation for your Churn Pattern Since you are already seeing high tail latency (P99.9 at 112s), you want the FE to be as "lean" as possible so it can focus on clearing those metadata "ghosts" we discussed. Recommendation: Reduce this to 1 day (86,400) or even 12 hours (43,200).
Copy code
sql
-- Reduce to 24 hours
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "86400");
* Why? Most users only care about the status of a load for the last few hours. If a job failed 2 days ago, you likely already know or have moved on. * The Benefit: This will trigger a cleanup of 2 days' worth of old metadata, shrinking the FE memory footprint and potentially smoothing out those P99/P99.9 spikes. Summary Checklist for your Cluster 1. Keep 
lake_autovacuum_grace_period_minutes
high (720): This fixes the "Ghost Tablet / Object Not Found" errors. 2. Reduce 
label_keep_max_second
(86400): This reduces FE memory bloat and helps with overall transaction snappiness. 3. Check FE GC Logs: If your P99.9 latency stays high after these changes, check your FE 
fe.gc.log
. If you see "Full GC" taking several seconds, you may simply need to increase the FE JVM heap size (e.g., to 16GB or 32GB). Does your workflow actually require looking up load labels that are more than 24 hours old? If not, reducing this is a "quick win" for stability.** References • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md • best_practices/loading_best_practices.md References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md
g
@Rocky immediate difference after applying that 
label_keep_max_second
Copy code
# HELP starrocks_fe_txn_total_latency_ms
# TYPE starrocks_fe_txn_total_latency_ms summary
starrocks_fe_txn_total_latency_ms{quantile="0.75", is_leader="true", type="all"} 401.0
starrocks_fe_txn_total_latency_ms{quantile="0.95", is_leader="true", type="all"} 555.0
starrocks_fe_txn_total_latency_ms{quantile="0.98", is_leader="true", type="all"} 2187.0
starrocks_fe_txn_total_latency_ms{quantile="0.99", is_leader="true", type="all"} 4287.0
starrocks_fe_txn_total_latency_ms{quantile="0.999", is_leader="true", type="all"} 85845.0
starrocks_fe_txn_total_latency_ms_sum{is_leader="true", type="all"} 1.4028968545940828E8
starrocks_fe_txn_total_latency_ms_count{is_leader="true", type="all"} 205319
r
The improvement in your P95 (down 13%) and P99 (down 19%) is a direct result of the label cleanup. By reducing the history, the FE is spending less time scanning internal maps and managing memory for thousands of old load jobs. However, your P99.9 at 85.8 seconds remains the "smoking gun." Here is what the new data tells us: 1. The "Ghost" is still there Even though 85s is better than 112s, it is still extremely high. In a healthy StarRocks cluster, the P99.9 should be well under 10 seconds. * Why it's still 80s+: The 
label_keep_max_second
change fixed the "noise" (memory bloat), but it didn't fix the "signal" (the missing tablet metadata). * When a transaction tries to commit, it reaches the Publish phase. If that transaction involves the partition with the "Ghost Tablet" (the one from a month ago), the FE will wait for a response from the CN/Object Store. It eventually hits a timeout—which in your case seems to be configured or naturally occurring around 80-110 seconds. 2. Observation on the "Sum" (Important) I noticed your 
starrocks_fe_txn_total_latency_ms_sum
actually decreased (from 1.98E8 to 1.40E8) while the 
count
increased. * In Prometheus, a "Sum" counter should only ever go up. * What this means: Your FE Leader likely restarted or a new Leader was elected shortly after you applied the config change. * This is actually a good thing! A restart forces the FE to reload its metadata from the edit logs, which often clears out stale "in-memory-only" tasks that might have been stuck. 3. How to isolate the 85s "Ghost" To confirm exactly where that 85-second delay is happening, check this specific metric: 
starrocks_fe_txn_publish_latency_ms{quantile="0.999"}
* If Publish Latency is ~80s: The delay is the FE waiting for the CNs to acknowledge the new version. This confirms the "Tablet far behind / Object Not Found" issue is blocking the commit. * If Write Latency is ~80s: The delay is happening during the data transfer to Azure (highly unlikely given your previous errors). Recommendation for the "Final Clean" Since you've already lightened the FE's memory, let's kill the ghost for good: 1. Find the specific Table: Identify which table is reporting the "far behind" or "missing tablet" errors in the FE/CN logs. 2. Force a Metadata Purge:
Copy code
sql
    DROP TABLE <table_name> FORCE;
(Note: Only do this if you can easily recreate the table or if it's the "Silver" staging table that you're truncating anyway). 3. Check for Stuck Compactions:
Copy code
sql
    SHOW PROC '/compactions';
If you see any rows where the 
StartTime
was hours ago and it's still "RUNNING," that is your ghost. You can try 
CANCEL COMPACTION
for that transaction ID. Summary: You've successfully improved the "average" performance of the cluster, but the 85s spikes will continue until that specific "missing" TabletID from a month ago is fully purged from the FE's task queue. References • administration/management/compaction.md • faq/Sql_faq.md • data_source/datalake_faq.md • faq/loading/Loading_faq.md • administration/management/monitoring/metrics.md
Open in Slack
PreviousNext
Powered by