http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> we have tablets which have no table...
# questions-and-troubleshooting
и
@Rocky we have tablets which have no tables associated with them, how could we clean these up? Are there any special commands to fix those stale metadata?
r
To clean up tablets that are not associated with any tables (often called "orphan" or "stale" tablets), StarRocks provides built-in mechanisms and specific configurations to manage metadata consistency. 1. Automatic Background Cleanup (Recommended) Starting from StarRocks v3.2.0, the FE (Front End) runs a background daemon called 
ConsistencyChecker
that automatically scans for tablets existing on BEs (Back Ends) that are missing from the FE's metadata (
TabletInvertedIndex
). * Key Configuration: 
consistency_tablet_meta_check_interval_ms
(FE parameter). * Default: 7,200,000 ms (2 hours). * Behavior: When the system detects a tablet on a BE that is not in the FE's metadata, it marks it for deletion. The actual deletion is typically delayed by half the interval to ensure safety. * How to fix faster: You can temporarily reduce this interval to trigger a scan sooner:
Copy code
sql
        ADMIN SET FRONTEND CONFIG ("consistency_tablet_meta_check_interval_ms" = "300000"); -- Set to 5 mins
* BE Deletion Threads: The BE uses 
drop_tablet_worker_count
to control how many threads handle the physical deletion. 2. Check the "Trash" Directories Sometimes tablets appear to be "there" but are actually just waiting in the recycle bin (trash). StarRocks does not delete data immediately to allow for recovery. * FE Recycle Bin: 
catalog_trash_expire_second
(Default: 24h). * BE Recycle Bin: 
trash_file_expire_time_sec
(Default: 24h). If you want to reclaim space immediately after a 
DROP TABLE
, use the 
FORCE
keyword:
Copy code
sql
DROP TABLE <table_name> FORCE;
3. How to Identify Orphan Tablets To verify if a specific tablet is an orphan: 1. Check FE metadata: Run 
SHOW TABLET <tablet_id>;
. If the tablet is an orphan, the result may be empty or return an error indicating the tablet does not exist in the FE. 2. Check BE reality: Query the 
information_schema.be_tablets
view to see what tablets the BE is actually reporting:
Copy code
sql
    SELECT * FROM information_schema.be_tablets WHERE TABLET_ID = <tablet_id>;
3. Check Cluster Statistics: Run 
SHOW PROC '/statistic';
to see the 
UnhealthyTabletNum
. High numbers here often indicate metadata inconsistencies. 4. Shared-Data (Cloud-Native) Cleanup If you are using a shared-data cluster (where data is on S3/OSS/HDFS), orphan data files or metadata files in object storage can be cleaned up using a specialized tool: * Tool: 
data_gc
tool (found in the StarRocks release package or via 
meta_tool.sh
). * Operation: 
lake_datafile_gc
. * Example:
Copy code
bash
    ./bin/meta_tool.sh --operation=lake_datafile_gc --root_path="<s3://your-bucket/path>" --do_delete=true
_Note: Always run with 
--do_delete=false
first to audit what will be removed._ Summary of Commands Action | Command / Config Trigger Scan | 
ADMIN SET FRONTEND CONFIG ("consistency_tablet_meta_check_interval_ms" = "...")
Verify Tablet | 
SHOW TABLET <tablet_id>
or 
SELECT * FROM information_schema.be_tablets
Force Cleanup | Use 
DROP ... FORCE
or reduce 
trash_file_expire_time_sec
Check Health | 
SHOW PROC '/statistic'
If tablets remain stuck even after the background scan, check the FE and BE logs for "failed to delete tablet" errors, which might indicate file system permission issues or stuck RPC calls to object storage. References • administration/http_interface.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md
и
@Rocky we had stale compaction and had to cancel it, now we have errors like this: 
E20260514 13:50:27.228618 139526345328192 lake_service.cpp:628] Fail to remove <staros://1548620>: Invalid argument: starlet err [RequestID=d84fd8a3b50c0980][StatusCode=403]List objects 962ff38e-e94f-4e4d-9e2c-8f6316c792f8/db437203/1512562/1548613/ error: The request signature we calculated does not match the signature you provided. Check your key and signing method..
how do I get rid of those stale metadata
r
The error message 
StatusCode=403 ... The request signature we calculated does not match the signature you provided
indicates that while your StarRocks cluster (specifically the "Starlet" component within the BE/CN) is attempting to perform garbage collection (GC) on stale files in object storage, the authentication request is being rejected by S3 (or your S3-compatible provider). This is common after a cancelled compaction because the system generates many temporary files that the Lake Service now needs to clean up. 1. Fix the Root Cause: Signature Mismatch (403) The system will continue to retry these deletions and log errors until the authentication is fixed. If StarRocks can't authenticate the 
DELETE
request, it cannot remove the stale metadata. * Check Clock Sync (Most Common): S3 signature calculation is extremely sensitive to time. Ensure all BE/CN nodes and FE nodes have their clocks synchronized via NTP. A drift of more than 5 minutes will cause this exact 
SignatureDoesNotMatch
error. * Verify S3 Permissions: Ensure the credentials used for the cluster have 
s3:DeleteObject
and 
s3:ListBucket
permissions. Sometimes a policy allows 
Put
and 
Get
but restricts 
Delete
. * Credential Consistency: If you recently updated your S3 keys or IAM roles, ensure they are updated in the storage volume configuration. You can check the current storage information:
Copy code
sql
    SHOW STORAGE VOLUMES;
If the credentials there are outdated, update them using 
ALTER STORAGE VOLUME
. 2. Manual Cleanup of Stale Metadata If the background GC is stuck or you want to force the cleanup of these orphaned files/metadata: Use the 
data_gc
tool StarRocks provides a utility in the 
bin
directory of the BE/CN package specifically for shared-data clusters. * Location: 
be/bin/meta_tool
or 
be/bin/starrocks_be --util lake_datafile_gc
(depending on your version). * Action: Use the 
lake_datafile_gc
operation. This tool scans the object storage and compares it against the FE metadata to find orphaned files.
Copy code
bash
    # Step 1: Scan for orphaned files (dry run)
    ./bin/meta_tool --operation=lake_datafile_gc --root_path="<s3://your-bucket/path>" --do_delete=false

    # Step 2: If the list looks correct, perform the deletion
    ./bin/meta_tool --operation=lake_datafile_gc --root_path="<s3://your-bucket/path>" --do_delete=true
Adjust Vacuum Grace Period StarRocks shared-data uses an "autovacuum" process. If the metadata is "stale" but within the grace period, it won't be deleted yet. You can reduce this period to trigger cleanup sooner: * FE Configuration: 
lake_autovacuum_grace_period_minutes
* Default: 1440 (24 hours). * Command: 
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
(Set to 1 hour for faster cleanup). 3. Cleaning Up "Cancelled" Compaction Tasks If the FE still thinks compaction tasks are running (blocking new ones), you may need to clear the task state: 1. Check for stuck tasks:
Copy code
sql
    SHOW PROC '/compactions';
2. If tasks are stuck in 
RUNNING
for a cancelled job, you can usually clear them by restarting the Leader FE, or by disabling/enabling compaction globally to reset the scheduler:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_lake_compaction" = "false");
    -- Wait 1 minute
    ADMIN SET FRONTEND CONFIG ("enable_lake_compaction" = "true");
Summary Checklist 1. Sync NTP on all nodes (Fixes the 403 signature error). 2. Check 
s3:DeleteObject
 permissions. 3. Lower 
lake_autovacuum_grace_period_minutes
 to 60. 4. Wait for the Lake Service to retry the deletions. Once the signature is valid, the 
Fail to remove
errors will disappear from the logs as the files are successfully deleted.
и
@Rocky give me a real documentation link about 
data_gc
tool
r
Detailed documentation for cleaning up orphaned data in shared-data clusters can be found in the StarRocks Knowledge Base: * Documentation Link: Shared-data Garbage Cleanup Key Information about the 
data_gc
tool: This tool is specifically designed for StarRocks shared-data clusters (where data resides on S3/OSS) to clean up "garbage" files that were not automatically removed due to failed compactions or ingestion tasks. * How it works: It compares physical data files in object storage against the Tablet Meta files. If a data file is not referenced by any metadata, it is flagged as garbage. * Safety First (Dry Run): It is strongly recommended to run with 
--do_delete=false
first to see what would be deleted. * Example Command:
Copy code
bash
    ./bin/meta_tool.sh --operation=lake_datafile_gc \
      --root_path="<s3://your-bucket/starrocks_cluster_id/>" \
      --expired_sec=86400 \
      --conf_file=./conf/be.conf \
      --do_delete=false
Authentication Fix Since you are seeing 403 SignatureDoesNotMatch errors, the 
data_gc
tool will also fail unless it can authenticate. 1. Check NTP: Ensure all nodes have synchronized clocks. 2. Verify 
be.conf
: The tool reads your object storage credentials from the BE config file. Ensure 
object_storage_access_key_id
and 
object_storage_secret_access_key
are correct and have 
s3:DeleteObject
permissions. Automatic Cleanup (Alternative) If you prefer the system to handle this automatically and have fixed the signature/permission issues, you can lower the retention window to trigger the built-in "AutoVacuum":
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
This tells StarRocks to clean up any historical versions older than 60 minutes. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/FE_configuration.md
и
r
I apologize for the broken link. That specific "knowledge" article is currently hosted within the internal StarRocks documentation repository and might not be fully indexed on the public site yet. However, your logs indicate a more fundamental issue: StarRocks cannot clean up the stale metadata because it is being rejected by your object storage (S3/OSS) with a 403 Signature Mismatch. Even if you run manual cleanup tools, they will likely fail with the same error until the authentication is fixed. 1. Fix the 403 "SignatureDoesNotMatch" Error This error occurs when the S3 service rejects the authentication header generated by StarRocks. * Check Clock Sync (Critical): S3 authentication fails if your BE/CN node clocks are off by more than 5 minutes. Run 
ntpdate
or 
chrony
on all nodes to ensure they are synchronized. * Validate Credentials: Ensure the Access Key and Secret Key used in your Storage Volume have 
s3:DeleteObject
and 
s3:ListBucket
permissions. * Bucket Region: Ensure the 
aws_s3_region
matches exactly where your bucket is located. 2. Manual Cleanup Tool (
lake_datafile_gc
) The tool you were looking for is actually built into the BE/CN binary. You can run it directly from your terminal. It scans object storage and deletes files not referenced in the FE metadata. The Command:
Copy code
bash
# Run from your BE or CN 'bin' directory
./meta_tool --operation=lake_datafile_gc \
  --root_path="<s3://your-bucket-name/your-cluster-id/>" \
  --conf_file=../conf/be.conf \
  --expired_sec=3600 \
  --do_delete=false
* 
--root_path
: The S3 path where your data is stored. * 
--do_delete=false
: Always run this first. It will output a list of "garbage" files it found without deleting them. * 
--do_delete=true
: Once you've verified the list, change this to 
true
to perform the actual cleanup. * 
--expired_sec
: Only files older than this (in seconds) will be cleaned. Setting it to 3600 (1 hour) is safe for stale compaction cleanup. 3. Cleaning Up "Stale" Metadata in FE If the "stale metadata" refers to tablets the FE still thinks exist but shouldn't, you can force the FE to ignore/cleanup those IDs using these configurations: * Bypass remote cleanup (Use with caution): If you want to drop the metadata from the FE without waiting for the (currently failing) S3 delete calls, set this:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("meta_sync_force_delete_shard_meta" = "true");
Note: You must manually clean S3 later using the tool in step 2, as the FE will stop trying to delete those files. * Reduce Vacuum Grace Period: To make the system's automatic garbage collector (AutoVacuum) more aggressive:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");
Summary of Actions 1. Sync your server clocks (NTP) to fix the 403 error. 2. Check 
SHOW STORAGE VOLUMES
to ensure the credentials are correct. 3. Once the 403 errors stop in the logs, the system should clean itself up automatically. 4. If it doesn't, use the 
./meta_tool
command provided above to force a manual scan and delete. References • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md • administration/management/monitoring/metrics-shared-data.md
2 Views
Open in Slack
PreviousNext
Powered by