http://starrocks.io logo
Join Slack
Powered by
This message was deleted.
# questions-and-troubleshooting
s
This message was deleted.
r
This error (
BE:21004
with "The specified key does not exist") indicates a metadata inconsistency between the StarRocks Frontends (FE) and your S3 object storage. In a shared-data cluster, the FE stores the "logical" metadata (which tablets exist and what their latest version is) in its BDBJE journals. The "physical" metadata (the 
.meta
files) and actual data (segment files) are stored in S3. After a disaster recovery (DR) operation, this error usually happens if: 1. FE Metadata is ahead of S3: You restored the FE metadata (BDBJE) to a more recent state than the S3 bucket snapshot. The FE thinks the table is at Version 3, but the S3 bucket only contains files up to Version 2. 2. Incomplete S3 Restore: The S3 sync/restore process missed some files or is still in progress. 3. Vacuuming: The 
lake_autovacuum_grace_period_minutes
(default 1 day) expired during the downtime or recovery, and the cleaner deleted files that the FE still thinks are needed. Steps to Diagnose and Fix 1. Verify the file in S3 Use the AWS CLI or S3 Console to check if the specific path mentioned in the error actually exists: 
<s3://dev-am-data-iceberg/starrocks-shared/012582ff-4a44-40e5-837a-347c564bf0a1/db10004/10905/11037/meta/0000000000000000_0000000000000003.meta>
* If it exists: There is a permissions or configuration issue (check 
aws_s3_access_key
, 
region
, or 
endpoint
in 
fe.conf
and 
be.conf
). * If it does NOT exist: Proceed to the next steps. 2. Check for Metadata Inconsistency You can check the versioning status of the affected partition:
Copy code
sql
-- Replace with your table name
SHOW PARTITIONS FROM <table_name>;
-- Or check the system table (v3.1.9+)
SELECT * FROM information_schema.partitions_meta WHERE TABLE_NAME = '<table_name>';
Look for the 
VisibleVersion
. In your error, the system is looking for a meta file for version 3 (
..._00000003.meta
). If S3 only has 
..._00000002.meta
, your FE is out of sync. 3. Attempt "Recovery Mode" StarRocks has a metadata recovery mode designed to handle inconsistencies in shared-data clusters. 1. Stop all FE nodes. 2. In 
fe.conf
, add: 
metadata_enable_recovery_mode = true
. 3. Start the FEs. 4. Check the recovery progress:
Copy code
sql
    SHOW PROC '/meta_recovery';
This mode allows the FE to re-examine the state of tablets. If it identifies missing versions that cannot be recovered, it will report them. 4. Fix via Rollback (Manual) If you restored the FE from a backup and S3 from another, and they don't match, the safest "fix" is to re-restore the FE metadata using an older image/checkpoint that matches the timestamp of your S3 bucket restore. 5. Last Resort: Drop/Truncate If only specific tables or partitions are affected and you cannot find the missing 
.meta
files in any S3 backup: * Truncate the partition: This will reset the version and clear the metadata, allowing new data to be loaded.
Copy code
sql
    TRUNCATE TABLE <table_name> PARTITION (<partition_name>);
* Drop and Recreate: If the whole table is affected. Prevention Tip When performing DR for shared-data clusters, always ensure that the FE metadata backup and the S3 bucket snapshot are taken at the same time. If you are using StarRocks' built-in 
BACKUP
and 
RESTORE
(available for shared-data in newer versions), the system handles this synchronization for you. References • administration/Meta_recovery.md
c
@Rocky How do I ensure FE metadata backup and the S3 bucket snapshot are taken at the same time? There is an automated metadata snapshot does with 
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON
. I have never done an S3 bucket snapshot
r
Ensuring that FE metadata and S3 data are synchronized during a backup is exactly what the Cluster Snapshot feature (introduced in v3.4.2 for shared-data clusters) is designed to handle. If you are using a shared-data cluster and have enabled automated snapshots with 
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON
, you do not need to perform a separate "S3 bucket snapshot" in your cloud provider's console. StarRocks handles the coordination for you. How Synchronization Works When 
AUTOMATED CLUSTER SNAPSHOT
is enabled: 1. Coordinated Timing: StarRocks automatically triggers a snapshot every time the FE completes a metadata checkpoint (the process where it creates a new metadata image). 2. Metadata Upload: The FE metadata (which is normally local) is automatically uploaded to your specified S3 Storage Volume. 3. Data State: Since you are on a shared-data cluster, your data is already in S3. The snapshot captures the consistent state of both this remote data and the uploaded metadata image. 4. Frequency: The system defaults to taking these snapshots every 600 seconds (10 minutes). You can adjust this by changing the FE configuration 
automated_cluster_snapshot_interval_seconds
. Why you don't need a manual S3 Bucket Snapshot A traditional "S3 bucket snapshot" (like AWS Backup for S3) is usually unnecessary for StarRocks because: * StarRocks manages object lifecycle: The Cluster Snapshot feature records the exact files and versions needed to restore the cluster state. * Consistency: If you were to take a raw S3 bucket snapshot via AWS while the cluster is running, it might not align perfectly with the local FE metadata files on your disk. Using the 
ADMIN
command ensures they are captured at the same "logical" moment. How to Monitor the Snapshots To ensure your backups are actually happening, you can query these system views:
Copy code
sql
-- View the latest available snapshots
SELECT * FROM information_schema.cluster_snapshots;

-- View the status of snapshot jobs
SELECT * FROM information_schema.cluster_snapshot_jobs;
Key Considerations * Storage Volume: If you didn't specify a volume in your command, it uses the default storage volume. You can specify a dedicated one like this: 
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME my_backup_s3_volume;
* Retention: Currently, StarRocks automatically maintains the latest snapshot and purges the previous one to save space. * Restoration: If you ever need to restore, you would start the Leader FE with the 
--cluster_snapshot
flag and point it to the snapshot URI in S3. If you are using a shared-nothing cluster (where data is stored on BE local disks), you should use the standard 
BACKUP
and 
RESTORE
commands instead. However, since you mentioned 
AUTOMATED CLUSTER SNAPSHOT
, it's almost certain you are on a shared-data architecture where this automated synchronization is built-in. References • sql-reference/sql-statements/backup_restore/BACKUP.md • administration/cluster_snapshot.md
c
@Rocky The snapshot gets overwritten every 10 minutes. But when I restored with the latest snapshot the metadata was all wrong. It was looking for files in a db10004 when the actual data files are in db10005
r
It sounds like you are experiencing a common issue when metadata and data get out of sync, typically occurring when manual "snapshots" (like copying the FE 
meta
directory) are used instead of the built-in StarRocks 
BACKUP
and 
RESTORE
commands. Why this happens In StarRocks, the Frontends (FE) manage metadata and assign internal IDs to databases, tables, and partitions. The Backends (BE) store the actual data in directories named after these IDs (e.g., 
/data/storage/data/10004/...
). If you are "snapshotting" by backing up the FE's 
meta
directory every 10 minutes: 1. ID Mismatch: If a database was dropped and recreated in the last 10 minutes, it likely received a new ID (e.g., 
10005
). 2. Brain-Body Disconnect: When you restore an older version of the 
meta
directory, you are rolling back the FE's "brain." The FE now believes the database ID is 
10004
(the old ID), but the BE storage contains data for 
10005
. 3. Result: The FE sends requests to the BE to read from directory 
10004
, but the BE cannot find those files because they are in 
10005
. Recommended Solution: Use Built-in Backup/Restore To avoid this, you should use the official StarRocks 
BACKUP
and 
RESTORE
workflow. These commands are designed to handle ID remapping and ensure consistency across the distributed cluster. 1. Create a Repository:
Copy code
sql
    CREATE REPOSITORY `my_repo`
    WITH BROKER `my_broker`
    PROPERTIES (
        "path" = "<s3://my-bucket/backups>",
        "aws.s3.endpoint" = "...",
        "aws.s3.access_key" = "...",
        "aws.s3.secret_key" = "..."
    );
2. Back up the database:
Copy code
sql
    BACKUP SNAPSHOT my_db.snapshot_name
    TO `my_repo`;
3. Restore the database: The 
RESTORE
command will automatically map the metadata to the correct data paths, even if the internal IDs change.
Copy code
sql
    RESTORE SNAPSHOT my_db.snapshot_name
    FROM `my_repo`
    PROPERTIES ("backup_timestamp" = "2023-...");
How to fix the current state If you are currently stuck with a cluster where the metadata points to the wrong IDs: * Option 1 (Cleanest): If you have the data elsewhere (like a Parquet/CSV source), drop the problematic database and re-import the data. This ensures all IDs are fresh and consistent. * Option 2 (Advanced Metadata Recovery): If you must recover the current state, you can try starting the FE in Metadata Recovery Mode. 1. Add 
metadata_enable_recovery_mode = true
to 
fe.conf
. 2. Restart the FE. 3. Check the progress with 
SHOW PROC '/meta_recovery';
. 4. Note: This is a last resort and can lead to data loss if not handled carefully. A quick question for you: Are you using the SQL 
BACKUP
command, or are you manually copying the 
meta
folder/using disk-level snapshots? Knowing this will help me give you more specific recovery steps! References • administration/Meta_recovery.md
The issue you're describing—where metadata points to one database ID (
db10004
) but the physical data files are in another (
db10005
)—is a classic symptom of an ID mismatch between the FE's catalog and the BE's storage directory structure. In StarRocks, the internal 
db_id
and 
table_id
are used to define the directory paths on the BE disks (e.g., 
storage_root/data/<db_id>/<table_id>/...
). This mismatch typically happens in two scenarios: 1. Manual Metadata/Data Manipulation (Disk Snapshots) If you are performing "snapshots" by taking disk-level backups (like EBS/LVM snapshots) of the FE 
meta/
directory and BE 
data/
directories separately, or trying to "restore" by copying these folders manually: * The Problem: If the database was dropped and recreated between snapshots, or if you are restoring into a cluster where the database was created independently, the internal IDs will differ. Even if the name is the same (e.g., 
mydb
), the ID might change from 
10004
to 
10005
. * The Result: The restored FE metadata tells the BEs to look for tablets in folder 
10004
, but the BE disks contain folders labeled 
10005
. Solution: Always use the built-in 
BACKUP
and 
RESTORE
commands for moving data between clusters or points in time. These commands explicitly handle the mapping of IDs between the source snapshot and the target cluster. 2. Overwriting Snapshots in the Repository You mentioned the snapshot gets overwritten every 10 minutes. If you are using the StarRocks 
BACKUP
command to the same snapshot name in a repository: * The Risk: If a 
RESTORE
job is initiated while a 
BACKUP
is overwriting the same snapshot, or if there is a race condition in the repository (especially with S3 or object storage), the 
__meta
file in the snapshot might point to one state while the tablet files point to another. * The Problem: If you 
RESTORE
into a database that already exists, StarRocks checks for schema consistency. If you are trying to restore a "new" version of a database over an "old" one where the IDs have shifted (due to a drop/recreate), the 
RESTORE
process might fail or result in inconsistent metadata if the job doesn't complete correctly. Recommended Troubleshooting Steps: 1. Check IDs: Run 
SHOW PROC '/dbs';
to find the current 
DbId
of your database. Compare this with the folder names on your BE data paths. 2. Clean Restore: If you are using the built-in 
RESTORE
command, try restoring the database under a different name using the 
AS
keyword:
Copy code
sql
    RESTORE FROM my_repo.my_snapshot
    ON db_name AS new_db_name;
This forces StarRocks to create a fresh database and map the snapshot's IDs to the new 
db_id
correctly. 3. Validate Repository: Ensure that your 10-minute snapshot process isn't deleting the snapshot before the restore is finished. A 
RESTORE
job is asynchronous and needs the snapshot to remain stable in the repository until the state reaches 
FINISHED
. 4. Avoid Manual Meta Swaps: If you are manually swapping the 
meta/
directory, ensure the BE 
data/
directories are perfectly synced from the exact same point in time. Any discrepancy in the 
image
or 
edit_log
will lead to the FE and BE being out of sync regarding tablet locations. Note: If you are seeing this after a 
RESTORE
job that reported 
FINISHED
, please check the FE 
fe.log
for any "tablet meta" or "root path" errors, as this might indicate the BEs failed to move the files into the correct ID-based directory during the 
COMMIT
phase of the restore. References • administration/Meta_recovery.md • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/RESTORE.md
c
@Rocky I have a shared data databased., I am using ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON to do regular snapshots. The lated data was written yesterday but snapshots are continuously overwritten. I tested a restored today using a new cluster configured with disasterRecovery: enabled: true generation: 1 The recovery completed and my database was there. But when I queried the tables I got errors like s3://dev-am-data-iceberg/starrocks-shared/012582ff-4a44-40e5-837a-347c564bf0a1/db10004/10941/10943/meta/0000000000000000_0000000000000008.meta does not exist. Upon inspect it looks like all metadat was using db10004 and all the files in S3 are in db10005.
r
Based on the error message and the behavior you're describing, it appears that the FE metadata image you restored into the new cluster is out of sync with the actual data files stored on S3. In StarRocks shared-data clusters, every database and table is assigned an internal ID (e.g., 
10004
, 
10005
). These IDs are used as part of the directory structure in your S3 bucket. Why the mismatch is happening The error 
<s3://.../db10004/...> does not exist
while the files are in 
db10005
indicates that: 1. The metadata (restored from the snapshot) thinks the database ID is 
10004
. 2. The data on S3 was written when the database ID was 
10005
. This typically happens if the database was dropped and recreated in the original cluster (which assigns a new ID) but the snapshot you restored from was taken before that recreation, or the snapshot process did not successfully capture the state after the ID changed. Troubleshooting and Resolution Steps 1. Verify the available snapshots Before restoring, check which snapshots are actually available in your storage volume. You can do this in the original cluster (if still available) or by inspecting the S3 bucket directly. In StarRocks, you can query:
Copy code
sql
SELECT * FROM information_schema.cluster_snapshots;
Look for the 
snapshot_name
and 
created_time
to ensure you are picking the most recent one. 2. Ensure you are restoring the correct snapshot path When using the StarRocks Operator with 
disasterRecovery
, the operator needs to point to the correct metadata image path. In the automated snapshot feature, metadata images are stored at: 
/{storage_volume_location}/{service_id}/meta/image/automated_cluster_snapshot_{timestamp}
If you are using a new cluster, ensure that the 
disasterRecovery
configuration in your 
StarRocksCluster
YAML points specifically to the latest timestamped directory. If you point to a parent directory or an older timestamp, you will get stale metadata. 3. Check for 
service_id
consistency The 
service_id
(the UUID 
012582ff-...
in your path) is unique to the cluster. If you are restoring to a new cluster for disaster recovery: * The new cluster must be configured to use the original cluster's storage volume and path. * If you are performing cross-cluster recovery, you must provide a 
cluster_snapshot.yaml
(or equivalent operator config) that maps the old cluster's metadata to the new environment. 4. The "Continuous Overwrite" behavior StarRocks' automated cluster snapshot keeps only the latest snapshot by default to save space. If the "latest" data was written yesterday but the snapshot you restored shows an old DB ID (
10004
), it's possible that: * The automated snapshot task (governed by 
automated_cluster_snapshot_interval_seconds
, default 600s) failed to run or upload after the database ID changed to 
10005
. * The FE metadata checkpoint didn't trigger a new snapshot upload before the cluster went down. Suggested Fix 1. Inspect S3: Look at the 
.../meta/image/
directory on S3. Find the directory with the latest timestamp. 2. Update Operator Config: Ensure your 
disasterRecovery
spec points to that specific latest snapshot. 3. Manual Restore (If Operator fails to sync): If the automated recovery continues to pick the wrong ID, you may need to manually start the FE with the 
--cluster_snapshot
flag and a 
cluster_snapshot.yaml
file that explicitly defines the 
cluster_snapshot_path
to the correct S3 URI. Note: If the metadata for 
db10005
was never successfully snapshotted to S3, the metadata you have (for 
10004
) is effectively for a different "incarnation" of that database and cannot point to the files in the 
10005
directory. You would need to find a snapshot that was taken after the database ID became 
10005
. References • administration/cluster_snapshot.md
c
@Rocky I tried again and for the second time got same result Is there a bug in starrocks operator version 1.10 that affects shared-data restore?
r
The issue you are encountering is likely related to a fundamental architectural limitation: Standard SQL 
BACKUP
and 
RESTORE
commands are currently not supported for StarRocks shared-data (Cloud Native) clusters. According to the official StarRocks documentation: * Limitation: The 
RESTORE
statement explicitly states that "Shared-data StarRocks clusters do not support data BACKUP and RESTORE." * Operator Behavior: The StarRocks Operator's 
StarRocksRestore
custom resource (CRD) typically automates the execution of these SQL commands. Since the underlying engine does not support these commands in shared-data mode, the restore operation will fail or produce inconsistent results. Is there a "Bug" in Operator 1.10? While there isn't a specific "bug" in the sense of broken code for an existing feature, the "bug" might be the attempt to use the 
StarRocksRestore
resource on a shared-data cluster where it isn't supported. However, if you are looking for disaster recovery or migration for shared-data clusters, StarRocks has introduced a different mechanism: 1. Cluster Snapshot (Shared-data only): For shared-data clusters, you should use the Cluster Snapshot feature instead of the standard Backup/Restore. This feature ensures both data (already in object storage) and FE metadata (snapshotted to object storage) are preserved. 2. Metadata Recovery: In shared-data mode, your data is already persisted in S3/MinIO/HDFS. "Restoring" often involves starting a new FE with the existing object storage metadata or using the 
CREATE TABLE ... AS SELECT
or 
INSERT INTO ... SELECT
if moving between clusters. Recommended Next Steps: * Check logs: Review your FE logs (
fe.log
) or the Operator events (
kubectl describe starrocksrestore <name>
). You will likely see an error message such as 
Operation not supported for cloud native table
. * Verify Cluster Mode: Ensure you are not trying to restore a backup taken from a shared-nothing cluster into a shared-data cluster, as this transformation is not supported. * Use Cluster Snapshot: If you are on StarRocks v3.3+, look into the Cluster Snapshot feature for shared-data disaster recovery. If you can provide the specific error message from the 
StarRocksRestore
status or FE logs, I can give you more precise details! References • deployment/shared_data/feature-support-shared-data.md • administration/management/Backup_and_restore.md • administration/cluster_snapshot.md • faq/Exporting_faq.md
Open in Slack
PreviousNext
Powered by