@Rocky I'm getting this back as an error when I try to run a query, can you help me troubleshoot?

ERROR 1064 (HY000): starlet err [RequestID=1DQ3T5KJDS19N96C][StatusCode=404]Get object <s3://trc-starrocks-prod/e6/e2bc42ef-c214-4fe6-b2e4-13afd78cfe32/db10003/338390/9159534/meta/0000000000000000_0000000000004936.meta> error: The specified key does not exist.: BE:9737146

The error message indicates that a StarRocks Backend (BE/CN) failed to find a required metadata (

.meta

) file in your S3 bucket. This typically happens in shared-data (cloud-native) clusters. Based on the path in your error (

.../meta/0000000000000000_0000000000004936.meta

), StarRocks was attempting to read metadata for a specific tablet, but the object is missing from S3 (HTTP 404). Common Causes 1. Premature Vacuum (Garbage Collection): StarRocks has a background process that cleans up old rowsets and metadata files after compaction or deletion. A known issue in some versions (like v3.x) can cause the vacuum process to delete metadata files that are still referenced by in-flight queries or haven't been fully "pinned." 2. External Deletion or Lifecycle Policies: Manual deletion of files from the S3 bucket or an S3 Lifecycle Policy (e.g., "Delete objects after 30 days") can remove files that StarRocks still needs. 3. Metadata Inconsistency: A race condition between the Frontend (FE) and the storage layer might occur where the FE believes a specific version of a tablet is valid, but the underlying file was never successfully written or was already replaced/deleted. 4. Network or S3 Endpoint Issues: Occasionally, intermittent S3 API errors or incorrect region/endpoint configurations can result in a 404 if the request is routed to the wrong bucket or shard, though this usually manifests as a more persistent failure. Troubleshooting Steps 1. Verify file existence Confirm if the file actually exists in your S3 bucket. Use the AWS CLI:

bash
aws s3 ls <s3://trc-starrocks-prod/e6/e2bc42ef-c214-4fe6-b2e4-13afd78cfe32/db10003/338390/9159534/meta/0000000000000000_0000000000004936.meta>

* If it exists: There may be a temporary network issue or a configuration error regarding the S3 endpoint/region in your StarRocks cluster. * If it's missing: The file was deleted. Check your S3 Bucket Versioning or Object Lock settings to see if you can recover it. 2. Check for S3 Lifecycle Policies Ensure you do not have any S3 lifecycle rules that automatically delete or transition files in the StarRocks data path. StarRocks manages its own data lifecycle; external cleanup will cause metadata corruption. 3. Identify the affected Table/Tablet The path segments usually correspond to

db_id/table_id/tablet_id

. * Database ID: 10003 * Table ID: 338390 * Tablet ID: 9159534 You can find the table name by running:

sql
SELECT * FROM information_schema.tables WHERE table_id = 338390;

4. Examine Logs for "Vacuum" activity Check the leader FE's

fe.log

and the BE/CN logs for any errors related to "vacuum" or "garbage collection." If you see errors during metadata cleanup, it may explain why the file was removed prematurely. Potential Fixes * Upgrade: If you are on an older v3.x version, this may be a known bug related to vacuuming in-flight query metadata. Check for recent bug fixes in v3.3.x or v3.4.x releases. * Table Repair: If the data is critical and only a few tablets are affected, you may need to restore from a backup. If it's a materialized view, dropping and recreating it should resolve the issue. * Contact Support: If the file is missing and you didn't delete it, this suggests a storage engine bug. Collect your FE/BE logs and version information to share with the StarRocks community or support. References • deployment/shared_data/minio.md • quick_start/shared-data.md • faq/shared_data_faq.md • deployment/shared_data/s3.md • quick_start/routine-load.md

@Rocky the file does not exist, and the FE logs don't seem to suggest any issues vacuuming or garbage collecting

Since you've confirmed the file is physically missing from S3 and FE logs don't show explicit vacuuming errors, you are likely hitting a known race condition in StarRocks' shared-data (cloud-native) storage engine. In some versions (including v3.x), the background vacuum process can occasionally delete metadata versions that are still referenced by the Frontend (FE) or a running query before they are correctly "pinned." This often appears "silent" because the vacuum task itself completes successfully; it simply targets a file it thinks is safe to delete. 1. Identify the Extent of Damage Use the following command to see if other tablets in the table are also missing metadata:

sql
-- Replace <table_name> with your table name
ADMIN SHOW TABLET STATUS FROM <table_name> WHERE STATUS = 'MISSING_META';

If you need to find the table name from the IDs in your error message:

sql
SELECT table_name, database_name 
FROM information_schema.tables 
WHERE table_id = 338390;

2. Attempt a Repair (Rollback) For shared-data clusters, StarRocks provides an

ADMIN REPAIR

command specifically to handle lost metadata by rolling back the affected tablets to the last known valid version.

sql
ADMIN REPAIR TABLE <table_name> 
PROPERTIES ("enforce_consistent_version" = "true");

* What this does: It searches the history for a consistent version that exists for all tablets in the partition and rolls the table back to that point. * Note: This may result in the loss of the most recent data (the version that was "lost"), but it will restore the table to a functional state. 3. Prevent Recurrence To prevent the vacuum process from being too aggressive while you investigate or upgrade: 1. Increase Grace Period: Increase the time StarRocks waits before cleaning up old metadata. The default is often too short for long-running queries or complex publishing cycles. * In

fe.conf

, increase:

lake_autovacuum_grace_period_minutes = 720

(e.g., 12 hours). 2. Upgrade: This issue is addressed in newer maintenance releases (e.g., v3.3.7+, v3.4.x, and v4.0+) which implement more robust "version pinning" to ensure metadata isn't deleted while a query might still need it. 4. Check for "File Bundling" (If applicable) If you are on a very recent version (v3.5+), StarRocks uses file bundling for metadata. A missing

.meta

error can sometimes be a side effect where the "real" underlying file is missing, but the error reports the bundle. If

ADMIN REPAIR

fails, you may need to check the FE

sys_log

for "StarOS" or "Starlet" related RPC failures around the time the data was loaded. If the table is a Materialized View, the simplest and safest fix is often to

DROP

and

CREATE

the MV again to rebuild the metadata from scratch. References • quick_start/shared-data.md • quick_start/routine-load.md • deployment/shared_data/s3.md • administration/management/BE_configuration.md • deployment/shared_data/minio.md • faq/shared_data_faq.md

@Rocky attempting roll back failed with ERROR 1064 (HY000): java.net.SocketTimeoutException: Read timed out

The

java.net.SocketTimeoutException: Read timed out

during an

ADMIN REPAIR TABLE

indicates that the internal communication between the StarRocks Frontend (FE) and the storage management layer (StarOS/Starlet) is taking longer than the allowed timeout period. In shared-data clusters, the repair process must scan metadata versions in S3/Object Storage. If your table has many partitions, tablets, or a long version history, this scan can easily exceed the default internal timeouts. 1. Increase gRPC Timeout The most likely culprit is the gRPC timeout for the Starlet manager. The default is often only 5 seconds. Try increasing this value on the leader FE:

sql
-- Increase the timeout to 60 seconds (or more if needed)
ADMIN SET FRONTEND CONFIG ("starmgr_grpc_timeout_seconds" = "60");

_Note: If

ADMIN SET FRONTEND CONFIG

returns an error that the config is not mutable, you will need to add

starmgr_grpc_timeout_seconds = 60

to your

fe.conf

and restart the FE._ 2. Repair Partition by Partition If the table is large, don't repair the whole table at once. Repairing specific partitions reduces the metadata overhead and prevents timeouts. 1. Identify the affected partition from the

db_id/table_id/tablet_id

path in your original error or by running:

sql
    SHOW TABLET 9159534; -- Look for the 'PartitionName' column

2. Run the repair on just that partition:

sql
    ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>)
    PROPERTIES ("enforce_consistent_version" = "true");

3. Check for Version Divergence If strict consistency recovery (

enforce_consistent_version

=

true

) continues to fail because a uniform version cannot be found, you may need to allow the tablets to roll back to their individual latest valid versions. This maximizes data recovery but might leave some tablets at slightly different versions.

sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>) 
PROPERTIES (
    "enforce_consistent_version" = "false",
    "allow_empty_tablet_recovery" = "true"
);

4. Verify FE Logs If the timeout persists, check the

fe.warn.log

or

fe.log

on the leader FE. Look for a stack trace associated with

SocketTimeoutException

. It will often pinpoint whether the timeout is happening during an S3 operation or a gRPC call to a specific service. Summary Checklist • ☐ Increase

starmgr_grpc_timeout_seconds

to

60

or

120

. • ☐ Run the repair command on a single partition instead of the whole table. • ☐ Check if the

fe.log

mentions "StarOS" or "Starlet" RPC failures. • ☐ As a last resort, if it's a Materialized View, drop and recreate it. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/FE_configuration.md • administration/management/resource_management/Replica.md • faq/shared_data_faq.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md