@Rocky - starrocks 3.5.20 - shared data mode. When restarting one of the nodes, I am getting [DefaultCoordinator.dealStatusToTryRetry():927] query af96ef80-c0f9-11f1-82b7-5e70bbafe142 failed: starlet err [RequestID=18DBBB78FACF058E][StatusCode=404]Get object s3://starrocks/a7f0ba56-f960-4602-bb98-e7be09c1fb37/db10029/10057/10059/meta/0000000000000000_000000000005F586.meta error: The specified key does not exist. How critical is it and will starrocks still be able to launch with such errors?

The error you are seeing,

StatusCode=404 Get object ... .meta error: The specified key does not exist

, indicates that StarRocks is attempting to access a specific version of tablet metadata in your S3 storage that is no longer present. In shared-data (lake) mode, StarRocks stores tablet metadata as versioned files in object storage. This error typically occurs when a query or background task (like a publish or compaction task) tries to access a metadata version that has already been cleaned up by the system's garbage collection (vacuum) process. How critical is it? * For Cluster Launch: This error is not critical for the node or cluster to launch. It appears in the logs of the

DefaultCoordinator

, which is an FE component responsible for managing and retrying query fragments. The error means a specific query or task failed, but it will not prevent the StarRocks FE or CN/BE services from starting. * For Data Integrity: It usually indicates a transient inconsistency or aggressive vacuuming. If it only happens once during a restart, it likely means a query that was "in-flight" before the restart was trying to reference a version that was deleted while the node was down. * For Service: If you see this error repeatedly for every query, it means your metadata vacuuming is too aggressive, and you may experience query failures. Why did it happen during a restart? When you restart a node, the FE may try to reschedule or resume tasks. If the node was down for a period longer than your metadata retention period, the metadata version the task was expecting might have been vacuumed. Recommended Actions 1. Check Query Success: Verify if your current queries are succeeding. If this was a one-time log entry during the restart, it can likely be ignored. 2. Adjust Vacuum Retention: If the error persists, increase the grace period for metadata retention. This ensures that older metadata versions stay in S3 longer, preventing queries from failing if they happen to reference a version that is being compacted/cleaned. * Modify the FE configuration:

lake_autovacuum_grace_period_minutes

(default is often 5-10 minutes). Increasing this to 30 or 60 minutes can help in unstable environments.

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");

3. Check S3 Consistency: Ensure there are no issues with your S3 provider's eventual consistency (though most modern S3-compatible stores are strongly consistent now). Summary: Your cluster will still launch and operate. This is a query/task-level error rather than a system-level fatal error. If the error persists across new queries, look into the

lake_autovacuum

settings. References • faq/Deploy_faq.md • faq/shared_data_faq.md • developers/code-style-guides/restful-api-standard.md • administration/management/BE_configuration.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md

@Rocky - another question - in a three-node shared-data 3.5.20 cluster using 3 FE and 3 CN nodes, deployed through kubernetes-operator, I've killed both follower nodes and increased their memory allocation, while keeping leader fe alive. CNs bounced and accepted new parameters, FEs that were bounced seem to have picked up the new parameters but are still replaying journals after 30 minutes. Will the cluster start

Since your Leader FE is still alive and the CN nodes have restarted and connected, your cluster is technically started and operational. The Leader FE can handle both metadata updates and queries. However, your high availability (HA) is currently degraded because the two Follower FEs are stuck in the "replaying journals" phase. Is it normal for replay to take 30 minutes? It depends on the volume of metadata and how long it has been since the last image (checkpoint) was created. * If the log IDs are progressing: Check your

fe.log

. If you see lines like

replayed journal id: 12345...

and the numbers are increasing, the Followers are simply processing a large backlog of changes. This is common if the cluster has many tablets or frequent metadata operations (like high-frequency loading). * If the logs are stuck: If the same journal ID repeats or there are no "replayed" messages, the process may be hung or encountering a BDBJE (Berkeley DB) issue. Will the cluster eventually start? Yes, the Followers will eventually transition to the

FOLLOWER

state once they catch up to the Leader's current journal ID. However, there is a risk of a Network Restore failure: If the Followers were down for a long time and the Leader has already performed a checkpoint and deleted the old journal files that the Followers need, the Followers will encounter an

INSUFFICIENT_LOG

error. In this case, they will never catch up automatically and will likely crash or stay in a loop. Recommended Steps 1. Verify Cluster Health: Connect to the Leader FE (using a MySQL client) and run:

sql
    SHOW FRONTENDS;

Check the

IsAlive

and

Join

columns for the other nodes. If the Leader is

true

, your cluster is functional for users. 2. Monitor Progress: Check the logs of the replaying FE pods:

bash
    kubectl logs <fe-follower-pod-name> -f | grep "replayed journal"

If the IDs are increasing, wait. Increasing memory (as you did) actually helps this process, as BDBJE replay is memory-intensive. 3. How to fix if they are "stuck" or too slow: If the replay is taking hours or shows an

INSUFFICIENT_LOG

or

RollbackException

in

fe.log

, the fastest fix is to trigger a Network Restore: * In Kubernetes (Operator): You may need to delete the PersistentVolumeClaim (PVC) associated with the Follower FEs. * When the FE pod restarts with an empty metadata directory, it will automatically perform a "network restore," which means it downloads the entire metadata image from the Leader instead of replaying incremental journals. This is often much faster than replaying 30+ minutes of logs. _ _Warning:* Only do this one node at a time and never delete the Leader's metadata. Summary * Cluster status: Up (via the Leader). * Criticality: Low, unless the Leader fails before the Followers catch up. * Action: Monitor

fe.log

for progress. If IDs aren't moving or you see BDBJE errors, wipe the Follower's metadata directory/PVC to force a clean sync from the Leader. References • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • administration/management/FE_management.md • faq/operation_maintenance_faq.md • deployment/starrocks_kubernetes_operator.md • administration/Monitor_and_Alert.md • faq/shared_data_faq.md References