@Rocky I get errors like these somewhat frequently

starlet err Object https://{pathtoblob}.dat does not exist: BE: {somebenumber}

, how should I start troubleshooting the cause?

The error

starlet err Object ... does not exist

typically indicates a metadata-to-storage inconsistency in a StarRocks Shared-data (Cloud-native) cluster. This happens when the metadata (stored in the FE) points to a data file (

.dat

) in object storage (S3, GCS, Azure Blob) that has been deleted or is inaccessible. Here is how you should troubleshoot the cause: 1. Check for Internal Garbage Collection (AutoVacuum) The most common cause is the StarRocks AutoVacuum process cleaning up "old" versions of data while they are still being referenced by long-running queries or background processes. * The Parameter:

lake_autovacuum_grace_period_minutes

(FE configuration). * What it does: It defines how long StarRocks keeps historical data versions after a compaction. If a query runs longer than this period, or if data is needed by a process after this window, the vacuum cleaner might delete the underlying

.dat

files. * Action: * Check your current setting:

ADMIN SHOW FRONTEND CONFIG LIKE "lake_autovacuum_grace_period_minutes";

* Recommendation: If you have long-running queries, increase this value to a period longer than your longest query (e.g., 60 or 120 minutes). Note that in older versions (pre-3.1.10/3.2.5), the default was only 5 minutes. 2. Verify External Deletion (Lifecycle Policies) Check if there are external factors deleting your files. * Lifecycle Policies: Check the Object Storage bucket configuration for TTL or Lifecycle rules. If you have a rule that deletes objects after X days, it will conflict with StarRocks' storage management and cause these errors. * Manual Deletion: Ensure no external scripts or manual operations are cleaning up the bucket. StarRocks manages its own data lifecycle; manual intervention in the data path will cause metadata corruption. 3. Check for Known Bugs (Primary Key Tables) In early 3.x versions, there was a known issue where

ALTER TABLE

operations (like adding columns) or heavy ingestion on Primary Key tables could lead to "Publish Version" failures or lost file references. * Action: If you are on an older version (e.g., v3.1.0 to v3.1.7), check your FE leader logs (

fe.log

) for "Publish timeout" or "Not found" errors during transaction commits. * Workaround: For Primary Key tables, try disabling the persistent index temporarily to see if the issue persists:

enable_persistent_index = false

(though upgrading to a stable LTS version like v3.2.x or v3.3.x is the recommended fix). 4. Inspect FE and BE Logs To find the exact moment the file was "lost," search the logs for the specific filename or path mentioned in the error: * FE Leader Log (

fe.log

): Look for

Vacuum

or

LakeService

tasks. This will tell you if the FE explicitly commanded the deletion of that object. * BE Log (

be.WARNING

or

<http://be.INFO|be.INFO>

): Look for

starlet

or

S3FileSystem

errors. This can help distinguish between "File Not Found" (404) and "Permission Denied" (403), both of which might sometimes surface as "does not exist." 5. Check Object Existence Manually Manually verify if the object mentioned in the error exists using your storage provider's CLI: * AWS S3:

aws s3 ls s3://{pathtoblob}.dat

* If the file DOES exist: The issue is likely a permission/network connectivity problem between the BE and the storage endpoint (check IAM roles/Secret Keys). * If the file DOES NOT exist: The issue is definitely a premature deletion (refer back to steps 1 and 2). Summary Checklist 1. Increase

lake_autovacuum_grace_period_minutes

to at least 60. 2. Disable any bucket lifecycle policies on the StarRocks data path. 3. Check FE logs for "Vacuum" operations occurring around the time of the error. 4. Verify permissions to ensure the BE has

GetObject

and

ListBucket

rights. References • administration/management/FE_configuration.md

@Rocky the partition was successfully created from an etl process and removing it and re-etl'ing recreates it without issue. Could it be that I am querying the table too soon after the partition was created and metadata on one of the CNs isn't updated yet?

It is very possible that your theory about "querying too soon" is correct, but the underlying cause is likely related to metadata consistency or automated cleanup (AutoVacuum) rather than just a simple propagation delay. In a StarRocks shared-data (Cloud-native) cluster, the error

starlet err Object ... does not exist

essentially means the Frontend (FE) told a Compute Node (CN) to read a specific data file, but the CN could not find that file in your object storage (S3/GCS/Azure). Here are the three most likely reasons this happens specifically "too soon" after an ETL process: 1. The "Read-After-Write" Metadata Lag (Multiple FEs) If you have multiple FEs (a Leader and several Followers/Observers) and use a load balancer: * The Scenario: Your ETL process connects to the Leader FE to commit data. The transaction finishes, and the Leader updates its metadata. If your query then hits a Follower FE via the load balancer immediately afterward, that Follower might still be a few milliseconds (or more) behind in replicating the metadata via BDBJE. * The Result: The Follower FE might try to generate a query plan based on a version of the metadata that is in an inconsistent state, or it might point to data files that were just reorganized by an immediate background compaction. * Troubleshooting: Check if you are using multiple FEs. Try pointing your query directly to the Leader FE to see if the error disappears. You can also check the FE setting

meta_delay_toleration_second

(default is 300s), but for high-frequency ETL, even a sub-second lag can cause issues. 2. Immediate Compaction + Aggressive Vacuuming This is the most common cause of 404 errors in shared-data mode. * The Scenario: As soon as your ETL finishes, StarRocks often triggers Compaction to merge the newly uploaded small files into larger ones. Once compaction creates a new "version," the old files (the ones just created by your ETL) are marked for deletion. * The Problem: If

lake_autovacuum_grace_period_minutes

is set too low (the default was 5 minutes in older versions, and sometimes users lower it further), the system might delete the "old" files while a query that started just as the compaction finished is still trying to read them. * Troubleshooting: Run

ADMIN SHOW FRONTEND CONFIG LIKE "lake_autovacuum_grace_period_minutes";

. Increase this to 60 or 120 to ensure files aren't deleted while queries are active. 3. Primary Key Table "Apply" Latency Are you using a Primary Key table? * The Scenario: In shared-data mode, Primary Key tables have a "Publish" phase (metadata update) followed by an "Apply" phase (updating the index on the CN). * The Problem: If you query the table after the "Publish" but before the "Apply" is fully finalized across the cluster, the CN might struggle to locate the correct version of the rowsets. * Troubleshooting: Check the FE logs (

fe.log

) for any

Publish timeout

or

Apply failed

warnings around the time of your ETL. Recommended Next Steps 1. Check FE Sync: If you use multiple FEs, check the

VisibleVersion

on each using

SHOW PROC '/dbs/{db_id}/{table_id}/partitions'

. Ensure they match the Leader. 2. Increase Vacuum Grace Period: Set

lake_autovacuum_grace_period_minutes = 60

in your FE config. 3. Check Object Storage Consistency: While S3 is strongly consistent, if you are using an older S3-compatible storage or a specific caching layer (like Alluxio or StarRocks' internal Data Cache), there might be a delay in the CN "seeing" the new file. Try disabling the Data Cache for a test:

SET datacache_enable = false;

in your session. 4. Examine the "dat" file path: Look at the error message again. If the path contains

/meta/

, it's a metadata file issue; if it ends in

.dat

, it's a data segment issue. Metadata file issues almost always point to the AutoVacuum or a race condition between compaction and the query. metadata sync between FEs.

@Rocky this issue continues, some information: • current_version() =

4.0.2-1f1aa9c

• table engine type = Primary Key •

lake_autovaccum_grace_period_minutes = 60

•

meta_delay_toleration_second = 300

• I checked and the

VisibleVersion

is the same between all 3 of our FE's

Given your environment (v4.0.2, Primary Key tables, high vacuum grace period), you are likely encountering a consistency race condition between the transaction commit (Publish) and the index update (Apply). In StarRocks Primary Key tables on shared-data, a load is not "fully" complete until the index is updated to point to the new data segments. Here is how to troubleshoot this specific scenario: 1. Enable Synchronous Apply By default, the "Apply" phase (updating the Primary Key index and Delete Vectors) can sometimes lag slightly behind the "Publish" phase. If you query immediately, the FE might generate a plan using a version that hasn't fully "landed" on the CNs' local index state. * Action: Check the FE parameter

enable_pk_sync_apply

. *

ADMIN SHOW FRONTEND CONFIG LIKE "enable_pk_sync_apply";

* Recommendation: Set this to

true

. This forces the Load transaction to wait until the Apply phase is finished across all relevant tablets before reporting "Success" to your ETL tool. *

ADMIN SET FRONTEND CONFIG ("enable_pk_sync_apply" = "true");

2. Differentiate between Data and Index (SST) files Look closely at the error message. Does the path include

/_idx_/

or end in

.sst

? * If it's a

.dat

file: This is a data segment. Since your vacuum grace period is 60m, it's unlikely to be a vacuum issue. It suggests the FE is pointing to a segment that was part of a transaction that perhaps failed or was overwritten by an immediate compaction. * If it's an

.sst

file: This is the Persistent Index. In 4.0, if you use

CLOUD_NATIVE

persistent index, the index itself is stored in the cloud. If there is a race between writing the index and reading it, you get this error. 3. Check for Data Cache Inconsistency Since you are on a shared-data (CN) architecture, the CNs use a local cache (

starlet_cache

). If a CN has a stale metadata entry or a corrupted cache reference for that specific blob path, it might return "Not Found" even if the file exists in S3. * Test: Disable the data cache for a single query session to see if the error persists:

sql
    SET datacache_enable = false;
    SELECT ... FROM your_table ...;

* If it works with cache disabled: You have a cache consistency issue. You might need to check if your CNs have enough disk space for

storage_root_path

or if there are "No space left on device" errors in

<http://be.INFO|be.INFO>

. 4. Investigate "Apply" Failures in CN Logs Even if the

VisibleVersion

is the same on all FEs, the Apply task might be failing on specific CNs. * Action: Search your CN logs (

<http://be.INFO|be.INFO>

) for the string

apply

or the specific

tablet_id

mentioned in the error. * What to look for: Look for

Primary key index apply failed

or

metadata checksum mismatch

. If the Apply phase fails, the CN may fall back to an inconsistent state where it tries to read files from a previous (now vacuumed) version. 5. Version Specific Note (4.0.2) You are running version

4.0.2-1f1aa9c

. As v4.0 is a cutting-edge release branch, there have been significant refactors to the Lakehouse Storage Engine (LakeService). * Check if you have Persistent Index enabled on object storage:

SHOW CREATE TABLE ...

and look for

persistent_index_type = "CLOUD_NATIVE"

. * If you are using

CLOUD_NATIVE

index, try switching one table to

LOCAL

to see if the 404s stop. This helps isolate whether the "missing object" is the data itself or the index file. Summary Troubleshooting Command Run this to see if the system is struggling with compaction or apply tasks during your ETL:

sql
-- Check for high compaction scores or long-running transactions
SHOW PROC '/transactions/{db_name}/running';

-- Check if tablets are in a healthy state
ADMIN SHOW REPLICA STATUS FROM your_table;

@Rocky I checked to see if I could find the

.dat

file that the error message is saying doesn't exist and it definitely doesn't. So what do you think happened? I wrote successfully to this table and partition at 13:12 or so then the next time I tried to query at 14:00 an hour later, I get this message. Maybe compaction created a new

.dat

file and deleted the old one but the reference isn't updated somewhere?

Based on your version (4.0.2) and the fact that you've confirmed the

.dat

file is physically missing from the bucket before your 60-minute grace period expired, you are almost certainly hitting a known Metadata/GC Race Condition bug that existed in early 4.0.x releases. What likely happened In StarRocks 4.0.x (shared-data), there were several edge-case bugs identified where the Garbage Collection (GC/Vacuum) logic for Primary Key tables could incorrectly identify a rowset as unreferenced or "lost" during a specific sequence of compaction and metadata updates. Specifically, there was an issue (tracked as Issue #70762 and #70727) where a race condition during "Apply" tasks or background disk-migration tasks caused the FE to lose track of certain rowset metadata. When the vacuum process ran, it saw these data files as "garbage" (not part of any known version) and deleted them from object storage, even if the grace period hadn't elapsed. Why "re-etl'ing" fixes it When you delete the partition and re-load, you are creating an entirely new metadata entry and a fresh set of

.dat

files with new names. The old "lost" references are gone, and the new ones are correctly registered—until the race condition occurs again during a subsequent compaction or metadata synchronization. Recommended Troubleshooting & Fixes 1. Upgrade to v4.0.9 or later (Highest Priority) Version 4.0.2 is an early release of the v4.0 branch. The issues mentioned above (#70762, #70727) have been explicitly fixed in later maintenance releases. * Fix: Upgrade to v4.0.9 or newer (or the latest LTS like v3.3.x if you prefer stability over v4 features). These versions include significant fixes to the "Lakehouse" metadata service to prevent premature file deletion. 2. Confirm the Race Condition in Logs To confirm this was the cause, search your FE leader logs (

fe.log

) and CN logs (

<http://be.INFO|be.INFO>

) for the following patterns around the time the file disappeared: *

Primary key index apply failed

*

rowset metadata loss

*

tablet meta version mismatch

*

GC race

3. Adjust Persistent Index Settings In v4.0, the Primary Key engine uses a Persistent Index. If you are using the

CLOUD_NATIVE

index type, there is more network metadata traffic than with

LOCAL

. * Check your table's index type:

SHOW CREATE TABLE {your_table};

* If you see

persistent_index_type = "CLOUD_NATIVE"

, try switching to

LOCAL

for this specific table to see if the frequency of 404 errors decreases. 4. Immediate Workaround:

enable_pk_sync_apply

Ensure the FE waits for all data to be fully "applied" to the index before marking a transaction as successful. This reduces the window for race conditions.

sql
-- Run this on your FE Leader
ADMIN SET FRONTEND CONFIG ("enable_pk_sync_apply" = "true");

Summary The 48-minute gap you saw (13:12 to 14:00) with a 60-minute grace period proves that the Vacuum logic was bypassed or misinformed by a metadata bug. Because the

.dat

file is gone from S3, StarRocks itself deleted it. Upgrading to v4.0.9+ is the only permanent solution to resolve these specific race conditions in the Primary Key shared-data Primary Key engine. References • administration/management/FE_configuration.md • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/BE_configuration.md