@Rocky i have multiple queries failing because of a shared data base mv file not found, even after recreating the mv. the query is being re-written to this mv CacheHitRatio=100% W20260909 055816.666879 281468847908800 lake_service.cpp:381] Fail to publish version: Not found: starlet err [RequestID=W7FZHVAMGVSX26AQ][StatusCode=404]Get object s3://chf-starrocks/69d4c7ad-4cee-42f8-ad31-c92b3ab25064/db2176154/4292694/4292818/log/00000000004180E6_00000000001E6EDD.log error: The specified key does not exist. be/src/storage/protobuf_file.cpp:175 value_or_err_L175 be/src/storage/lake/tablet_manager.cpp:824 file.load(meta.get(), fill_cache) be/src/storage/lake/tablet_manager.cpp:845 value_or_err_L845 be/src/storage/lake/transactions.cpp:145 value_or_err_L145. tablet_info={tablet_id: 4292838} txn_ids=txn_id: 1994461 commit_time: 1788325904 combined_txn_log: false txn_type: TXN_NORMAL force_publish: false gtid: 441427646109188096 version=3

The error log indicates that a StarRocks Backend (BE) is failing to publish a version for a shared-data (lake) tablet because a transaction log file (

.log

) is missing from S3 (HTTP 404). This usually happens when the metadata in the Frontend (FE) expects a certain data version to exist, but the physical file has been deleted or was never successfully uploaded. Since you mentioned this persists even after recreating the Materialized View (MV), here is how to troubleshoot and resolve the issue: 1. Identify the Source Table First, determine if the missing file belongs to the MV or the Base Table. * The error log mentions

tablet_id: 4292838

. * Run the following command in StarRocks to find the owner:

sql
    SHOW TABLET 4292838;

* Check the

DbName

and

TableName

in the output. * If it belongs to the old (deleted) MV: Your FE might be using a stale plan cache or the metadata wasn't fully cleared. Try running

FLUSH PCACHe;

or restarting the FE. * If it belongs to the Base Table: The base table itself has data inconsistency, and the MV refresh/query will continue to fail regardless of the MV's state. 2. Verify Base Table Health To confirm if the base table is the issue, try running your query with MV rewrite disabled:

sql
SET enable_materialized_view_rewrite = false;
-- Run your query here

* If the query still fails with the same "File not found" error, the base table is corrupted. You may need to restore the base table from a backup or re-import the data into a new table. * If the query succeeds, the issue is isolated to the MV's metadata or the rewrite logic. 3. Check for Aggressive Vacuuming In shared-data mode, StarRocks automatically cleans up old files. If the "grace period" is too short, files might be deleted before a transaction (like an MV refresh) can finish publishing. * Check your FE configuration:

ADMIN SHOW FRONTEND CONFIG LIKE 'lake_autovacuum%';

* If

lake_autovacuum_grace_period_minutes

is very low (e.g., the default is often 5 minutes), try increasing it to 30 or 60 minutes:

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "60");

4. Why "Recreating the MV" might not have worked If you dropped and recreated the MV but the error still references the exact same file path and tablet ID, it implies: 1. Stale rewrite: The optimizer is still trying to rewrite to the "ghost" of the old MV. 2. Base table issue: The "file not found" is actually coming from the base table tablets that the MV depends on. Recommended Next Steps 1. Verify Tablet Ownership: Run

SHOW TABLET 4292838;

to see exactly which table is broken. 2. Check S3 Directly: Ensure there are no S3 Lifecycle Policies on your bucket that are automatically deleting files in the

log/

or

meta/

prefixes. 3. Force Refresh: If the tablet belongs to the new MV, try a manual refresh with a clean state:

sql
    REFRESH MATERIALIZED VIEW your_mv_name;

4. Contact Support: If the base table is a large production table and

SHOW TABLET

confirms it is missing files, you may need to check BE logs (

<http://be.INFO|be.INFO>

) for "Vacuum" or "Garbage Collection" entries to see why the file was removed. References • knowledge/trouble_shooting/fe_deadlock_problems.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md

@Rocky the tablet belongs to a completely different materialized view not referenced in the original query

This situation confirms that the StarRocks query optimizer is attempting to use a different, corrupted Materialized View (MV) to rewrite your queries, even if you didn't explicitly reference it. When you run a query, StarRocks' Cost-Based Optimizer (CBO) scans all available MVs to see if any can satisfy the query more efficiently. If one of those MVs is in a corrupted state (e.g., missing transaction logs in S3), the query planning or the "publish version" phase for that MV can fail, causing your original query to crash. 1. Confirm the "Broken" MV Use the tablet ID from your error log to identify exactly which MV is causing the issue:

sql
SHOW TABLET 4292838;

Look at the

DbName

and

TableName

. This is the MV you need to address, not the one you recently recreated. 2. Why "Fail to publish version" occurs The error

starlet err ... Get object <...>.log error: The specified key does not exist

means the BE is trying to finalize a data update (publish a version) but the required transaction log file is missing from S3. * Cause A: The MV has a background refresh task that is failing repeatedly because its metadata and physical S3 files are out of sync. * Cause B: An aggressive S3 lifecycle policy or a manual deletion removed the

.log

files before the StarRocks garbage collector finished with them. 3. Immediate Workaround To stop this broken MV from affecting other queries immediately, disable MV rewriting globally or for your session:

sql
-- For your current session
SET enable_materialized_view_rewrite = false;

-- OR, disable the specific "broken" MV for rewrite
ALTER MATERIALIZED VIEW <broken_mv_name> SET ("enable_rewrite" = "false");

Once rewrite is disabled for the broken MV, the optimizer will ignore it, and your other queries should start succeeding. 4. Permanent Fix Since recreating the other MV didn't help, you must fix the one actually owning tablet `4292838`: 1. Drop the broken MV:

sql
    DROP MATERIALIZED VIEW <broken_mv_name>;

2. Clean up Metadata: If the drop hangs or doesn't resolve it, ensure no background tasks are stuck:

sql
    TERMINATE JOB <job_id> FROM <broken_mv_name>; -- If it's an asynchronous MV

3. Check for S3 Consistency: Ensure your S3 bucket does not have a "Short Retention" policy on the

log/

or

meta/

prefixes. Shared-data StarRocks relies on these files for transaction consistency. 4. Recreate the broken MV: Only after dropping it and verifying the errors stop should you attempt to recreate it. Summary The query fails because StarRocks tries to "help" by rewriting your query to a background MV that is physically broken. Identify that specific MV using

SHOW TABLET

, then disable its rewrite capability or drop it to restore service. References • faq/Deploy_faq.md • data_source/datalake_faq.md • administration/management/FE_configuration.md • administration/Meta_recovery.md • introduction/versioning.md • faq/datalake.mdx • deployment/shared_data/azure.md