Any idea on how to resolve issues like this @Rocky

com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[13300, 13296, 13305, 13301, 13297, 13298, 13304, 13302, 13303, 13299]], error msg: starlet err Object s3:/…../81365b09-4d47-4695-98ee-48648ce46d19/db13277/13293/13295/meta/0000000000000000_0000000000000003.meta does not exist, host: starrocks-cn

I mostly just want to cleanly reset this and prevent it in the future

The error

com.starrocks.rpc.RpcException: Fail to publish version... starlet err Object ...meta does not exist

occurs in StarRocks Shared-data (Cloud Native) clusters. It means the system is trying to finalize a transaction (Publish Version), but the metadata file (

.meta

) that describes the data segments in S3 is missing. This usually happens if a Vacuum/Garbage Collection task or a compaction job deleted the file prematurely, or if there was a metadata synchronization issue during an FE leader switch. 1. How to identify the affected data First, you need to find out which table and partition are affected using any of the Tablet IDs from your error message (e.g.,

13300

):

sql
SHOW TABLET 13300;

This will return the

DbName

,

TableName

, and

PartitionName

. ────────── 2. How to "Cleanly Reset" Since the

.meta

file is missing from S3, that specific version of the tablet is corrupted. To resolve the stuck "Publish Version" tasks and clear the error: 1. Drop the affected partition: The most reliable way to reset the state is to drop the partition and reload the data. Use the

FORCE

keyword to bypass the recycle bin and ensure the metadata is cleared immediately.

sql
    ALTER TABLE <table_name> DROP PARTITION <partition_name> FORCE;

2. Re-create and Reload: Re-add the partition and reload the data from your source.

sql
    ALTER TABLE <table_name> ADD PARTITION <partition_name> VALUES ...;

Note: If the table is unpartitioned, you may need to truncate the table or drop/recreate it. ────────── 3. How to prevent this in the future This issue is often caused by the Vacuum process being too aggressive or a known bug in specific versions. A. Increase Vacuum Grace Period The Vacuum process cleans up old files. If it runs too soon after a version is replaced (e.g., during high compaction or long-running loads), it might delete files still needed by a pending Publish task. Increase the grace period in your FE configuration (default is often 5–30 minutes):

sql
ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440"); -- Set to 24 hours

B. Check for known bugs / Upgrade * Version 3.3.10 was specifically "Yanked" (taken offline) because of a bug that caused metadata loss in shared-data clusters during FE leader shifts. * If you are on a version around 3.3.x, 3.2.x, or 3.1.x, ensure you upgrade to the latest patch release (e.g., 3.3.11+, 3.2.13+), which includes fixes for "metadata loss after compaction" and "missing meta files." C. Monitor Compaction If the error persists, check if there are stuck compactions that might be conflicting with your loads:

sql
SHOW PROC '/compactions';

If you see partitions with multiple compaction tasks where

FinishTime

is empty, it indicates a hang that compaction is stuck and may be contributing to the metadata inconsistency. Dropping the partition (Step 2) is the standard fix here as well. References • sql-reference/sql-statements/table_bucket_part_index/SHOW_TABLET.md

@Rocky Is there a way to reset the database or all the metadata cleanly? I dont care if I delete data

If you want to perform a total "nuclear" reset and you do not care about the existing data, the cleanest way to resolve metadata inconsistencies is to wipe the cluster state and start fresh. Depending on how much "resetting" you want to do, here are two approaches: Option 1: The "Safe" Clean Reset (SQL Only) If you want to keep your cluster running but wipe all data and clear the metadata stuck in the "recycle bin," follow these steps. 1. Drop all Databases with FORCE: Using

FORCE

ensures that StarRocks bypasses the "Recycle Bin" and immediately attempts to delete both the metadata and the physical files in S3.

sql
    DROP DATABASE <db_name> FORCE;

2. Clear the Recycle Bin (Trash): Even with

FORCE

, sometimes references linger. You can speed up the background cleanup by temporarily lowering the retention settings in your FE configuration:

sql
    ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "1");

Note: Remember to set this back to a reasonable value (e.g.,

86400

for 1 day) after the cleanup is done. ────────── Option 2: The "Nuclear" Reset (Wipe Everything) This is the only way to guarantee a 100% clean state if your FE metadata has become corrupted or out of sync with S3. This will destroy all data and all users/permissions. 1. Stop all StarRocks processes: Stop all FE and BE (or CN) nodes. 2. Wipe Cloud Storage: Log into your S3 console (or use

aws s3 rm

) and delete everything inside the bucket path configured for your cluster. _ _Path example:*

<s3://your-bucket/starrocks-cluster-id/>

3. Wipe Local FE Metadata: On every FE node, delete the contents of the

meta_dir

(defined in your

fe.conf

).

bash
    rm -rf /path/to/starrocks/fe/meta/*

4. Wipe Local BE/CN Data: On every BE/CN node, delete the contents of the

storage_root_path

(defined in

be.conf

). For Shared-data, this mostly contains cache, but clearing it ensures no stale metadata remains.

bash
    rm -rf /path/to/starrocks/be/storage/*

5. Restart the Cluster: * Start the Leader FE first. It will initialize a brand-new cluster ID and metadata structure in the empty S3 path. * Add your BE/CN nodes back to the cluster using

ALTER SYSTEM ADD COMPUTE NODE ...

(or

BACKEND

). How to prevent this in the future The "meta does not exist" error is almost always a result of premature garbage collection. To prevent it: 1. Upgrade: Ensure you are not on version 3.3.10 (which was yanked due to this exact bug). Move to 3.3.11+ or 3.2.13+. 2. Increase Vacuum Grace Period: In Shared-data mode, StarRocks automatically deletes "expired" files. If you have high load or frequent compactions, increase this window to 24 hours to give the FE plenty of time to publish versions:

sql
    ADMIN SET FRONTEND CONFIG ("lake_autovacuum_grace_period_minutes" = "1440");

References • administration/Meta_recovery.md

@Rocky Is there a way to limit CPU usage in starrocks docker containers?

To limit CPU usage in StarRocks Docker containers, you should use a combination of Docker resource constraints and StarRocks internal configurations. Starting with v3.2.0, StarRocks is "container-aware" and respects cgroup limits (CPU quota and cpuset), but manual tuning is often recommended for the best stability. 1. Docker-Level Limits (Hard Limits) You can restrict the CPU resources available to the container using standard Docker flags or Docker Compose. * Docker Run: Use

--cpus

to specify how many CPUs the container can use.

bash
    docker run -d --name starrocks-be --cpus="4" starrocks/be-ubuntu:latest

* Docker Compose:

yaml
    services:
      starrocks-be:
        image: starrocks/be-ubuntu:latest
        deploy:
          resources:
            limits:
              cpus: '4.0'

2. StarRocks BE Configuration (

be.conf

) While StarRocks attempts to auto-detect limits, it is best practice to explicitly set the core count in your

be.conf

to match your Docker limit. This ensures the BE correctly sizes its internal thread pools (like

pip_exec

and

pip_scan

). *

num_cores

: Set this to the number of CPUs allocated to the container.

properties
    # In be.conf
    num_cores = 4

If set to

0

(default), StarRocks auto-detects the cores via

/proc/cpuinfo

and cgroup limits. 3. StarRocks Resource Groups (Workload Isolation) If you want to limit CPU usage for specific users or queries (so one query doesn't peg the container at 100%), use Resource Groups. From StarRocks v3.3.5 onwards, you can set a hard CPU limit for a group:

sql
CREATE RESOURCE GROUP query_limit_group
PROPERTIES (
    "cpu_core_limit" = "2",        -- Limit this group to 2 cores
    "exclusive_cpu_cores" = "2",   -- Hard isolation (v3.3.5+)
    "mem_limit" = "20%"            -- Memory limit
);

-- Bind a user to this group
ALTER USER 'developer' SET ARCHIVE_CORE_RESOURCE_GROUP = 'query_limit_group';

4. Query-Level Parallelism (

pipeline_dop

) The

pipeline_dop

variable controls how many threads a single query can use. If your container is limited to 4 cores but

pipeline_dop

is set too high, you will experience heavy context switching. * Default Behavior: In newer versions, StarRocks sets this adaptively. * Manual Override: You can set this globally or per session. A common rule of thumb is to set it to half the number of cores allocated to the container.

sql
    SET GLOBAL pipeline_dop = 2; -- For a 4-core container

Summary Checklist 1. Docker: Use

--cpus="X"

to enforce the limit. 2. be.conf: Set

num_cores = X

to help StarRocks size thread pools. 3. Resource Groups: Use

cpu_core_limit

if you need to partition the container's CPU between different workloads. 4. Verify: Run

SELECT * FROM information_schema.be_threads;

to see how many threads StarRocks has actually spawned and if they match your expectations. References • faq/Others.md • administration/management/BE_configuration.md • deployment/plan_cluster.md • administration/management/resource_management/resource_group.md • deployment/environment_configurations.md