hey @Rocky how would you suggest recovering from Fail to publish partition 859206 error Fail to publish version for tablets [859221]: delvec inconsistent tablet:859221, I think it's jamming up other transactions

The error

delvec inconsistent

is specific to Primary Key tables. It indicates that the Delete Vector (which tracks deleted rows for the Delete+Insert strategy) has become inconsistent with the tablet's metadata or data version on a specific replica. This error blocks the Publish Version phase of a transaction. Since StarRocks publishes versions sequentially per partition to ensure data consistency, one failing tablet can indeed "jam up" the publish queue for other transactions on that same partition. Suggested Recovery Steps To recover, you should identify the unhealthy replica and mark it as "bad" so StarRocks can automatically repair it from a healthy replica. 1. Identify the problematic Backend (BE) Check the status of tablet

859221

to see which Backend is reporting a version mismatch or error.

sql
SHOW TABLET 859221;

Look at the

DetailCmd

column in the result. It will provide a command like:

SHOW PROC '/dbs/123/456/partitions/859206/859207/859221';

Run that command to see the status of all replicas for this tablet. Look for: * Replicas with a lower

Version

than others. * Replicas where

LstFailedVersion

is not

-1

. * Any replica that shows

IsBad

as

true

. 2. Mark the inconsistent replica as "bad" Once you identify the

BackendId

of the replica that is inconsistent (the one causing the

delvec

error), use the following command to mark it as bad:

sql
ADMIN SET REPLICA STATUS PROPERTIES (
    "tablet_id" = "859221", 
    "backend_id" = "<THE_FAILED_BACKEND_ID>", 
    "status" = "bad"
);

_Replace

<THE_FAILED_BACKEND_ID>

with the actual Backend ID you found in step 1._ 3. Monitor the repair After marking it as bad, StarRocks will discard that replica and attempt to clone a fresh, healthy copy from the remaining replicas. You can monitor the progress with:

sql
SHOW PROC '/cluster_balance/tablet_sched/working_slots';

Or check the tablet status again after a few minutes:

sql
SHOW PROC '/dbs/.../859221'; -- (The DetailCmd from Step 1)

What if all replicas are inconsistent? If all replicas of the tablet report the same

delvec

inconsistency (meaning there is no "healthy" source to clone from), you may need to: 1. Drop and Restore the Partition: If you have a backup. 2. Truncate/Reload: If the data can be re-imported, truncating the affected partition and reloading is the cleanest way to clear the metadata corruption. 3. Check BE Logs: If this is happening frequently, check the BE

<http://be.INFO|be.INFO>

or

be.WARNING

logs for "primary key index" or "delvec" errors to see if there is an underlying hardware issue or a specific bug. Note: If the publish queue is severely backed up, you might see many transactions in

PREPARED

state. Once the tablet is healthy, the FE will attempt to clear the queue. If it remains stuck, a restart of the Leader FE can sometimes help clear the internal transaction manager state, but marking the replica as bad is the primary fix for this specific error. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_REPLICA_STATUS.md • administration/management/resource_management/Replica.md

Note @Rocky, this is shared data, does that change anything?

Yes, using Shared-data architecture (disaggregated storage and compute) in StarRocks does change several things compared to the traditional shared-nothing (local storage) setup. Here are the key differences and considerations: *1. Architecture & Nodes* * Compute Nodes (CN): In a shared-data cluster, you typically use Compute Nodes (CN) instead of Backend (BE) nodes. CNs are stateless; they handle the computation and caching but don't own the "source of truth" for the data. * Remote Storage: The data is stored centrally in Object Storage (like S3, GCS, Azure Blob, or MinIO) or HDFS, rather than on the local disks of each node. *2. Scaling and Elasticity* * No Data Rebalancing: This is the biggest advantage. When you add or remove CN nodes, StarRocks doesn't need to move data around (rebalance replicas) because the data stays in remote storage. Scaling can happen in seconds. * Cache Sharing: To prevent performance drops during scaling, StarRocks supports Cache Sharing, where nodes can fetch cached data from each other over the network instead of re-downloading it from remote storage. *3. Data Caching* * Data Cache: Performance relies on the local disk cache on the CNs. The first time you query data (cold read), it's fetched from remote storage. Subsequent queries (warm reads) hit the local cache. * Table Properties: You can control caching behavior at the table level using properties like: * `datacache.enable`: Whether to use the local cache (default is

true

). * `datacache.partition_duration`: How long data remains valid in the cache. * Warmup: You can proactively "warm up" the cache using the

CACHE SELECT

command to ensure high performance before users run queries. *4. Reliability & Replicas* * Storage Durability: Data reliability is offloaded to the remote storage system (e.g., S3's 99.999999999% durability). * No Multiple Replicas: Unlike shared-nothing where you might have 3 replicas of data on 3 BEs, shared-data usually keeps only one copy in remote storage (though the storage layer itself may be replicated). *5. Table Types & Metadata* * Cloud Native Tables: Tables in this architecture are internally identified as

CLOUD_NATIVE

. * Materialized Views: Support for synchronous materialized views in shared-data clusters was introduced in v3.4.0. Does it change your SQL or usage? * Syntax: Most DDL and DML (like

INSERT

,

SELECT

,

UPDATE

) remain exactly the same. * Table Creation: When creating tables, you don't need to worry about replica counts as much, but you should consider your caching strategy (

datacache.enable

). * Performance Tuning: Instead of tuning disk I/O on BEs, you focus on Cache Hit Rates and Network Bandwidth between your compute nodes and your object storage. If you are transitioning an existing workload to shared-data, the primary "change" is the shift from managing disk space for storage to managing disk space for cache. References • data_source/data_cache.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md • deployment/shared_data/hdfs.md • data_source/External_table.md • quick_start/shared-data.md

I'm asking if your response to repair this hey @Rocky how would you suggest recovering from Fail to publish partition 859206 error Fail to publish version for tablets [859221]: delvec inconsistent tablet:859221, I think it's jamming up other transactions changes in shared data arch?

Hello! I'm Rocky, and I can help you with that error. The error "Fail to publish version... delvec inconsistent" in a Shared-Data (compute-decoupled) architecture typically indicates a mismatch between the Delete Vector (delvec) state in your object storage and what the FE (Frontend) metadata expects for that tablet. This is specific to Primary Key tables and can indeed "jam up" subsequent transactions because StarRocks ensures strict version consistency for Primary Key operations. Here is a recommended plan to recover and unjam your transactions: 1. Identify and Clear Stuck Transactions Since you suspect it's jamming other transactions, first check for any long-running or failed-but-not-aborted transactions: * Run

SHOW TRANSACTION WHERE TableName = 'your_table_name';

to see if any transactions are in the

COMMITTED

state but haven't reached

VISIBLE

. * If you find a stuck transaction associated with partition

859206

, it may be blocking the version increment. 2. Clear Local Cache (Restart BEs/CNs) In Shared-Data mode, Compute Nodes (CNs) or BEs maintain a local cache for Primary Key indexes and Delete Vectors. If the local cache has become inconsistent: * Restart the CN/BE nodes that are hosting or processing the load for tablet

859221

. This often forces the node to reload the correct state from the object store and can resolve transient "inconsistent" errors. 3. Check for Compaction Conflicts Sometimes a background compaction and a heavy load can conflict on the delete vector. * Run

SHOW PROC '/compactions';

to check if there are multiple compaction tasks for that partition that haven't finished. * If you see two compaction transactions with empty

FinishTime

for the same partition, this is a known cause for publishing hangs in some 3.x versions. 4. Recovery via Configuration (BE Level) There is a specific BE configuration that can help with Primary Key recovery: * Check if

enable_primary_key_recover

is set to

true

in your BE/CN configurations. This allows the system to attempt a recovery of the PK index if inconsistencies are detected. 5. The "Nuclear" Option: Drop and Reload If the metadata for that specific partition (

859206

) is corrupted and blocking the entire table's progress: * Forced Drop: You can drop the problematic partition to clear the jam:

sql
    ALTER TABLE <table_name> DROP PARTITION <partition_name> FORCE;

Note: This will delete the data in that partition. You will need to reload the data for that specific partition after it's dropped. 6. Verify with

SHOW TABLET

To get more details on why tablet

859221

is failing, run:

sql
SHOW TABLET 859221;
-- Then use the DetailCmd from the result
SHOW PROC '/dbs/<db_id>/<table>/partitions/<part_id>/<index_id>/859221';

Look for the

LstFailedVersion

and

LstFailedTime

to confirm if it has been failing consistently. Recommendation: Start by restarting the CN/BE nodes to clear the memory state. If the "jam" persists, check for stuck transactions and consider the

DROP PARTITION ... FORCE

route if the partition data can be reloaded. References • administration/Meta_recovery.md • using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md • release_notes/release-3.3.md

@Rocky I rolled the CNs but it didn't seem to fix anything. I then cancelled the loads that were hung on 99% but I still see several transactions here

SHOW PROC '/transactions/events/running';

with

Fail to publish partition 859242 error No alive backend or compute node in warehouse name: default_warehouse.

, will they just go away?

The error message

Fail to publish partition [ID] error No alive backend or compute node in warehouse name: default_warehouse

indicates that the Frontends (FE) are attempting to complete the Publish Version phase of a transaction but cannot find any active Compute Nodes (CNs) or Backends (BEs) in the specified warehouse to handle the request. Here is what is likely happening and how to address it: 1. Will they "just go away"? Yes, eventually. Transactions in StarRocks have a lifecycle managed by the FE. * Timeouts: If the transactions were cancelled, they should transition to the

ABORTED

state. However, if they were already in the process of committing or publishing when you cancelled them, they might remain in the internal "running events" list while the FE retries the publish action or waits for a timeout. * Retry Logic: StarRocks aggressively retries the

PUBLISH

phase because once a transaction is

COMMITTED

, the system must ensure the data becomes

VISIBLE

. If no nodes are alive in the warehouse, these retries will fail and log the error you see. 2. Why do I still see them after rolling CNs? When you "rolled" (restarted) the CNs, there was a period where no nodes were "Alive" from the FE's perspective. * If the FE still reports "No alive backend," check the status of your nodes using:

sql
    SHOW PROC '/compute_nodes';
    -- or if using standard backends
    SHOW BACKENDS;

* Ensure the

Alive

column is

true

. If they are alive now, the FE should eventually succeed in publishing those versions, and the transactions will move from

COMMITTED

to

VISIBLE

(and disappear from the "running events" list). 3. Verification and Manual Cleanup If they are stuck for an extended period (e.g., hours) and not progressing despite nodes being healthy: * Check Warehouse State: If you are using the Multi-Warehouse feature, ensure the

default_warehouse

is actually in the

RUNNING

state:

sql
    SHOW WAREHOUSES;

* Identify the Transactions: You can get more details on the specific stuck transactions by looking at the database-specific transaction list:

sql
    SHOW PROC '/transactions/<db_id>';

(You can find the

<db_id>

from

SHOW PROC '/transactions';

). 4. Known Causes * Empty Warehouse: If the

default_warehouse

has 0 nodes assigned to it, the FE cannot publish data to tables associated with that warehouse. * FE/CN Metadata Mismatch: Sometimes after a rapid roll, the FE's internal cache of "alive" nodes takes a few moments to heart-beat and update. Recommendation: First, verify that

SHOW PROC '/compute_nodes';

shows nodes as

Alive: true

. If they are alive, wait 5–10 minutes; the FE's

PublishVersionDaemon

should pick them up and clear the queue. If they still don't clear, check the FE leader's

fe.log

for any "database lock" or "publish version" related exceptions that might indicate a deeper stall. References • administration/management/FE_configuration.md • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • faq/loading/Broker_load_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md