http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> any thoughts on this error &gt; {"e...
# questions-and-troubleshooting
g
@Rocky any thoughts on this error
{"errorMessage":"java.lang.RuntimeException: TaskRun killed","errorTime":"2026-04-14 090550"}
from a pipe?
r
The error 
java.lang.RuntimeException: TaskRun killed
indicates that the internal task execution unit (TaskRun) responsible for your Pipe job was forcefully terminated by the StarRocks Task Manager. Since Pipe jobs in StarRocks are built on the asynchronous task framework, this usually happens for one of the following reasons: 1. Manual Intervention The most common cause is that the Pipe or the task itself was manually stopped. * Action: Check if someone executed 
ALTER PIPE <pipe_name> STOP
, 
DROP PIPE <pipe_name>
, or used 
CANCEL TASK
or 
KILL QUERY
on the specific Query ID associated with that task run. 2. FE Leader Switch Task management is handled by the Frontend (FE) leader. If the FE leader changed (due to a restart, network issue, or manual failover) around 
09:05:50
, the new leader might have killed the stale task runs from the previous leader to re-schedule them. * Action: Check your 
fe.log
on all FE nodes for messages like 
is transfer to leader
or 
is transfer to follower
at that timestamp. 3. Resource Group or Timeout Limits If you have resource isolation configured (Resource Groups), a task might be killed if it exceeds the group's 
concurrency_limit
or if the system is under heavy memory pressure and needs to reclaim resources. * Action: Check if there are other heavy loads running at the same time. Verify the 
pipe_driver_queue_len
and 
pipe_scan_executor_queuing
metrics if you have monitoring. 4. Task Conflict or Overlap If a Pipe task is stuck for too long and a new scheduling cycle attempts to start, or if there's a conflict in file processing, the manager might kill the current run. ────────── Troubleshooting Steps: 1. Check Task History: Query the 
information_schema.task_runs
table to see the status and 
EXTRA_MESSAGE
of the failed task:
Copy code
sql
    SELECT * FROM information_schema.task_runs
    WHERE ERROR_MESSAGE LIKE '%TaskRun killed%'
    ORDER BY CREATE_TIME DESC;
2. Check Pipe Status: Use 
SHOW PIPES
or check the 
information_schema.pipes
view to see the current state of the pipe. 3. Inspect FE Logs: Look at 
fe.log
and 
fe.warn.log
around 
2026-04-14 09:05:50
. Search for the string 
TaskRun
or the specific Pipe name to find the internal stack trace or the reason for the kill (e.g., 
cancel by user
or 
task timeout
). Is this recurring? If it happens once, it's likely a leader switch or a manual stop. If it's recurring, it may point to a resource limit or a specific file causing the task to hang until it's killed. References • using_starrocks/async_mv/materialized_view_task_run_details.md • developers/build-starrocks/Build_in_docker.md • faq/Sql_faq.md • sql-reference/information_schema/task_runs.md
g
@Rocky getting 
{"errorMessage":"task failed with unknown status","errorTime":"2026-04-15 11:04:21"}
from a pipe, any hints on how to dig in and troubleshoot?
r
The error 
{"errorMessage":"task failed with unknown status","errorTime":"2026-04-15 11:04:21"}
is a generic failure message from the StarRocks Pipe execution engine. It typically means a task was interrupted or crashed before it could report a specific success or failure state. Here is how you can dig deeper into the root cause: 1. Check the Pipe's Last Error First, look at the high-level status of the pipe to see if a more descriptive error was caught after the task failure.
Copy code
sql
SELECT PIPE_NAME, STATE, LAST_ERROR 
FROM information_schema.pipes 
WHERE PIPE_NAME = '<your_pipe_name>';
Or use:
Copy code
sql
SHOW PIPES WHERE NAME = '<your_pipe_name>' \G
2. Identify Failing Files If the pipe was processing multiple files, the "unknown status" might be linked to a specific corrupted or inaccessible file.
Copy code
sql
SELECT FILE_NAME, LOAD_STATE, ERROR_MSG, START_LOAD_TIME 
FROM information_schema.pipe_files 
WHERE PIPE_NAME = '<your_pipe_name>' 
  AND LOAD_STATE = 'ERROR';
3. Inspect Task Run History Pipes use the internal Task Framework. You can find detailed execution logs for these tasks in 
task_runs
. Look for the entry around 
11:04:21
.
Copy code
sql
SELECT 
    QUERY_ID, 
    TASK_NAME, 
    STATE, 
    ERROR_CODE, 
    ERROR_MESSAGE, 
    EXTRA_MESSAGE 
FROM information_schema.task_runs 
WHERE CREATE_TIME <= '2026-04-15 11:04:21' 
ORDER BY CREATE_TIME DESC 
LIMIT 10;
* Note: If 
STATE
is 
FAILED
but 
ERROR_MESSAGE
is empty, check the 
EXTRA_MESSAGE
column (which contains JSON metadata). 4. Search Frontend (FE) Logs The "unknown status" often occurs if the FE leader restarts or if there is an unhandled Java exception. * Log file: 
fe.log
or 
fe.warn.log
* What to look for: Search for the timestamp 
2026-04-15 11:04
and keywords like 
Pipe
, 
TaskRun
, or 
Exception
. * If you see 
Master Transfer
, it means the FE leader changed, which kills running pipe tasks. 5. Check BE Resource Usage If a Backend (BE) node runs out of memory (OOM) while processing a pipe's 
INSERT
task, the task might vanish, leading to an "unknown" state on the FE side. * Check 
<http://be.INFO|be.INFO>
logs on the nodes for 
Memory limit exceeded
or 
OOM killer
events. Common Causes for this specific error: * FE Leader Change: If the FE leader switched, all active pipe tasks are killed and may report this status. * Connection Timeout: If the pipe was reading from an external source (S3/HDFS) and the connection was severed abruptly. * Task Framework Cleanup: If the task was manually cancelled or timed out based on system-level task configurations. Recommendation: If the pipe is now in a 
SUSPENDED
or 
ERROR
state, you can try to resume it after checking the logs:
Copy code
sql
ALTER PIPE <pipe_name> RESUME;
References • sql-reference/information_schema/task_runs.md • sql-reference/information_schema/pipes.md • sql-reference/information_schema/pipe_files.md • sql-reference/information_schema/tasks.md • using_starrocks/async_mv/materialized_view_task_run_details.md • sql-reference/sql-functions/meta-functions/inspect_task_runs.md • sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md
g
ah @Rocky i've seen this before 
Getting analyzing error. Detail message: fail to begin transaction. Label [pipe-mypipe_8f23283c28244a769553ae45b193971b-task-1936512-0] has already been used..
though this seems like it should go away on a retry
r
While it seems like something that should resolve itself on a retry, this error indicates a conflict in the Global Transaction Manager (GTM) that might require a moment of patience or manual inspection. What happened? Every data-loading operation in StarRocks (including Pipe tasks) requires a unique Label. The label you see (
pipe-mypipe_...-task-1936512-0
) is automatically generated by the Pipe using its ID and the Task ID. The error 
Label [...] has already been used
means the FE (Frontend) has already registered a transaction with that exact name. This typically happens in two scenarios: 1. The "Ghost" Transaction: The previous attempt actually succeeded in reaching the FE and started a transaction, but the connection dropped before the FE could acknowledge it back to the Pipe's task manager. The Pipe thinks the task failed and retries it, but the FE says, "I'm already working on this label!" 2. FE Leader Switch: If your FE leader changed recently, the new leader might attempt to re-schedule "pending" tasks. If the old leader had already started the transaction, the new leader’s attempt will collide with the existing label. Will it go away on a retry? Not immediately. * If the previous transaction is still in a 
PREPARED
or 
COMMITTING
state, any retry with the same label will continue to fail. * If the previous transaction actually finished (COMMITTED), the Pipe will likely eventually move on to the next task ID or skip the file if it detects it was already loaded. * If the transaction aborted, StarRocks allows label reuse only after a brief internal cleanup or if the previous state is fully cleared. How to troubleshoot and fix: 1. Check the status of that specific label Run this to see if the previous attempt actually succeeded or is still "Running":
Copy code
sql
SHOW LOAD WHERE LABEL = 'pipe-mypipe_8f23283c28244a769553ae45b193971b-task-1936512-0';
* If state is 
FINISHED
, the data is already in. * If state is 
LOADING
or 
PREPARED
, a "zombie" task is still active. 2. Check for running transactions If the 
SHOW LOAD
doesn't give enough detail, check the internal transaction list:
Copy code
sql
SHOW PROC '/transactions/<your_db_id>/running';
_(You can find your DB ID via 
SHOW DATABASES
or 
SELECT DATABASE_ID FROM information_schema.databases
)_ 3. Check FE Logs Search your 
fe.log
for the string 
1936512-0
. If you see 
transfer to leader
, that confirms a leader switch caused the collision. Recommendation: Usually, waiting 1–5 minutes allows the internal transaction to either timeout (and abort) or finish. If the Pipe is stuck in an 
ERROR
state and doesn't auto-resume, you can manually kick it:
Copy code
sql
ALTER PIPE mypipe RESUME;
If this happens frequently, it might indicate that your FE is under heavy load or that the 
pipe_task_timeout
(default 1 hour) is being hit, causing the task to be killed and retried prematurely. References • faq/loading/Loading_faq.md • loading/loading_introduction/loading_concepts.md • sql-reference/information_schema/pipes.md • loading/loading_introduction/troubleshooting_loading.md • loading/Stream_Load_transaction_interface.md • loading/SQL_transaction.md • faq/loading/Broker_load_faq.md References • administration/management/monitoring/metrics.md
g
@Rocky is see several tasks sitting here 
SHOW PROC '/transactions/events/running';
r
When you see tasks "sitting" in 
SHOW PROC '/transactions/<your_db>/running'
, you are looking at the Global Transaction Manager's list of active data-loading transactions. In your case, the "events" database has active transactions that haven't reached a final state (COMMIT or ABORT). This is exactly why you're getting the 
Label already used
error—the Pipe is trying to start a new load with a label that the transaction manager still thinks is "In Progress." Why are they stuck? 1. Hanging BE Task: The Backend (BE) node might be struggling to finish the write or is stuck in an I/O wait (e.g., slow S3/HDFS connection), so it hasn't reported back to the FE. 2. Network/Connection Loss: The task was interrupted, but the FE hasn't timed out the transaction yet. Transactions have a default timeout (often controlled by 
stream_load_default_timeout_second
or the Pipe's internal settings), and until that expires, the label remains "locked." 3. FE Leader Transition: If the leader switched, the new leader might be waiting for the standard timeout to expire before cleaning up "orphaned" transactions from the previous leader. How to handle them: 1. Identify the Labels Check the output of your 
SHOW PROC
command. It should look something like this:
Copy code
text
+-------+-------------------------+-------+-----------+---------+---------------------+...
| TxnId | Label                   | State | Source    | DbId    | CreateTime          |...
+-------+-------------------------+-------+-----------+---------+---------------------+...
| 12345 | pipe-mypipe_8f23...-0   | PREPARE| STREAM_LOAD| 10001 | 2026-04-15 11:04:21 |...
+-------+-------------------------+-------+-----------+---------+---------------------+...
* State 
PREPARE
or `VISIBLE`: The transaction is nearly done but waiting for a final acknowledgment. * State `RUNNING`: The data is still being transferred/processed. 2. Manually Abort the Transaction (if necessary) If you know a task is stuck and you want to "unblock" the Pipe immediately, you can manually roll back the transaction using its label. Note that StarRocks doesn't have a 
KILL TRANSACTION
SQL command, but you can use the HTTP API or wait for timeout. Using the Curl API (replace with your details):
Copy code
bash
curl --location-trusted -u <user>:<password> \
    -H "label:<stuck_label_from_proc>" \
    -H "db:events" \
    -XPOST http://<FE_IP>:<FE_HTTP_PORT>/api/transaction/rollback
3. Adjust Pipe Timeout If this happens frequently, your Pipe tasks might be timing out internally before the transaction manager does. You can check your pipe configuration:
Copy code
sql
SHOW CREATE PIPE mypipe;
If your files are very large, ensure the Pipe has a sufficient timeout period so it doesn't "give up" and retry while the transaction is still active. Summary recommendation: If there are only a few and they aren't moving, wait about 5–10 minutes. Most internal load transactions have a default timeout. Once they transition to 
ABORTED
or 
FINISHED
in that 
SHOW PROC
view, the Pipe will be able to reuse the label (or move to a new one) and continue automatically. References • loading/SQL_transaction.md • faq/loading/Loading_faq.md • loading/Stream_Load_transaction_interface.md
g
ahh @Rocky i think this is the crux of my issue in 
/transactions/events/running
Copy code
Fail to publish partition 1889270 error Fail to publish version for tablets [1889274, 1889277]: prepare_primary_index: load primary index failed: Already exist: PersistentIndexMemtable<53> insert found duplicate key <longkey>, old_val 201863467336 old_ver 25 new_val 201863467337
be/src/storage/lake/lake_persistent_index.cpp:301 _memtable->insert(n, keys, values, version)
be/src/storage/lake/lake_persistent_index.cpp:806 insert(pkc->size(), reinterpret_cast<const Slice*>(pkc->raw_data()), values.data(), rowset_version), host: <myhost>
r
This is a specific and known issue related to Primary Key Index corruption or inconsistency in a Shared-Data (Lake) cluster. The error 
Already exist: PersistentIndexMemtable insert found duplicate key
occurs during the Publish Version phase. Even though Primary Key tables are designed to handle "duplicates" by performing an upsert, this error indicates that the internal index structure (which tracks where each key is located) has become inconsistent. It thinks it's trying to insert a key that shouldn't be "newly" appearing in that specific memory buffer, or it's hitting a conflict between the cached index and the actual data version. This is likely why your transactions are "sitting" in the queue—they cannot finish the publish phase, and they eventually hit the 
max_running_txn_num_per_db
limit (default 1000), causing the "fail to begin transaction" errors. How to Fix 1. Identify the Affected Tablets Your error message already identifies them: 
[1889274, 1889277]
. 2. Clear the Persistent Index (Recommended Fix) To resolve index inconsistency, you need to force StarRocks to rebuild the index for those specific tablets. The most reliable way is using the 
meta_tool
utility on the Backend (BE) or Compute Node (CN) that is hosting those tablets. Note: You must stop the BE/CN process before running 
meta_tool
because it needs to lock the metadata files. 1. Stop the BE/CN node where the error is occurring (the "host" in your log). 2. Run the meta_tool to delete the persistent index for the problematic tablets:
Copy code
bash
    # Replace <storage_root_path> with your actual storage path (e.g., /data/starrocks/storage)
    # Run this for each tablet ID identified in the error
    ./bin/meta_tool --operation=delete_persistent_index_meta \
                    --root-path=<storage_root_path> \
                    --tablet_id=1889274

    ./bin/meta_tool --operation=delete_persistent_index_meta \
                    --root-path=<storage_root_path> \
                    --tablet_id=1889277
3. Start the BE/CN node. When the node restarts and a new load starts, StarRocks will see the index is missing and automatically rebuild it from the data segments in object storage. 3. Alternative: Restart the Node If you cannot easily run 
meta_tool
, simply restarting the BE/CN node sometimes clears the 
PersistentIndexMemtable
if the corruption is only in-memory. However, if the inconsistency has been flushed to the local disk cache, the restart won't fix it, and you'll need the 
meta_tool
step above. 4. Clean up the Stuck Transactions Since your transaction queue is likely full, after fixing the index, you may need to clear the "zombie" transactions to allow the Pipe to start fresh: 1. Abort the stuck transactions using the Label/TxnId found in 
SHOW PROC '/transactions/events/running'
. 2. Resume the Pipe:
Copy code
sql
    ALTER PIPE mypipe RESUME;
Why did this happen? In Shared-Data mode, this can occasionally happen due to: * Interrupted Publish: A node crash or network failure during the critical "publish" phase of a very large transaction. * Version Mismatch: If multiple nodes were assigned the same tablet during a cluster scaling event or failover, and the local index caches became out of sync. * Disk/IO Issues: Local disk errors where the persistent index is cached. Long-term suggestion: If you see this frequently on version 3.1.x or 3.2.x, consider upgrading to the latest maintenance release (e.g., 3.3.x), as there have been significant stability improvements to the 
LakePersistentIndex
logic. References • administration/management/BE_configuration.md • loading/loading_introduction/troubleshooting_loading.md
g
@Rocky how would i stop the cn service before trying that meta_tool stuff?
r
To stop the Compute Node (CN) service and perform the manual index cleanup, follow these steps. 1. Stop the CN Service Go to the 
be
directory (CN uses the BE binary) on the specific host mentioned in your error log and run:
Copy code
bash
# Standard stop
./bin/stop_cn.sh

# OR for a graceful stop (recommended if you want to allow current queries to finish)
./bin/stop_cn.sh -g
Verification: Run 
ps aux | grep starrocks_be
to ensure the process is actually gone before running 
meta_tool
. If it hangs, you may need a 
kill -9 <pid>
, though try to avoid that if possible. ────────── 2. Run the 
meta_tool
 Once the process is stopped, you can run the tool to clear the persistent index for the problematic tablets (
1889274
, 
1889277
):
Copy code
bash
# Navigate to the be directory
cd /path/to/your/cn/be/

# Run the command for each tablet ID
./bin/meta_tool --operation=delete_persistent_index_meta \
                --root-path=/your/storage/path/ \
                --tablet_id=1889274

./bin/meta_tool --operation=delete_persistent_index_meta \
                --root-path=/your/storage/path/ \
                --tablet_id=1889277
* 
--root-path
: This should be the same as the 
storage_root_path
defined in your 
cn.conf
(or 
be.conf
). ────────── 3. Start the CN Service After the tool reports success, start the service again:
Copy code
bash
./bin/start_cn.sh --daemon
When the CN starts, it will realize the local persistent index is missing for those tablets and will automatically rebuild them from the data stored in S3/Object Storage. ────────── 💡 A Much Easier Alternative (No Restart Required) If your StarRocks version is 3.3.9 or higher, you don't actually need to stop the service or use 
meta_tool
. You can fix this directly via SQL from your MySQL client:
Copy code
sql
ALTER TABLE events.your_table_name 
DROP PERSISTENT INDEX ON TABLETS(1889274, 1889277);
This command clears the persistent index for those specific tablets across the cluster without any downtime. If you are on a recent version, try this first. After the fix: Once you've cleared the index (via either method), the "zombie" transactions sitting in 
/transactions/events/running
should eventually time out or you can manually abort them (as mentioned earlier) to clear the 
Label already used
error and let the Pipe resume. References • deployment/deploy_manually.md • administration/management/graceful_exit.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/BE_configuration.md References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md • knowledge/admin_and_operation/shared_data_compaction.md
g
@Rocky that sql is much easier, two questions: 1. how dangerous is dropping these persistent indexes 2. how do i prevent this in the future?
r
Great questions. Moving from the "manual surgery" of 
meta_tool
to the SQL command makes this much less stressful. 1. How dangerous is dropping these persistent indexes? In short: It is not dangerous to your data, but it has a temporary performance cost. * No Data Loss: The Persistent Index is just a "map" that tells StarRocks where each primary key lives (which file and which row). The actual data is stored safely in your object storage (S3/GCS/etc.) in rowsets. Dropping the index does not delete your data. * Automatic Rebuild: StarRocks is designed to be self-healing. When you drop the index for those tablets, the next time a load or a query hits that tablet, the Compute Node (CN) will realize the index is missing. It will then scan the existing data rowsets for that tablet and rebuild the index from scratch. * The "Danger" (Performance): The tablet will be "cold" for a moment. The first load after dropping the index will take longer because it has to perform this rebuild. If the tablet is very large (multiple GBs), you might see a spike in CPU and I/O on that CN during the reconstruction. * Why we do it: We only do this when the "map" (index) gets out of sync with the "ground truth" (data rowsets). Dropping it forces StarRocks to look at the ground truth and fix the map. ────────── 2. How do I prevent this in the future? This specific error (
Already exist: PersistentIndexMemtable...
) is usually a sign of a race condition or a version-tracking bug in the primary key logic, often triggered by high-pressure environments. Here is how to harden your cluster: A. Upgrade to a Stable 3.3.x Release If you are on an early 3.1 or 3.2 version, many of these edge cases (specifically "Publish Version" hanging or index corruption in Shared-Data mode) have been fixed in the 3.3.x branch (e.g., 3.3.8+). Version 3.3 introduced a much more robust "Cloud Native" index management. B. Use Cloud Native Persistent Index Ensure your table is using the cloud-native index type. This stores the index on S3 rather than just the local disk of the CN, which prevents inconsistencies when tablets move between nodes.
Copy code
sql
-- Check your table properties
SHOW CREATE TABLE events.your_table;

-- If it's not already, you can't easily "alter" to it, 
-- but for new tables, ensure this is set:
PROPERTIES (
    "enable_persistent_index" = "true",
    "persistent_index_type" = "CLOUD_NATIVE"
);
C. Monitor and Control "Compaction Score" When compaction falls behind, you get hundreds of "versions" of data. The more versions there are, the more complex the index updates become, increasing the chance of a collision. * Check scores: 
SELECT * FROM information_schema.partitions_meta WHERE MAX_CS > 100;
* Tune: If scores are high, increase 
compact_threads
in 
be.conf
or slow down your ingestion frequency to give the cluster time to merge files. D. Batching is Your Friend If you are using the Pipe or Stream Load with very frequent, tiny files (e.g., every 1 second), try to increase the batch size or the interval. Primary Key tables perform best when they have 30–60 seconds of data per "commit" rather than constant micro-updates. Recommendation: Run the 
ALTER TABLE ... DROP PERSISTENT INDEX...
now to unblock your production, and then plan an upgrade to the latest 3.3.x maintenance release to prevent the underlying logic bug from resurfacing. References • knowledge/admin_and_operation/shared_data_compaction.md
It is very common to be cautious with index operations, but for Primary Key tables, dropping a persistent index is a safe and standard recovery procedure. 1. How dangerous is dropping these persistent indexes? In short: It is not dangerous at all for data integrity. Here is why: * No Data Loss: The Persistent Index in StarRocks is not the data itself; it is just a "map" (a hash table) that tracks which Primary Key is located in which row of which file. * Automatic Rebuild: StarRocks treats the index as a cache/acceleration structure. If you drop it, StarRocks will simply realize it's missing during the next data load. It will then scan the existing data files (rowsets) for those specific tablets to rebuild the index from scratch. * Query Impact: Queries can still run while the index is missing or being rebuilt. However, the Primary Key table needs the index to perform "upserts" (Delete+Insert). * The Only "Risk": The first load that touches those specific tablets after the drop will take longer and consume more CPU/IO. This is because StarRocks must read all the primary keys from the existing data to recreate the mapping before it can finish the load. 2. How do I know when it’s done rebuilding? Unlike a 
SCHEMA CHANGE
(which you can track with 
SHOW ALTER TABLE COLUMN
), there isn't a specific "rebuild progress" bar for a persistent index. It happens as part of the Publish Version phase of your next load job. How to verify it's working: 1. Monitor the next Pipe/Load job: If the load succeeds without the 
Already exist: PersistentIndexMemtable...
error, the index has been successfully rebuilt for those tablets. 2. Check BE Logs: You can tail the 
<http://be.INFO|be.INFO>
log on the host node. You will see messages related to 
lake_persistent_index
or 
rebuild index
when the load starts. 3. Check Tablet Status:
Copy code
sql
    SHOW TABLET 1889274;
This won't show "indexing status," but it will confirm the tablet is 
NORMAL
. Final Recommendation: 1. Run the SQL: 
ALTER TABLE events.your_table DROP PERSISTENT INDEX ON TABLETS(1889274, 1889277);
2. Clean the Txn Queue: If the "sitting" transactions don't clear within 1-2 minutes, manually abort them using the 
curl
command provided earlier. 3. Resume Load: Your Pipe should then pick up the next file, rebuild the index for those two tablets in the background, and continue as normal. References • table_design/table_types/primary_key_table.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • best_practices/primarykey_table.md • administration/management/monitoring/metrics.md
g
@Rocky I tried hitting the api to cancel the transaction but got 
"Message": "class com.starrocks.common.StarRocksException: Can not abort COMMITTED transaction 1338889
r
This error occurs because of where the transaction is in its lifecycle. In StarRocks, once a transaction reaches the 
COMMITTED
 state, it has officially passed the "point of no return." Why you can't abort it The transaction flow is: 
PREPARE
-> 
COMMIT
-> 
PUBLISH
-> 
VISIBLE
. * COMMITTED means the Backend (BE/CN) has already confirmed to the Frontend (FE) that the data is written to disk/object storage. * The FE has now marked this transaction as "successful" in its metadata and is currently trying to Publish it (making it visible to queries). * Because the data is already "permanently" recorded, the system does not allow an `ABORT`—it must finish the 
PUBLISH
phase. How to resolve the "Stuck" transaction You don't need to manually cancel this transaction. Instead, you need to fix the index blockage that is causing the 
PUBLISH
phase to fail. Once you clear the persistent index (using the SQL or 
meta_tool
method we discussed), the following will happen: 1. FE Retries: The FE automatically retries "Publish" tasks for 
COMMITTED
transactions every few seconds. 2. Index Rebuild: On the next retry, the BE/CN will notice the persistent index for tablets 
1889274
and 
1889277
is missing. 3. Success: The BE will rebuild the index from the data files, the 
Already exist
error will disappear, and the Publish task will finally succeed. 4. Transaction Clears: The transaction will then move to the 
VISIBLE
state and disappear from your 
running
transactions list. Summary of Steps: 1. Drop the index:
Copy code
sql
    -- Try this first if on v3.3.9+
    ALTER TABLE <your_table> DROP PERSISTENT INDEX ON TABLETS(1889274, 1889277);
_(If the SQL fails or you are on an older version, use the 
meta_tool
method while the CN is stopped)._ 2. Wait 1-2 minutes: The FE will retry the publish task in the background. 3. Check status: Run 
SHOW PROC '/transactions/events/running';
again. The stuck transaction 
1338889
should clear on its own once the index is healthy. What if it still doesn't clear? If the transaction stays 
COMMITTED
forever even after fixing the index, check your Leader FE log (
fe.log
) for "publish" errors. It will tell you if it's still hitting the same error or a new one. References • knowledge/trouble_shooting/fe_deadlock_problems.md • administration/Meta_recovery.md
g
@Rocky still getting the same publish error after runnign 
ALTER TABLE <your_table> DROP PERSISTENT INDEX ON TABLETS(1889274, 1889277);
r
If the 
ALTER TABLE ... DROP PERSISTENT INDEX
SQL didn't clear the error, it's likely because the Compute Node (CN) still has a "dirty" state in its local memory or the local disk metadata is corrupted in a way that the SQL command couldn't gracefully override. Since the transaction is in a COMMITTED state, the Frontend (FE) will keep retrying the Publish phase indefinitely until the CN succeeds. To break this loop, you need to manually wipe the index metadata from the CN's local storage. Follow these exact steps to perform the "manual" cleanup: 1. Identify the CN Host Check your 
<http://be.INFO|be.INFO>
logs (on the CN) or the FE logs to confirm which specific CN host is throwing the 
Already exist: PersistentIndexMemtable
error for tablets 
1889274
and 
1889277
. You must run the following steps on that specific host. 2. Stop the CN Service The CN must be offline because the 
meta_tool
needs exclusive access to the local metadata database.
Copy code
bash
cd /path/to/your/starrocks/cn/
./bin/stop_cn.sh
Verify it is stopped: 
ps aux | grep starrocks_be
(CN uses the BE binary). 3. Use 
meta_tool
to Delete the Index Meta Run the tool for each tablet. The 
--root-path
must match the 
storage_root_path
defined in your 
cn.conf
.
Copy code
bash
# Repeat for both tablets
./bin/meta_tool --operation=delete_persistent_index_meta \
                --root-path=/your/storage/path/ \
                --tablet_id=1889274

./bin/meta_tool --operation=delete_persistent_index_meta \
                --root-path=/your/storage/path/ \
                --tablet_id=1889277
4. (Optional but Recommended) Manual Directory Cleanup If the 
meta_tool
finishes successfully, the local index files should be gone. However, as an extra precaution, check the physical directory for these tablets in your storage path. Look for a directory similar to: 
{storage_root_path}/persistent_index/{tablet_id}
If it still exists after running 
meta_tool
, you can manually 
rm -rf
the folder for those two tablet IDs. 5. Start the CN Service
Copy code
bash
./bin/start_cn.sh --daemon
What happens next? 1. FE Retries: Once the CN is back up, the FE will automatically send the "Publish Version" task for transaction 
1338889
again. 2. Rebuild: The CN will see that the local persistent index for those tablets is missing. Instead of throwing an "Already exist" error, it will now trigger a rebuild from the data stored in S3. 3. Success: Once the rebuild finishes (usually takes a few seconds to a minute depending on tablet size), the transaction will move to 
VISIBLE
and the Pipe will resume. If you still see the error after this: Please check if there are multiple storage paths in your 
cn.conf
. You must run the 
meta_tool
pointing to the specific root path where the tablet is stored. You can find the exact path by running 
SHOW TABLET 1889274;
in your MySQL client. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/BE_configuration.md • deployment/shared_data/s3.md • administration/management/resource_management/Resource_Group.md References • administration/management/FE_configuration.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • faq/shared_data_faq.md • deployment/shared_data/s3.md
g
@Rocky I'm seeing this in the fe logs, that seems good 
lake tablet 1889277 publish rebuild pindex version 25
r
Yes, that is exactly what you want to see! That log message is the "All Clear" signal indicating that the system is successfully self-healing. Here is what is happening under the hood: 1. Recovery Triggered: Because you dropped the persistent index (or cleared the metadata), the system recognized that the index for those tablets was missing. 2. Rebuilding: Instead of crashing with the "Already exist" error, the Compute Node (CN) is now scanning the data files for those tablets from S3/object storage to rebuild the index mapping from scratch. 3. Publishing: The 
publish rebuild pindex version 25
part means it is specifically fixing the index so it can finalize transaction version 25 and make it 
VISIBLE
. What to expect next: * Wait for completion: Depending on the amount of data in those specific tablets, this "rebuild" might take anywhere from a few seconds to a couple of minutes. * Transactions will clear: Once the rebuild for tablet 
1889277
(and 
1889274
) finishes, the 
COMMITTED
transaction that was stuck will finally transition to 
VISIBLE
. * Pipe resumes: Your StarRocks Pipe or routine load will automatically resume ingestion once that "blockage" version is published. Verification: Run this in a few minutes:
Copy code
sql
SHOW PROC '/transactions/<your_db_id>/running';
If the list is empty or the transaction ID 
1338889
is gone, you are fully back in business. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md
g
@Rocky the only problem is, i'm seeing that log immediately after
Copy code
2026-04-16 20:31:51.585Z ERROR (lake-publish-task-549686|20440206) [PublishVersionDaemon.publishPartitionBatch():637] Fail to publish partition 1889270 of txnIds [1338889, 1340827, 1342740, 1344690, 1351644, 1353575, 1355496, 1357408, 1359333, 1366204]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets [1889272, 1889278]: prepare_primary_index: load primary index failed: Already exist: PersistentIndexMemtable<53> insert found duplicate key long_key_here
536613837626632316131, old_val 201863465520 old_ver 25 new_val 201863465522
r
This is what we call the "whack-a-mole" stage of this recovery. In StarRocks Shared Data (Cloud Native) clusters, the Persistent Index is stored locally on the Compute Node (CN) to speed up upserts. When a partition's publish task runs, it attempts to update the index for all tablets in that partition. The reason you see a "success" log for tablet 
1889277
followed immediately by an error for tablets 
1889272
and 
1889278
is that the publish task is processing the tablets one by one. You fixed the first set, but the "Already exist" corruption likely exists in other tablets within that same partition. The Fix You need to repeat the index-clearing process for the new tablets mentioned in your error: 
1889272
and 
1889278
. Option 1: The SQL way (Easiest)
Copy code
sql
ALTER TABLE <your_table_name> DROP PERSISTENT INDEX ON TABLETS(1889272, 1889278);
Option 2: The 
meta_tool
way (If CN is stopped) If you are currently performing the manual cleanup with the CN stopped, run the 
meta_tool
for these two IDs:
Copy code
bash
./bin/meta_tool --operation=delete_persistent_index_meta --root-path=/your/storage/path/ --tablet_id=1889272
./bin/meta_tool --operation=delete_persistent_index_meta --root-path=/your/storage/path/ --tablet_id=1889278
Proactive Step: Find all tablets in this partition Instead of fixing them one by one as they error out, you can find all tablets belonging to partition 
1889270
and clear them all at once. 1. Find the tablet IDs:
Copy code
sql
    SHOW TABLETS FROM <your_table_name> WHERE PartitionId = 1889270;
2. Clear them all: Take the 
TabletId
list from that result and run:
Copy code
sql
    ALTER TABLE <your_table_name> DROP PERSISTENT INDEX ON TABLETS(<id1>, <id2>, <id3>...);
Why is this happening? This specific error (
Already exist: PersistentIndexMemtable... insert found duplicate key
) usually triggers when: 1. Disk was 100% full: If the CN's local storage filled up recently, the index files can get "truncated" or corrupted, leading to these version mismatches. Please check your CN disk space (
df -h
). 2. Version Bug: There were known edge cases in versions prior to 3.2.10/3.3.1 where index metadata could get out of sync during high-concurrency upserts. Once you clear the remaining tablets in that partition, the FE will retry, rebuild those indices from S3, and your 
COMMITTED
transactions will finally turn 
VISIBLE
. References • faq/loading/Loading_faq.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md • deployment/shared_data/feature-support-shared-data.md
g
@Rocky are these sensible pipe defaults? ('auto_ingest'='TRUE','batch_files'='256','batch_size'='1GB','poll_interval'='1200')
r
Those are very close to the default values for StarRocks Pipe, but with a significantly longer polling interval. Here is a breakdown of whether they are "sensible" for your use case: Breakdown of your properties: 1. 
'auto_ingest'='TRUE'
(Default: TRUE) * Sensible? Yes. This is required for continuous loading. It tells StarRocks to keep watching the source path for new files. 2. 
'batch_files'='256'
(Default: 256) * Sensible? Yes. This is a safe upper limit. It prevents a single "batch" from trying to process thousands of small files at once, which could overwhelm the FE scheduler or cause long-running transactions. 3. 
'batch_size'='1GB'
(Default: 1GB) * Sensible? Yes. This is the target data volume per transaction. For Primary Key tables (which you seem to be using based on your previous errors), 1GB is a healthy size that balances commit frequency and compaction efficiency. 4. 
'poll_interval'='1200'
(Default: 300) * Sensible? It depends. 1200 seconds is 20 minutes. * If you only care about the data being available every 20-30 minutes, this is fine and reduces the number of "List Objects" calls to your S3/GCS/Azure bucket (saving a tiny amount of money). * If you want near real-time data, this is likely too high. Most users leave this at the default 
300
(5 minutes) or lower it to 
60
(1 minute) for faster ingestion. When should you change these? * If you have many small files: If your upstream process dumps thousands of 10KB files, you might want to increase 
batch_files
so that each transaction covers more data, but usually, 
256
is a good sweet spot. * If you see "Version Count" errors: If you are hitting 
too many versions
errors on your table, you should increase 
batch_size
(e.g., to 
2GB
or 
4GB
) or increase 
poll_interval
to force StarRocks to group more data into a single commit. * If you want lower latency: Decrease 
poll_interval
to 
60
or 
120
. Summary Recommendation Your defaults are safe and conservative. The only thing to double-check is if a 20-minute delay (
poll_interval=1200
) is acceptable for your business requirements. If you want the data to show up faster, change it to 
300
. References • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md • loading/loading_introduction/loading_concepts.md • loading/hdfs_load.md • loading/s3.md • administration/management/BE_configuration.md • sql-reference/System_variable.md References • sql-reference/sql-statements/loading_unloading/pipe/CREATE_PIPE.md • loading/loading_introduction/loading_concepts.md • loading/hdfs_load.md • loading/s3.md • administration/management/BE_configuration.md • sql-reference/System_variable.md
g
@Rocky can i drop the prefix index for the entire table?
r
First, a quick terminology check: you likely mean the Persistent Index (pindex), as that is what your logs are referencing. In StarRocks, a "Prefix Index" (or Short Key Index) is something else entirely and cannot be manually dropped or rebuilt. Currently, StarRocks (starting from v3.3.9) does not have a single "DROP ALL" command for the persistent index. You must specify the tablet IDs. However, you can easily drop it for the entire table in one command by fetching all tablet IDs first. How to drop for the entire table: Step 1: Get all tablet IDs for the table Run this query to get a comma-separated list of all tablets belonging to your table:
Copy code
sql
SELECT GROUP_CONCAT(tablet_id) 
FROM information_schema.be_tablets 
WHERE table_name = 'your_table_name';
_(Note: If the list is extremely long, you might need to copy the IDs from the result of 
SHOW TABLETS FROM your_table_name;
)_ Step 2: Run the DROP command Copy those IDs into this command:
Copy code
sql
ALTER TABLE <your_table_name> 
DROP PERSISTENT INDEX ON TABLETS(id1, id2, id3, ...);
Why do this instead of tablet-by-tablet? Doing it for the entire table is actually recommended in your situation. Since you've already seen two or three tablets report the "Already exist" error, it is highly likely that other tablets in the same partition (or other partitions) are also out of sync. * Cleaning the slate: Dropping the index for the whole table ensures that every single tablet starts a fresh "rebuild" from the data stored in S3/object storage. * Consistency: It guarantees that you won't hit another "Already exist" error 10 minutes from now on a different tablet. What happens after you run this? 1. Immediate cleanup: The local (corrupt) index metadata on the Compute Nodes (CN) is deleted. 2. Automatic Rebuild: The next time a write (or the pending 
publish
task) hits those tablets, the CN will see the index is missing and trigger a rebuild. 3. Logs: You will see a flurry of 
lake tablet ... publish rebuild pindex version ...
logs in your FE. 4. Recovery: Once the rebuilds finish, your stuck transactions will finally transition to 
VISIBLE
. Important Note: Rebuilding the index for an entire large table can consume significant CPU and IO on your CN nodes for a few minutes while it scans the data files to re-map the keys. If this is a massive table, you might want to do it partition-by-partition during a low-traffic window. References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md • table_design/table_types/primary_key_table.md References • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • sql-reference/sql-statements/table_bucket_part_index/DROP_INDEX.md • table_design/indexes/inverted_index.md • table_design/indexes/vector_index.md
g
@Rocky I got pretty aggressive trying to solve this but still no solution. I dropped the persistent index on all partitions in the month this pipe is monitoring and even dropped the problematic partition but now trying to retry the pipe gives me 
Getting analyzing error. Detail message: fail to begin transaction. Label [mydata_uuid-task-2029649-0] has already been used..
r
This error happens because the StarRocks Pipe automatically generates a transaction label for every batch it attempts to load. When you manually triggered a "retry" or when the pipe tried to resume after your partition-dropping operation, it attempted to use a label that the Frontend (FE) still has in its recent history (likely from the task that failed during your index cleanup). StarRocks prevents reusing labels for 30 minutes (by default) to ensure data idempotency. The Solution: "Reset" the Pipe Since you have already dropped the partition and the persistent indices, the Pipe's internal metadata (tracking which files are "Finished") is now out of sync with your table's actual data. The cleanest way to fix this and bypass the label conflict is to re-create the pipe. 1. Stop and Clean up First, suspend the pipe to stop it from spamming retries:
Copy code
sql
SUSPEND PIPE <pipe_name>;
2. Check for "Hanging" Transactions Check if the transaction with the problematic label is still in a 
PREPARE
or 
COMMITTED
(but not visible) state.
Copy code
sql
SHOW PROC '/transactions';
-- Look for the label: mydata_uuid-task-2029649-0
If you find it and it's stuck, you can try to leave it alone (it will eventually timeout) or wait for the label to expire. 3. Re-create the Pipe (The "Hard Reset") Because you dropped the partition, the Pipe still thinks those files are "FINISHED" and won't naturally try to load them again unless you use 
RETRY
. To avoid the label issue and the metadata mismatch, use `CREATE OR REPLACE`:
Copy code
sql
CREATE OR REPLACE PIPE <pipe_name>
PROPERTIES (
    "AUTO_INGEST" = "TRUE",
    "BATCH_FILES" = "256",
    "BATCH_SIZE" = "1GB",
    "POLL_INTERVAL" = "300" -- Recommended to lower this from 1200
)
AS 
INSERT INTO <your_table>
SELECT * FROM FILES(...) 
-- Ensure you use the exact same S3/GCS path and credentials as before
Why this works: • 
CREATE OR REPLACE
wipes the Pipe's internal file-tracking state. • It will start scanning the source bucket from scratch. • It will see the files for the month you dropped and treat them as "New" files, assigning them brand new labels that won't conflict with the old ones. Alternative: If you don't want to re-scan the whole bucket If your bucket is massive and you only want to fix the specific files: 1. Wait 30 minutes: This is the default 
label_keep_max_second
. After this, the FE clears the label from its memory, and your 
RETRY
command will work. 2. Manually Abort: If you can find the transaction ID from the error logs or 
SHOW PROC
, you can try:
Copy code
sql
    -- Note: This is usually for manual Stream Loads, Pipes are harder to manually abort
    -- but check the transaction status first.
Recommendation Given you've already "cleaned the slate" by dropping indices and partitions, re-creating the pipe (Step 3) is the most reliable way to ensure the pipe starts fresh and successfully re-populates the missing data without hitting label collisions. One final check: Ensure your Compute Node (CN) disk space is healthy. Persistent index "found duplicate key" errors are almost always triggered by a disk-full event or a node crash that prevented the index from being flushed correctly to the first. References • sql-reference/sql-statements/loading_unloading/pipe/SHOW_PIPES.md • sql-reference/information_schema/pipe_files.md • sql-reference/information_schema/pipes.md • sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md • sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md
Open in Slack
PreviousNext
Powered by