http://starrocks.io logo
Join Slack
Powered by
Hey <@U0A71G31CDV> i am on Starrocks 3.5.16 upgrad...
# questions-and-troubleshooting
m
Hey @Rocky i am on Starrocks 3.5.16 upgraded from 3.5.11. i am using stream load to transfer data from my oracle source to my db in starrocks. my table are primary key types. i created some dags to Continuously fetch data from my source and transfer those data to my destination. the issues is that i may face some tablet error like below and i don't know why exactly i can prevent these. i also may encounter some transaction to be stuck in commited state for long without any raised error . i want you to guide me through these. here is an example: date_by_columnfalse 6 seg1 row4305249 del495728 bytes878861193 row_size0 compaction_score-104443272 compaction_level-1 partial_update_by_columnfalse 7 seg1 row1805721 del27673 bytes370654616 row_size0 compaction_score-73817420 compaction_level-1 partial_update_by_columnfalse 8 seg1 row2001359 del9801 bytes318956733 row_size0 compaction_score-42711347 compaction_level-1 partial_update_by_columnfalse 9 seg1 row2296866 del90240 bytes439795806 row_size0 compaction_score-84966165 compaction_level-1 partial_update_by_columnfalse 10 seg1 row1781908 del40707 bytes334422971 row_size0 compaction_score-27788700 compaction_level-1 partial_update_by_columnfalse 11 seg1 row1986831 del43477 bytes393997575 row_size0 compaction_score-82453694 compaction_level-1 partial_update_by_columnfalse 12 seg1 row1430268 del25784 bytes298732430 row_size0 compaction_score-3370144 compaction_level-1 partial_update_by_columnfalse 13 seg1 row2608861 del133 bytes505031996 row_size0 compaction_score-236467810 compaction_level-1 partial_update_by_columnfalse 14 seg1 row146325 del0 bytes33950788 row_size0 compaction_score234484668 compaction_level-1 partial_update_by_columnfalse 15 seg1 row4473 del0 bytes1445282 row_size0 compaction_score266990174 compaction_level-1 partial_update_by_columnfalse 16 seg1 row5020 del0 bytes1473390 row_size9456692 compaction_score266962066 compaction_level-1 partial_update_by_columnfalse be10.10.14.150 BE28721285
r
Hello! I'm Rocky. It sounds like you are encountering some of the common challenges when running high-frequency ingestion into Primary Key (PK) tables. Since you've upgraded to 3.5.16, you are on a very recent version, but the patterns you're seeing are typical when the ingestion rate exceeds the "Apply" and "Compaction" capabilities of the cluster. 1. Transactions Stuck in 
COMMITTED
State In StarRocks, a transaction for a PK table goes through: 
PREPARE
-> 
COMMITTED
-> 
VISIBLE
. * COMMITTED means the data has been written to the BEs and the transaction is technically successful. * VISIBLE means the Apply process (updating the PK index and generating Delete Vectors) has finished, and the data is now queryable. If transactions stay in 
COMMITTED
for a long time, it means the Apply thread pool is likely congested. This happens because: * High Frequency / Small Batches: You mentioned "continuously fetching" from Oracle. If you are sending many small Stream Loads (e.g., every second), you create a massive backlog of versions that need to be applied sequentially per tablet. * Resource Contention: The Apply process is CPU and IO intensive. If your BEs are busy with compaction or queries, the Apply tasks wait in the queue. How to guide through this: * Increase Batch Size: This is the most effective fix. Instead of many small loads, buffer your data and send larger batches (e.g., every 15–30 seconds or every 100MB+). * Check BE Configs: * Increase 
transaction_apply_worker_count
(default is CPU cores). If your CPU has room, you can increase this in 
be.conf
. * Increase 
transaction_publish_version_worker_count
. * Monitor the Queue: Check the BE logs (
<http://be.INFO|be.INFO>
) for keywords like 
publish version
or 
rejected from agent-task-pool
. You can also check the metric 
publish_version_queue_count
. ────────── 2. Tablet "Errors" and Negative Compaction Scores The snippet you provided shows segment-level metadata. The values like 
compaction_score-104443272
and 
compaction_level-1
look alarming, but here is what they usually mean in the context of PK tables: * Negative Compaction Score: In PK tables, the compaction score is often calculated based on the size of data in rowsets rather than just the number of versions (as in other models). In some 3.x versions, this reporting can suffer from an integer overflow in the display/monitoring UI or log, resulting in large negative numbers. While the number looks "broken," the underlying meaning is that the tablet has a very large amount of un-compacted data. * Compaction Level -1: This indicates that the segment has not yet been assigned to a specific level in the updatable compaction hierarchy. For segments just recently written and not yet merged, 
-1
is a common placeholder. * Row Size 0: This is often a reporting artifact when the tablet metadata hasn't been fully refreshed or when the segment consists mostly of deletes/updates that haven't been "finalized" in terms of the row-size calculation for the display. ────────── 3. Recommendations to Prevent These Issues To stabilize your Oracle-to-StarRocks pipeline on version 3.5.16, I recommend the following: 1. Batching (Crucial): Ensure your DAGs/Jobs are not firing Stream Loads too frequently. Aim for fewer, larger transactions. 2. Persistent Index: If you haven't already, ensure your PK tables have 
enable_persistent_index = true
. This moves the PK index from RAM to disk (SSD recommended). This prevents memory pressure and stabilizes the Apply phase for large datasets. 3. Adjust Compaction Threads: * In 
be.conf
, you can increase 
update_compaction_num_threads_per_disk
(default 1) to 2 or 4 if your IO and CPU can handle it. * Lower 
update_compaction_per_tablet_min_interval_seconds
(default 120) if you want the system to try compacting individual tablets more frequently.
4. Monitor Tablet Versions: Run the following to see if specific tablets are lagging:
Copy code
sql
    SELECT * FROM information_schema.be_tablets WHERE TABLE_NAME = 'your_table' ORDER BY NUM_ROWSET DESC;
If 
NUM_ROWSET
is consistently approaching 1000, you must reduce ingestion frequency. Summary for your next steps: Check your BE 
mem_tracker
to see if 
update
(PK Index memory) is high, and look at the 
<http://be.INFO|be.INFO>
log for "apply" or "publish" timeouts. If the transactions are stuck, the first thing to try is slowing down the frequency of your DAG triggers. References • administration/management/monitoring/alert.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md
m
how can i find out what is the reason that a transaction stuck on commited state? also how can i cancel a transaction? because when i cancel the related job to that txn like job 'insert_1234' the txn still remains even though the insert job is cancelled
@Rocky how can i find out what is the reason that a transaction stuck on commited state? also how can i cancel a transaction? because when i cancel the related job to that txn like job 'insert_1234' the txn still remains even though the insert job is cancelled
r
It is frustrating when transactions get stuck in the 
COMMITTED
state, especially when the original job has been canceled. In StarRocks, specifically for Primary Key (PK) tables, the 
COMMITTED
status means the data has been successfully written to the backends (BEs), but the *Apply/Publish phase*—where the PK index is updated and Delete Vectors are generated—is still pending. Here is how to diagnose the cause and handle these "ghost" transactions. 1. How to find the reason for a stuck 
COMMITTED
transaction When a transaction is in 
COMMITTED
but not 
VISIBLE
, the issue is almost always a bottleneck in the Publish Version or Apply thread pools. Step A: Check the Transaction Manager Run this command to see all currently active transactions in your database:
Copy code
sql
-- Replace <db_name> with your database name
SHOW PROC '/transactions/<db_name>/running';
* Look at the 
ErrMsg
field: This often contains messages like: * 
Partition's compaction score is larger than 100.0, delay commit...
(The system is intentionally slowing down to let compaction catch up). * 
tablet [12345] publish timeout
(One specific tablet is holding up the entire transaction). * Look at `ErrorReplicasCount`: If this is greater than 0, specific BEs are failing to complete the publish task. Step B: Inspect Backend (BE) Logs Search your 
<http://be.INFO|be.INFO>
logs on the BE nodes involved (look for the BE IDs mentioned in the tablet metadata you shared) for these keywords: * 
rejected from agent-task-pool
: This means the thread pool responsible for publishing versions is full. * 
apply
or 
rowset_commit_apply
: Look for long durations. If the "apply" phase for a PK table is taking minutes, it means the BE is struggling with the PK index or I/O. * 
Too many versions
: If you see this, the tablet has hit the 1000-version limit, and the system is refusing to make new versions visible until compaction runs. Step C: Check System Metrics Check the following metrics (via Prometheus/Grafana or 
curl http://<BE_IP>:<BE_HTTP_PORT>/metrics
): * `starrocks_be_update_apply_queue_count`: If this is high, the "Apply" tasks are queued up. * `starrocks_be_update_apply_thread_pool_queue_size`: Indicates thread pool saturation. ────────── 2. How to cancel a stuck transaction The reason the transaction remains after you cancel the job (like 
insert_1234
) is that in StarRocks, 
COMMITTED
is a point of no return. * PREPARE/LOADING State: If you cancel here, the transaction is Aborted, and data is cleaned up. * COMMITTED State: The FE has already told the BEs that the data is "safe." Even if the parent job is killed, the FE's 
DatabaseTransactionMgr
will keep trying to "Publish" that version until it succeeds or hits a critical failure. Can you force-kill it? 1. Manual Abort (Rarely works for 
COMMITTED
): You can try 
ROLLBACK
if you are using an explicit SQL transaction (
BEGIN
... 
COMMIT
), but for Stream Load, there is no "Abort Committed" command. 2. Wait for Timeout: Transactions have a timeout. If the "Publish" phase fails repeatedly for a long time (usually 24 hours by default for prepared transactions), it may eventually be cleaned up, but this is not immediate. 3. Restart the FE Leader: If a transaction is truly "zombie" (the BEs think it's done but the FE is stuck), restarting the Leader FE will force the transaction manager to re-initialize and potentially clear the hung state. Note: This is a last resort. 4. Clear Compaction Backlog: Since your snippet shows very high (or overflowing) compaction scores, the most likely reason the transaction is "stuck" is that it's waiting for a slot to finish. By increasing compaction threads (see previous response), the system may finally "find room" to move that transaction from 
COMMITTED
to 
VISIBLE
. Summary Checklist for your DAGs:
* Batching: If you are sending small updates from Oracle every few seconds, you will get stuck transactions. Increase your buffer to 30-60 seconds per load. * Persistent Index: Ensure 
enable_persistent_index = true
is set on your PK tables to speed up the Apply phase. * Check 
be.conf
: Ensure 
transaction_publish_version_worker_count
is at least 64 or more if you have many tablets. References • loading/SQL_transaction.md • sql-reference/System_variable.md • knowledge/admin_and_operation/shared_data_compaction.md
m
@Rocky i did all above you said but i still have one commited stuck txn. by experience, i have understood that the reason that this txn would not be finished is some problems with tablets or replicas. i want to check those suspected replicas that may cause this situation
r
To identify the specific replicas or tablets causing a transaction to hang in the 
COMMITTED
state, you need to look at the "Publish Version" phase details. In StarRocks, a transaction stays 
COMMITTED
until all (or a quorum of) replicas have acknowledged that they have applied the data and updated their version. Here is the step-by-step guide to finding the "suspected" replicas: 1. Identify the Transaction and Database IDs First, get the exact ID of the stuck transaction and your Database ID.
Copy code
sql
-- 1. Find your Database ID (DbId)
SHOW PROC '/dbs';

-- 2. List running transactions for that database
-- Replace <db_id> with the ID from step 1
SHOW PROC '/transactions/<db_id>/running';
Look for your transaction in the list. Note the 
TransactionId
. If the 
ErrorReplicasCount
is greater than 0, it confirms that some backends are failing the publish step. 2. Find the Problematic Tablet IDs (FE Logs) The easiest way to find exactly which tablets are blocking the transaction is to search the Leader FE logs (
fe.log
and 
fe.warn.log
). * Search for the Transaction ID: 
grep "txn_id=<your_txn_id>" fe.log
* Look for messages like: * 
publish version timeout
* 
publish version failed
* 
tablet <tablet_id> publish timeout
* 
replicas [10001, 10005] failed to publish
3. Inspect the Suspected Tablets Once you have a 
tablet_id
from the logs, use these commands to see the replica states:
Copy code
sql
-- Get the metadata for the tablet
SHOW TABLET <tablet_id>;
The output will have a column called 
DetailCmd
. Copy and run that command (it looks like 
SHOW PROC '/dbs/123/table/456/...'
). In the output of the 
DetailCmd
, check these columns for each replica: * 
Version
: The current visible version on that BE. * 
LstSuccessVersion
: The last version this BE successfully published. * 
IsBad
: If this is 
true
, the replica is corrupted. * 
State
: Should be 
NORMAL
. If it says 
CLONE
or 
ROLLUP
, the replica is busy with another task. The "Culprit": Any replica where the 
Version
is lower than the other replicas is the one holding up the transaction. 4. Check for "Error State" Tablets (Specific to Primary Key Tables) Since you are using Primary Key tables, tablets can enter an Error State if the "Apply" process (updating the PK index) fails.
Copy code
sql
-- Replace <db_id> with your actual Database ID
SHOW PROC '/statistic/<db_id>';
Look for the 
ErrorStateTablets
column. If it is not empty, it will list the tablet IDs that are broken. If a tablet is in an Error state, it cannot move from 
COMMITTED
to 
VISIBLE
. 5. Check the BE "Apply" Status On the Backend (BE) nodes identified as lagging, check the 
<http://be.INFO|be.INFO>
logs for: * 
apply task failed
: Indicates the PK index update failed (could be due to disk errors or memory issues). * 
rowset_commit_apply
: Search for this to see if the "Apply" phase is taking an unusually long time (e.g., thousands of seconds). Summary of Actions 1. If a tablet is in an Error State: You may need to repair it. Usually, restarting the affected BE can trigger a recovery, or you may need to use 
ADMIN SET REPLICA STATUS
to mark it as bad and let the system rebuild it from healthy replicas. 2. If versions are lagging: Check if the disk is full or if there are IO hangs on that specific BE. 3. If the PK index is too large: Ensure you have 
enable_persistent_index = true
. Without this, the BE might be swapping the PK index to disk constantly, causing the publish phase to time out. Important Note: If you find one BE is consistently the "suspected replica" for multiple tablets, that backend node likely has a hardware issue (slow disk) or a thread pool exhaustion problem with its thread pool configuration. References • best_practices/primarykey_table.md
m
@Rocky i am facing a weird situation. my replication number of tables is 2. but now i have an error state tablet with 3 replica one decommisioning and 2 normal statethat causes my table to be unreachable. what shall i do? | ReplicaId | BackendId | Version | VersionHash | LstSuccessVersion | LstSuccessVersionHash | LstFailedVersion | LstFailedVersionHash | LstFailedTime | SchemaHash | DataSize | RowCount | State | IsBad | IsSetBadForce | VersionCount | PathHash | MetaUrl | CompactionStatus | IsErrorState | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+------------+----------+--------------+-------+---------------+--------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+ | 33626330 | 21721646 | 362 | 0 | 362 | 0 | -1 | 0 | NULL | 1254240101 | 6573367643 | 34144641 | DECOMMISSION | true | true | 17 | -8136412698892935164 | http://starrocks-sharednothing-be-8.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/meta/header/28904557 | http://starrocks-sharednothing-be-8.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/compaction/show?tablet_id=28904557&amp;schema_hash=1254240101 | true | | 33648168 | 28957305 | 364 | 0 | 364 | 0 | -1 | 0 | NULL | 1254240101 | 6556917278 | 34078476 | NORMAL | false | false | 16 | 4556285071401253086 | http://starrocks-sharednothing-be-10.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/meta/header/28904557 | http://starrocks-sharednothing-be-10.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/compaction/show?tablet_id=28904557&amp;schema_hash=1254240101 | false | | 28904559 | 4002237 | 364 | 0 | 364 | 0 | -1 | 0 | NULL | 1254240101 | 7295366515 | 34078476 | NORMAL | false | false | 16 | -4513158338164820548 | http://starrocks-sharednothing-be-4.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/meta/header/28904557 | http://starrocks-sharednothing-be-4.starrocks-sharednothing-be-search.starrocks-sharednothing-arch.svc.cluster.local:8040/api/compaction/show?tablet_id=28904557&amp;schema_hash=1254240101 | false | +-----------+-----------+---------+-------------+-------------------+-----------------------+------------------+----------------------+---------------+------------+------------+----------+--------------+-------+---------------+--------------+----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+ 3 rows in set (0.00 sec)
r
This is a specific situation often encountered during node maintenance or rebalancing in StarRocks, especially with Primary Key (PK) tables. Why your table is unreachable In StarRocks, the "unreachable" state usually triggers when the FrontEnd (FE) determines that a tablet does not have enough healthy and consistent replicas. Even though you have 2 "NORMAL" replicas at version 364, the existence of the 3rd replica (v362) in 
IsErrorState = true
is likely blocking the FE's health check logic. For PK tables, 
IsErrorState
is a "hard" error—it means the Apply process (updating the PK index) failed on that backend. Because the system is in the middle of a decommissioning/migration process, the FE is "confused" because it's trying to maintain consistency across the replica set, but one of the members is physically unable to progress. How to resolve this 1. Prioritize the Repair Run the following command to tell the FE to prioritize the repair of this table. This often forces the FE to re-evaluate the tablet replicas and realize it can fulfill the replication requirement (2) using just the healthy ones.
Copy code
sql
ADMIN REPAIR TABLE your_table_name;
2. Check Decommissioning Status The replica on Backend 21721646 is stuck in 
DECOMMISSION
. Check if the backend is actually still "alive" or if the process is hung:
Copy code
sql
SHOW BACKENDS;
* If that Backend is 
Alive = false
, the FE should eventually drop the replica. * If it is 
Alive = true
but the 
SystemDecommissioned
status is 
false
, the decommissioning is stuck. 3. Force the FE to "Ignore" the bad replica Since the replica on BE 
21721646
is already marked as 
IsBad = true
and 
IsErrorState = true
, the system is already trying to ignore it, but the state is cached. To force a refresh, you can try to mark it as bad again (this sounds redundant, but it triggers an FE state transition):
Copy code
sql
ADMIN SET REPLICA STATUS TABLET 28904557 BACKEND 21721646 STATUS 'bad';
4. The "Nuclear" Option: Restart the Leader FE If the table remains unreachable and 
SHOW TABLET
still shows that same "ErrorState" replica blocking progress, restarting the Leader FE is the most effective fix. * When the FE restarts, it clears its in-memory metadata cache and rebuilds the tablet health map from scratch. * During the rebuild, it will see that you have 2 healthy replicas at version 364 and one broken replica at version 362. Since your 
replication_num
is 2, it will likely drop the broken one immediately and mark the tablet as healthy. 5. Check for "Stuck" Alter Jobs Sometimes a decommissioning is treated like a schema change/rollup. Check if there are any hung tasks:
Copy code
sql
SHOW ALTER TABLE COLUMN;
If you see a task related to this table that is stuck, it might be preventing the decommissioning replica from being deleted. Summary of Recommendation: 1. Run 
ADMIN REPAIR TABLE <table_name>;
. 2. If it doesn't recover in 5 minutes, restart the Leader FE. This is a safe operation and is the standard way to clear "zombie" tablet states that are stuck in the FE's scheduler. 3. Once the table is healthy, ensure the decommissioning of Backend 
21721646
actually completes. If you don't need that node, make sure it is fully removed from 
SHOW BACKENDS
. Note for PK Tables: 
IsErrorState
on a BE is usually caused by a Persistent Index corruption or an Out-of-Memory (OOM) event during the "Apply" phase. Since you are decommissioning that node anyway, you don't need to fix the index on that BE; you just need the FE to stop trying to sync to include it in the tablet's quorum. References • table_design/table_types/primary_key_table.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
m
@Rocky how can i use recover_tablet via http or anywhere else? because i know it exsits in code on script.cpp
r
The 
recover_tablet
functionality you are referring to is an internal Backend (BE) HTTP API designed to manually restore tablets that have been moved to the 
trash
directory (e.g., after a drop or an error) but haven't been physically deleted yet. While the primary way to handle tablet issues is via SQL (
ADMIN REPAIR TABLE
), this HTTP interface is used for "surgical" recovery when the FE has lost track of the replica but the data still exists on the disk of a specific BE. 1. How to use 
recover_tablet
via HTTP Based on the code structure in 
script_action.cpp
(or 
restore_tablet_action.cpp
depending on your version), you can trigger this on a specific Backend (BE) node. Endpoint Syntax:
Copy code
bash
curl -X POST "http://<BE_IP>:<BE_HTTP_PORT>/api/restore_tablet?tablet_id=<TABLET_ID>&schema_hash=<SCHEMA_HASH>"
* BE_IP: The IP of the backend where the tablet data resides. * BE_HTTP_PORT: Usually 
8040
. * TABLET_ID: The ID of the tablet you found in 
SHOW TABLET <id>
. * SCHEMA_HASH: This is required for the BE to locate the specific folder path. You can find this in the 
DetailCmd
output of 
SHOW TABLET
. 2. Usage via the "Script" Action If your version uses the generic 
script_action.cpp
dispatcher (which the user mentioned specifically), the syntax is often:
Copy code
bash
curl "http://<BE_IP>:<BE_HTTP_PORT>/api/script/recover_tablet?tablet_id=<ID>&schema_hash=<HASH>"
3. When to use this (And when not to) * Scenario: You accidentally dropped a table, and the 
RECOVER TABLE
SQL command failed or the 
catalog_trash_expire_second
has passed, but you can still see the files in the 
/storage/trash/
directory on the BE disks. * Scenario: A tablet is in 
IsErrorState
or 
IsBad
, and you want to try and force the BE to reload it from the disk instead of waiting for a clone from another node. Warning: * This command is local to one BE. If you have multiple replicas, you may need to run it on each BE where the data exists. * After running this, you usually need to alert the FrontEnd (FE) to the tablet's presence. You can do this by running 
SHOW TABLET <id>
or 
ADMIN REPAIR TABLE <table_name>
to trigger a report. * If the tablet metadata is severely corrupted or the version on disk is too old compared to the FE's current expected version, the FE will likely reject the recovered tablet and delete it again. How to find the Tablet ID and Schema Hash If you don't have the info, run this in your MySQL client:
Copy code
sql
SHOW TABLET <tablet_id>;
Look for the 
DetailCmd
column. It contains a path like: 
/dbs/<db_id>/<table_id>/partitions/<partition_id>/index/<index_id>/<tablet_id>
The 
schema_hash
is usually found in the tablet's metadata or logs. If you are exploring the BE disk, the directory structure is typically: 
storage/data/<shard_id>/<tablet_id>/<schema_hash>/
. Summary Use the 
POST /api/restore_tablet
(or 
/api/script/recover_tablet
if your version requires it) to pull tablets back from the dead. However, for your specific "unreachable" table issue from the previous message, restarting the Leader FE is still the safer and more effective first step, as it forces the FE to re-examine the tablet's health from all BEs. References • administration/http_interface.md
m
@Rocky tell me how can i see different endpoints
r
You can see and interact with the different StarRocks HTTP endpoints on both the FrontEnd (FE) and BackEnd (BE). Most administrative tasks use the FE port (default 8030) or the BE port (default 8040). 1. Most Useful BE Endpoints (Port 8040) The BE endpoints are used for tablet-level diagnostics and manual overrides (like the 
restore_tablet
you mentioned). * Tablet Metadata & Status: * `GET /api/compaction/show?tablet_id=<id>`: Shows the rowsets and versions for a specific tablet. * `GET /api/meta/header/<tablet_id>`: Dumps the raw metadata header for a tablet. * `GET /varz`: Shows all current BE configurations. * Manual Actions: * `POST /api/restore_tablet?tablet_id=<id>&schema_hash=<hash>`: The one you found in the code; used to recover from the trash. * `POST /api/compact?tablet_id=<id>&compaction_type=base`: Manually triggers compaction. * `POST /api/update_config?config_key=value`: Updates BE config without a restart. * Health & Performance: * `GET /api/health`: Basic health check. * `GET /metrics`: Prometheus-format metrics. * 
GET /pprof/heap
or `/pprof/profile`: Performance profiling (requires 
gperftools
installed). 2. Most Useful FE Endpoints (Port 8030) The FE endpoints are for cluster-wide management and metadata inspection. * Cluster Metadata: * `GET /api/show_proc`: Allows you to browse the same information as the 
SHOW PROC
SQL command via HTTP. * `GET /api/_get_ddl?db=<db>&tbl=<tbl>`: Returns the 
CREATE TABLE
statement. * `GET /api/show_meta_info`: Dumps general metadata status. * Query & Transaction Info: * `GET /api/query_detail`: Shows details of recently executed queries. * `GET /api/profile?query_id=<id>`: Gets the execution profile for a specific query. * Configuration: * `GET /api/_set_config?config_key=value`: Dynamically updates FE configurations. 3. How to "Discover" them If you are looking for a directory: 1. FE Web UI: Navigate to 
http://<FE_IP>:8030
. The UI provides links to "System Info", "Logs", and "Queries" which call these APIs behind the scenes. 2. Internal "Script" Endpoints: Regarding your mention of 
script.cpp
, this is often part of an internal dispatcher. In recent versions, these are "hidden" and not listed in a directory. They are usually called as: 
http://<BE_IP>:8040/api/script/<command>?params...
The commands available in 
script_action.cpp
usually include things like 
recover_tablet
or internal diagnostic tools. There is no "help" endpoint for these; you generally have to refer to the source code for the exact parameter strings. 4. Authentication Most of these endpoints require HTTP Basic Auth. Use your StarRocks user (e.g., 
admin
) and password:
Copy code
bash
curl -u admin:password "http://<FE_IP>:8030/api/show_proc"
Warning: Be very careful with 
POST
actions on the BE (like 
restore_tablet
or 
update_config
). Doing these manually can bypass the FE's safety checks and lead to metadata inconsistency if the FE isn't notified of the changes. References • administration/http_interface.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/Monitor_and_Manage.md • administration/Query_Detail_API.md References • administration/http_interface.md
m
hey @Rocky, i have faced a critical problem in my cluster: Subject: Persistent Index Inconsistency (Duplicate Key Error) on PK Table - v3.5.16 Cluster Environment: Version: StarRocks v3.5.16 (Shared-nothing) Topology: 12 BEs, 3 FEs Table Type: Primary Key Table (40B+ records) Configuration: enable_persistent_index = true, replication_num = 2, write_quorum = ALL Problem Description: An INSERT INTO transaction became stuck in the COMMITTED state for several hours. Investigation revealed that a tablet replica entered an error state due to a primary index rebuild failure. The specific error in the BE logs is: Already exist: _apply_rowset_commit error: load primary index failed: Already exist: FixedMutableIndex<16> insert found duplicate key, new(rssid=5 rowid=...), old(rssid=3 rowid=...). persistent_index.cpp:5333 Analysis: The tablet state shows that rowset 3 (referenced as old in the error) is no longer in the active rowset list (likely compacted into rowset 2), yet the index rebuild process still encounters it. This suggests a stale L0 snapshot or a partial apply (crash/IO error) left the Delete Vectors inconsistent. Because replication_num = 2 and both replicas are inconsistent, automatic repair via cloning fails with make snapshot failed. Proposed Solutions for Verification: We are considering the following manual interventions and would like your confirmation on the safest approach for v3.5.16: Manual Index Recovery via Wren Script: Can we safely trigger the recover_tablet(tablet_id) function defined in be/src/script/script.cpp? If so, what is the recommended way to invoke this in v3.5.16 (e.g., via a specific HTTP endpoint or internal tool)? Manual Deletion of Index Files: If the index rebuild from the existing metadata is failing due to a corrupted L0 snapshot, is it safe to manually delete the index.l0.* and index.l1.* files for the specific tablet on the BE disk? Our understanding is that this would force a full rebuild from rowsets using upsert semantics (via LocalPrimaryKeyRecover), which should resolve the duplicate key conflict. Note: i can not set replication_number = 3 because of lack of storage
r
This is a known issue where the Persistent Index (PI) becomes inconsistent with the actual rowsets on disk (often due to an interrupted compaction or a partial apply). Since both of your replicas are inconsistent and 
replication_num = 2
, the FE cannot perform a "self-healing" clone because it lacks a healthy source. In v3.5.16 (Shared-nothing), the most reliable and safest way to resolve this is your Solution 2: Manual Deletion of Index Files. Recommended Action: Manual Index Rebuild Deleting the persistent index files forces the BE to trigger a 
LocalPrimaryKeyRecover
. This process scans all valid segments (rowsets) for that tablet and reconstructs the index and delete vectors from scratch. Step-by-Step Procedure: 1. Identify the Tablet Path: Run 
SHOW TABLET <tablet_id>
to get the 
DetailCmd
. Find the local path on the BE. It will look like: 
<storage_root_path>/data/<shard_id>/<tablet_id>/<schema_hash>/
2. Stop the Backend (BE): While it is technically possible to delete files while the BE is running, stopping the BE is highly recommended to avoid race conditions during the rebuild. 3. Back up and Delete Index Files: Navigate to the tablet directory and move the index files to a backup location (or delete them):
Copy code
bash
    # Locate files matching index.l0._ and index.l1._
    mv index.l0.* /tmp/backup_index/
    mv index.l1.* /tmp/backup_index/
4. Start the Backend: Restart the BE. When the BE starts and the storage engine loads this tablet, it will notice the index files are missing. 5. Monitor the Rebuild: Check the 
<http://be.INFO|be.INFO>
log. You should see entries related to 
PrimaryKeyRecover
or 
load primary index
. Note: With 40B+ records, this rebuild will be IO-intensive and may take significant time depending on the tablet size. 6. Verify Tablet Health: Once the log shows the index is loaded/rebuilt, check the tablet status in StarRocks:
Copy code
sql
    ADMIN REPAIR TABLE your_table_name;
    SHOW TABLET <tablet_id>;
────────── Regarding your other proposed solution: 1. Manual Index Recovery via Wren Script (
recover_tablet
) In v3.5.x, the 
recover_tablet
function in 
script.cpp
is primarily an internal helper used for moving tablets out of the 
trash
directory. It does not specifically perform a deep consistency check or a "repair" of the persistent index beyond what a standard load does. * How to invoke it: If you still wish to try, it is usually exposed via: 
curl -X POST "http://<BE_IP>:8040/api/script/recover_tablet?tablet_id=<ID>"
* Why it might fail: If the index files themselves contain the duplicate key error (as your logs show), simply "recovering" the tablet metadata won't fix the corrupted index structure. Manual deletion (Solution 2) is the standard "hard reset" for the index. Summary For a 40B record table where both replicas are stuck, manually deleting the 
index.l0._
and 
index.l1._
files is the most direct way to force StarRocks to re-scan the actual data and resolve the "Already exist" conflict. Do this on one BE first. Once that replica is healthy, the second replica can either be repaired the same way or cloned from the first one. Important Warning: Ensure you have enough disk IOPS and CPU headroom, as rebuilding the index for very large tablets can impact performance during the recovery phase.
Open in Slack
PreviousNext
Powered by