@Rocky Broker load job is struck for more than a day meaning SHOW load; shows prepared state but nothing is happening so help me to cancel this ? PREPARED ETL:100%; LOAD:99% BROKER NORMAL 462224198 I tried to restart starRocks pods but still it didn't go away so what are the options to clean this up?

In StarRocks, when a Broker Load job shows a state of

PREPARED

and progress of

99%

, it means the data has been successfully sent to the Backends (BEs) and the transaction has been committed in the Frontends (FEs) metadata. The job is currently stuck in the Publish Version phase. During this phase, the FE notifies all BEs holding replicas of the data to make the new rowsets visible. A job remains in

PREPARED

until a quorum of replicas for every tablet involved has successfully acknowledged the new version. Why you cannot cancel it The

CANCEL LOAD

command is only valid for jobs in the

PENDING

or

LOADING

states. Once a job reaches

PREPARED

, the system considers the data "committed." Because StarRocks ensures atomicity, it will keep retrying the publish task until it succeeds. It cannot be cancelled because the transaction is already logically complete. Why it is stuck for more than a day If a job is stuck here for a day, one of the following is likely occurring: 1. Unhealthy Backends: One or more BEs containing replicas for this load are down or disconnected. Check the status with:

sql
    SHOW BACKENDS;

If any BE shows

Alive: false

, the publish task may be unable to reach quorum. 2. Version Count Limit: If the destination table has reached the maximum allowed versions (e.g., due to frequent small loads and slow compaction), the BEs may refuse to publish new versions. Look for "too many versions" in your

<http://be.INFO|be.INFO>

logs. 3. Publish Task Backlog: The FE's publish queue might be blocked by a massive number of tablets or a previous failed task. 4. Metadata Inconsistency: In some older versions (e.g., v3.3.x and earlier), certain edge cases could cause publish tasks to loop infinitely. Recommended Actions to Clean Up Since you have already restarted the pods and the state persisted (as it is stored in the FE's BDBJE metadata), follow these steps: 1. Check FE Logs: Look in

fe.log

or

fe.warn.log

for your Job ID (

462224198

). Search for "publish" or "transaction" to see the specific error preventing the version from becoming visible. 2. Inspect Transactions: Use the following command to see if the underlying transaction is still active:

sql
    -- Replace <db_name> with your database name
    SHOW TRANSACTION FROM <db_name> WHERE id = 462224198;

_(Note: Sometimes the Job ID and Txn ID differ; check the

JobDetails

column in

SHOW LOAD

to find the correct

txn_id

.)_ 3. Force a Leader Handover: If it's a metadata "ghost" job caused by a hang in the FE Leader's memory, sometimes restarting the current Leader FE specifically to force a new leader to take over can clear the task queue. 4. Fix Node Health: Ensure all BEs are

Alive

and have sufficient disk space. If a BE is permanently gone, you may need to decommission it so the FE stops waiting for its replicas. If the table is no longer needed or can be recreated, dropping the table will usually remove all associated load metadata, but this should be a last resort. For further help, check the

fe.log

specifically for "failed to publish version" to identify which tablets are causing the hold up the hang. References • loading/SQL_transaction.md • administration/management/compaction.md • knowledge/trouble_shooting/fe_deadlock_problems.md