Harit Rajkotiya
06/16/2026, 11:39 AM{
"Msg": "",
"IndexId": "18434",
"Warehouse": "default_warehouse",
"OriginIndexId": "18434",
"SchemaVersion": "1:0",
"Timeout": "86400",
"CreateTime": "2026-06-12 09:28:07",
"TransactionId": "31495534",
"IndexName": "WORKFLOW_EXECUTION",
"TableName": "WORKFLOW_EXECUTION",
"FinishTime": "2026-06-12 09:28:53",
"State": "FINISHED_REWRITING",
"JobId": "1061023"
}
one transaction is stuck in checked logs for this transaction id didnt found anything what can we check further routine load is not able to ingestRocky
06/16/2026, 11:40 AMFINISHED_REWRITING and the symptoms you described, your table is currently stuck in a Schema Change (ALTER TABLE) operation. In StarRocks, FINISHED_REWRITING means the data has been rewritten to the new schema format, but the job is waiting to "publish" the new version and swap the metadata.
Because a schema change holds a metadata lock, it will block Routine Load and other ingestion tasks from committing.
1. Check for Metadata Locks
The most likely reason the transaction is "stuck" is that it's waiting for a lock to finalize the schema change, or it's holding a lock that blocks others. Run this to see active locks:
sql
SELECT * FROM sys.fe_locks WHERE hold_time_ms > 10000;
Note: This requires OPERATE privileges and is available in newer versions (3.x+).
2. Verify Schema Change Status
Check the detailed status of the alter job to see if there is an error message (Msg field) that wasn't in your snippet:
sql
SHOW ALTER TABLE COLUMN WHERE TableName = 'WORKFLOW_EXECUTION' \G
3. Check for Version Gaps (Shared-Data Mode)
If you are using Shared-data mode (StarRocks 3.x Lake tables), a known issue exists where a schema change can hang in FINISHED_REWRITING if there is a "version discontinuity" (a missing version number in the sequence).
Check your FE logs (fe.log) for errors like:
* partition.getVisibleVersion() + 1 != version.get(0)
* readyToPublishVersion() requires commitVersion
4. How to resolve
If the job is permanently stuck and blocking your Routine Load:
* Attempt to Cancel: Try canceling the job using the JobId from your snippet.
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_EXECUTION;
_Note: If the state is FINISHED_REWRITING, the system might say "Job cannot be cancelled."_
* Restart FE (Leader): If a metadata lock is orphaned, restarting the Leader FE node can sometimes force a cleanup of the lock manager state.
* Clean up Transactions: Check if there are other extremely old running transactions that might be preventing this one from finishing:
sql
SHOW PROC '/transactions/<db_id>'; -- Replace <db_id> with your DB's ID
What is your StarRocks version and cluster type (Shared-data or Shared-nothing)? Knowing this will help determine if you've hit a specific version bug.
References
• sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.mdHarit Rajkotiya
06/16/2026, 11:45 AMRocky
06/16/2026, 11:46 AMFINISHED_REWRITING state and blocking Routine Load, it means the data transformation is complete on the Backends (BEs), but the Frontend (FE) cannot finalize the metadata (the "Publish" phase).
This usually happens because a transaction is preventing the table version from advancing. Here is how to troubleshoot and resolve this:
1. Identify the Stuck Transaction
The TransactionId: 31495534 in your snippet is the transaction for the schema change itself. If it is stuck, there is likely an older transaction (with a smaller ID) that hasn't finished, blocking this one from becoming visible.
Run these commands to find the blocker:
1. Get your Database ID:
sql
SHOW PROC '/transactions';
2. Check for running transactions in that database (replace <db_id>):
sql
SHOW PROC '/transactions/<db_id>/running';
* What to look for: Look for any transactions with a smaller ID than 31495534. If an older transaction is stuck in PREPARE or COMMITTED but not VISIBLE, it will block your schema change from finishing.
2. Check for Version Gaps (Shared-data/Lake Tables)
If you are using StarRocks 3.x with Shared-data mode, there is a known issue where a "version discontinuity" can happen.
Check your FE leader's fe.log for this specific error:
partition.getVisibleVersion() + 1 != version.get(0)
If you see this, the version sequence is broken (e.g., the table is at version 100, and the system is trying to publish version 102, but version 101 is missing). This requires a manual metadata fix or a specific patch depending on your version.
3. Check Routine Load Tasks
Since Routine Load is failing, it might be the victim (blocked by the schema change lock) or the cause (an old task holding a version open).
* Check the status of current tasks:
sql
SHOW ROUTINE LOAD TASK WHERE JobName = 'your_job_name';
* If you see tasks in COMMITTED state for a long time, they are waiting for the schema change to finish.
4. Recommended Actions
1. Kill long-running queries/transactions: If you find an old transaction in SHOW PROC '/transactions/<db_id>/running', try to identify its source (e.g., a hung Stream Load or Insert) and stop it.
2. Restart the FE Leader: If the FE is in a deadlock state or an internal metadata lock is orphaned, restarting the Leader FE node often triggers a cleanup and allows the FINISHED_REWRITING job to finally move to FINISHED.
3. Check for Disk/IO issues: Ensure your BEs are not under extreme disk pressure, as the final "move" of data files can hang if the filesystem is unresponsive.
Can you confirm:
1. Are you using Shared-data (Cloud-native) or Shared-nothing?
2. Did you find any transactions with an ID smaller than 31495534 in the running list`running` list?
` list?
References
• sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md
• sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md
• sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.mdNirav Dobariya
09/13/2026, 7:20 AMfe.log for this specific error:
partition.getVisibleVersion() + 1 != version.get(0)
my current version is 3.3.17 how can i fix?Rocky
09/13/2026, 7:20 AMpartition.getVisibleVersion() + 1 != version.get(0) indicates a version discontinuity.
This happens when a partition's VisibleVersion is (for example) 100, but the next transaction arriving is version 102. The system expects version 101 to exist. Because the schema change is stuck in FINISHED_REWRITING, it is likely the "missing" transaction that is holding up the entire pipeline.
Step 1: Confirm the Version Gap
Run the following to see the current state of the table's partitions:
sql
SHOW PARTITIONS FROM WORKFLOW_EXECUTION;
Look for:
* VisibleVersion: The current version.
* NextVersion: The version the system is trying to publish.
* Compare these values with the IDs in your fe.log error message (... + 1 != version.get(0) [VisibleVersion] [NextVersion]).
Step 2: Use the Automated Repair Tool
For Shared-data (Lake) tables, StarRocks provides an ADMIN REPAIR command designed to handle metadata inconsistencies by rolling back to the last consistent state.
sql
ADMIN REPAIR TABLE WORKFLOW_EXECUTION
PROPERTIES ("enforce_consistent_version" = "true");
* What this does: It attempts to find the last valid, continuous version and aligns all tablets to it.
* Note: This may cause a small amount of data loss (the data in the "skipped" or "incomplete" versions), but it will unblock the table and the schema change.
Step 3: Manual Version Adjustment (High Risk)
If ADMIN REPAIR does not solve it, you can manually force the partition version to match what the system expects. Use this with caution as it manually overrides metadata.
sql
-- Replace <version_number> with the VisibleVersion mentioned in the error log
ADMIN SET TABLE WORKFLOW_EXECUTION PARTITION(WORKFLOW_EXECUTION) VERSION TO <version_number>;
Step 4: Restart the FE Leader
The FINISHED_REWRITING state is often cached in the Frontend's memory. If the metadata is repaired but the job remains stuck:
1. Identify the FE Leader node.
2. Restart the FE service. This forces the Leader to reload the metadata and the schema change job state from the image and edit logs, which often triggers the completion of a "ready" job.
Why did this happen?
In StarRocks 3.3.x, this race condition is usually caused by:
* Concurrent heavy writes: A Routine Load and a Schema Change competing for the same version sequence.
* A failed "Publish" phase: A previous transaction crashed during the publish step, leaving a "hole" in the version history.
Recommendation: After resolving this, ensure your Routine Load is paused during major schema changes if you notice this happening frequently, or consider upgrading to a later 3.3.x patch (like 3.3.22+) which includes further fixes for Lake table publish stability.
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md
• faq/shared_data_faq.md
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• administration/management/resource_management/Replica.md
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md
• administration/management/compaction.mdNirav Dobariya
09/13/2026, 7:22 AMNirav Dobariya
09/13/2026, 7:26 AMRocky
09/13/2026, 7:27 AMTable does not exist or is not OLAP table usually occurs if the table name is slightly different, the database context is incorrect, or if the system doesn't recognize the "LAKE" engine (used in shared-data mode) for that specific command in your version.
Since you are on 3.3.17 (Shared-data mode) and hitting a version discontinuity, here is the refined plan to fix this:
1. Verify Table and Partition Names
First, confirm the exact table name and partition name you are targeting. Your first message mentioned WORKFLOW_EXECUTION but your command used WORKFLOW_NODE_EXECUTION.
sql
-- Ensure you are in the correct database
USE your_db_name;
-- Confirm the table exists and check its engine type
SHOW CREATE TABLE WORKFLOW_NODE_EXECUTION;
-- Confirm the partition name exists
SHOW PARTITIONS FROM WORKFLOW_NODE_EXECUTION;
Note: Shared-data tables use ENGINE=LAKE. If the table is an external table (like Iceberg or Hive), ADMIN REPAIR will not work.
2. Alternative: Use ADMIN SET PARTITION VERSION
If ADMIN REPAIR continues to fail with the "not OLAP" error, you can manually "jump" the version gap using the ADMIN SET PARTITION VERSION command. This is often more effective for fixing the specific getVisibleVersion() + 1 != version.get(0) error.
How to find the correct version:
In your fe.log, look at the error message: partition.getVisibleVersion() + 1 != version.get(0) [A] [B] [C].
* [B] is the current VisibleVersion.
* [C] is the NextVersion the system is trying to jump to.
* The "gap" is the version between them.
To bridge the gap:
sql
-- Manually set the partition to the version it is trying to reach
-- Replace <partition_name> and <version_C> with values from your logs
ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (<partition_name>) VERSION TO <version_C>;
3. Clear the Schema Change Lock
If the Schema Change is still stuck in FINISHED_REWRITING after fixing the version gap:
1. Cancel the job:
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
2. Restart the FE Leader: In shared-data mode, the state of a "Finished Rewriting" job is held in the FE's memory. Restarting the FE Leader node forces it to re-evaluate the table's state from the metadata, which often transitions the job to FINISHED immediately.
Why the "not OLAP table" error?
In some older 3.x patch versions, the ADMIN REPAIR command had a check that only looked for ENGINE=OLAP. Since Cloud-native tables are ENGINE=LAKE, the command would mistakenly reject them. If you cannot get ADMIN REPAIR to recognize the table, the ADMIN SET PARTITION VERSION (Step 2) is the standard workaround.
Summary Recommendation:
If you can identify the version gap from your logs, use Step 2 to set the version manually. If the schema change is still stuck after that, restart your FE Leader node.
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• administration/management/resource_management/Replica.md
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.mdNirav Dobariya
09/13/2026, 7:30 AMRocky
09/13/2026, 7:31 AMADMIN commands mistakenly validate for the ENGINE=OLAP property and do not yet fully recognize ENGINE=LAKE (Shared-data) tables, even though the underlying repair logic is needed.
Since you have a version discontinuity (VisibleVersion + 1 != NextVersion) and your schema change is stuck in FINISHED_REWRITING, you are in a "version gap deadlock." Here is the path to resolve it:
1. Restart the FE Leader (Recommended First Step)
In Shared-data mode, the state of a job in FINISHED_REWRITING is primarily managed in the Frontend (FE) Leader's memory.
* Action: Restart the FE Leader node.
* Result: When the FE Leader comes back up, it reloads the metadata and re-evaluates the schema change job. In many cases, it will see that the rewriting is done and successfully transition the job to FINISHED, bridging the version gap automatically.
2. Try the "Partition ID" Workaround
If the restart doesn't work and you still need to use ADMIN SET PARTITION VERSION, try using the Partition ID instead of the name. Sometimes this bypasses the specific validation check that triggers the "not a OLAP table" error.
1. Find the `PartitionId`:
sql
SHOW PARTITIONS FROM WORKFLOW_NODE_EXECUTION;
2. Run the command using the ID (no quotes):
sql
-- Replace <partition_id> with the number from the previous step
ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (<partition_id>) VERSION TO 7720;
3. Clear the Batch Publish Cache
If you see the version gap error repeating in logs, toggle the batch publish setting. This can force the FE to stop trying to "batch" the missing version and move on:
sql
-- Disable batch publish temporarily
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
-- Wait 1-2 minutes to see if the job completes, then turn it back on
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Manual Job Cancellation
If the job is still stuck in FINISHED_REWRITING and blocking everything:
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
* Note: If the system says "Job cannot be cancelled in this state," a FE Leader restart is the only way to clear the memory lock.
Summary of why this happened
In StarRocks 3.3.x shared-data mode, a race condition can occur where a Routine Load (or other write) commits a version (e.g., 7721) while the Schema Change is trying to claim the previous version (7720). Because Lake tables require strict version continuity, the "hole" at 7720 stops the pipeline.
Recommendation: After fixing this, consider upgrading to a later 3.3.x patch (like 3.3.20+) where several "gtid mismatch" and "version discontinuity" deadlocks in Shared-data mode have been resolved.
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
• administration/management/FE_configuration.md
• release_notes/release-3.3.md
References
• faq/operation_maintenance_faq.md
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
• faq/Sql_faq.mdNirav Dobariya
09/13/2026, 7:35 AMRocky
09/13/2026, 7:35 AMADMIN SET PARTITION VERSION is strictly limited to shared-nothing (Local Storage/OLAP) tables. In StarRocks 3.3.x (Shared-data mode), tables use the LAKE engine, and metadata is managed differently.
Because you are seeing partition.getVisibleVersion() + 1 != version.get(0), your table's metadata has a "hole" (e.g., it has version 7719 and is trying to jump to 7721, but 7720 is missing).
Here are the specific steps to fix this for a Lake table:
1. The Correct Repair Command for Lake Tables
In Shared-data mode, you must use ADMIN REPAIR with specific properties. If the basic ADMIN REPAIR failed with a "not OLAP" error, it is likely a validation bug in v3.3.17. Try running it without the partition name first to see if it targets the whole table:
sql
ADMIN REPAIR TABLE WORKFLOW_NODE_EXECUTION
PROPERTIES ("enforce_consistent_version" = "true");
* Why this works: For Lake tables, this command forces the FE to look at the underlying object storage (S3/OSS), find the last valid continuous version, and roll the metadata back to that state.
* Note: This may result in the loss of the data that was in the "skipped" version, but it will unblock the table.
2. Restart the FE Leader (Highly Effective)
In StarRocks 3.3.x, the state of a schema change job in FINISHED_REWRITING and the version sequence are often cached in the FE Leader's memory.
1. Identify the FE Leader node.
2. Restart the FE service.
3. Upon restart, the FE will reload the image and edit logs. It will re-verify the table's state against the object storage. In many cases, this forces the FE to "discover" the missing version or skip the gap, transitioning the job to FINISHED.
3. Clear the Transaction Lock
If the version gap is caused by a stuck transaction:
1. Check for any "Running" or "Prepare" transactions on that table:
sql
SHOW PROC '/transactions';
2. If you see a transaction related to WORKFLOW_NODE_EXECUTION that is very old, it might be the "missing" version. You may need to wait for it to time out or contact support if it's stuck in a PREPARE state indefinitely.
4. Toggle Batch Publish (Workaround)
Sometimes the FE gets stuck trying to batch a version that doesn't exist. Disabling batching can force it to process versions one-by-one and potentially clear the error:
sql
-- Disable batch publish
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
-- Wait a few minutes to see if the table unblocks, then re-enable
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
Summary Recommendation
Since ADMIN SET PARTITION VERSION will never work for Lake tables:
1. Restart the FE Leader. This is the safest and most common fix for the FINISHED_REWRITING lock in 3.3.17.
2. If the restart fails, use Step 1 (ADMIN REPAIR on the whole table) to force a metadata rollback.
3. Upgrade: This specific "version gap" race condition during schema changes was a known issue in early 3.3.x releases and has been significantly improved in 3.3.20+. Consider a patch upgrade to prevent recurrence.
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• faq/shared_data_faq.md
• administration/management/FE_configuration.md
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• faq/Sql_faq.md
• knowledge/admin_and_operation/shared_data_compaction.mdNirav Dobariya
09/13/2026, 7:36 AMRocky
09/13/2026, 7:36 AMPROPERTIES Syntax Error: The PROPERTIES clause for ADMIN REPAIR TABLE was introduced in v4.1. In v3.3, this command only accepts the table/partition name and does not support the consistency properties you tried.
2. "Not an OLAP table" Error: In v3.3, the ADMIN SET PARTITION VERSION command is strictly validated for ENGINE=OLAP (Shared-nothing) tables and will reject ENGINE=LAKE (Shared-data) tables.
To resolve the version gap (VisibleVersion + 1 != NextVersion) and unblock your schema change in v3.3.17, follow these steps:
1. Identify the "Missing" Version
Run the following to see the status of the transaction that corresponds to the "hole" (e.g., if Visible is 7719 and Next is 7721, look for the transaction that would have produced 7720):
sql
-- Check for transactions stuck in PREPARE or COMMITTED state
SHOW PROC '/transactions';
-- Check for stuck compaction tasks that might be consuming a version
SHOW PROC '/compactions';
If you find a transaction in a COMMITTED but not VISIBLE state, it is likely the one blocking the sequence.
2. Restart the FE Leader (Highly Recommended)
In Shared-data mode, the FINISHED_REWRITING state and the version publish queue are managed in the FE Leader's memory.
* Action: Restart the FE Leader node.
* Effect: Upon restart, the FE reloads the metadata and re-scans the transaction states. This often triggers a re-publish that bridges the version gap or clears the "Finished Rewriting" lock, allowing the job to transition to FINISHED.
3. Disable Batch Publishing (Workaround)
If a specific transaction is failing to publish because it's part of a "batch" that has a hole, you can force the FE to publish versions one-by-one:
sql
-- Disable batching to force individual version publishing
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
-- Wait a few minutes to see if the table becomes visible/job finishes.
-- Afterward, re-enable it for performance:
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Manual Job Cancellation
If the FE restart does not clear the FINISHED_REWRITING state, try to cancel the job manually. If it was truly finished but just failed to publish, a cancel might force a cleanup:
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
Summary for v3.3.17
Since the ADMIN repair/set commands are not yet fully compatible with Lake tables in your version, a FE Leader restart is the standard operational fix for this specific metadata deadlock. If the issue persists, it may require a patch upgrade to 3.3.20+, which includes several fixes for Lake table version continuity during schema changes.
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md
• administration/management/FE_configuration.md
• faq/shared_data_faq.md
• sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md
References
• faq/shared_data_faq.md
• knowledge/admin_and_operation/shared_data_compaction.mdNirav Dobariya
09/13/2026, 7:55 AMRocky
09/13/2026, 7:56 AMADMIN commands (like ADMIN SET PARTITION VERSION) mistakenly validate for the ENGINE=OLAP property and do not yet recognize ENGINE=LAKE (Shared-data) tables, even though the underlying logic is needed.
Additionally, your partition info shows a version discontinuity:
* VisibleVersion: 7718
* NextVersion: 7730
* Gap: There is a hole from version 7719 to 7729.
In Shared-data mode, a Schema Change job stuck in FINISHED_REWRITING is usually waiting for its specific version (likely 7720) to become "Visible." Because the sequence is broken (the system jumped to 7730), the job is in a "version gap deadlock."
To resolve this in v3.3.17, follow these steps:
1. Restart the FE Leader (Highly Recommended)
In Shared-data mode, the state of the schema change job and the publish queue are managed in the FE Leader's memory.
* Action: Identify and restart the FE Leader node.
* Result: When the FE Leader comes back up, it reloads the metadata and re-evaluates the schema change job. In many cases, it will detect that the rewriting is done and transition the job to FINISHED (bridging the version gap automatically) or at least allow you to cancel it.
2. Check for Stuck Transactions
The "hole" at version 7719 might be caused by a transaction that is COMMITTED but not yet VISIBLE.
sql
SHOW PROC '/transactions';
Look for any transaction with a TXN_ID that corresponds to the timeframe when the gap occurred. If it is stuck in COMMITTED or PREPARE, it is blocking the version sequence.
3. Clear the Batch Publish Cache
Sometimes the FE gets stuck trying to "batch" the missing version. Toggling this setting can force it to process individual versions and potentially clear the deadlock:
sql
-- Disable batch publish temporarily
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
-- Wait 2-3 minutes to see if the schema change job finishes.
-- If it finishes, turn it back on:
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Try the "Partition ID" Workaround
If you still need to force the version and the previous command failed, try using the Partition ID without quotes. Sometimes this bypasses the specific "OLAP" check:
sql
-- Use the Partition ID 90889287 directly (no quotes)
ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (90889287) VERSION TO 7720;
(Note: If this still gives the "not OLAP" error, the command is simply not supported for Lake tables in your current patch.)
Summary & Prevention
This "version discontinuity" during schema changes is a known race condition in early 3.3.x releases when concurrent writes occur.
* Long-term fix: Upgrade to 3.3.20 or later. Several bug fixes for LakeTableSchemaChangeJob version continuity and "GTID mismatch" were specifically added to address this exact deadlock in Shared-data mode.
References
• faq/shared_data_faq.md
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md
• administration/management/FE_configuration.md
• release_notes/release-3.3.md
References
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md
• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.mdRocky
09/13/2026, 7:56 AM