<@U0A71G31CDV> ``` { "Msg": "", "...
# questions-and-troubleshooting
h
@Rocky
Copy code
{
  "Msg": "",
  "IndexId": "18434",
  "Warehouse": "default_warehouse",
  "OriginIndexId": "18434",
  "SchemaVersion": "1:0",
  "Timeout": "86400",
  "CreateTime": "2026-06-12 09:28:07",
  "TransactionId": "31495534",
  "IndexName": "WORKFLOW_EXECUTION",
  "TableName": "WORKFLOW_EXECUTION",
  "FinishTime": "2026-06-12 09:28:53",
  "State": "FINISHED_REWRITING",
  "JobId": "1061023"
}
one transaction is stuck in checked logs for this transaction id didnt found anything what can we check further routine load is not able to ingest
r
Based on the state
FINISHED_REWRITING
and the symptoms you described, your table is currently stuck in a Schema Change (ALTER TABLE) operation. In StarRocks,
FINISHED_REWRITING
means the data has been rewritten to the new schema format, but the job is waiting to "publish" the new version and swap the metadata. Because a schema change holds a metadata lock, it will block Routine Load and other ingestion tasks from committing. 1. Check for Metadata Locks The most likely reason the transaction is "stuck" is that it's waiting for a lock to finalize the schema change, or it's holding a lock that blocks others. Run this to see active locks:
Copy code
sql
SELECT * FROM sys.fe_locks WHERE hold_time_ms > 10000;
Note: This requires
OPERATE
privileges and is available in newer versions (3.x+).
2. Verify Schema Change Status Check the detailed status of the alter job to see if there is an error message (
Msg
field) that wasn't in your snippet:
Copy code
sql
SHOW ALTER TABLE COLUMN WHERE TableName = 'WORKFLOW_EXECUTION' \G
3. Check for Version Gaps (Shared-Data Mode) If you are using Shared-data mode (StarRocks 3.x Lake tables), a known issue exists where a schema change can hang in
FINISHED_REWRITING
if there is a "version discontinuity" (a missing version number in the sequence). Check your FE logs (
fe.log
) for errors like: *
partition.getVisibleVersion() + 1 != version.get(0)
*
readyToPublishVersion() requires commitVersion
4. How to resolve If the job is permanently stuck and blocking your Routine Load: * Attempt to Cancel: Try canceling the job using the
JobId
from your snippet.
Copy code
sql
    CANCEL ALTER TABLE COLUMN FROM WORKFLOW_EXECUTION;
_Note: If the state is
FINISHED_REWRITING
, the system might say "Job cannot be cancelled."_ * Restart FE (Leader): If a metadata lock is orphaned, restarting the Leader FE node can sometimes force a cleanup of the lock manager state. * Clean up Transactions: Check if there are other extremely old running transactions that might be preventing this one from finishing:
Copy code
sql
    SHOW PROC '/transactions/<db_id>'; -- Replace <db_id> with your DB's ID
What is your StarRocks version and cluster type (Shared-data or Shared-nothing)? Knowing this will help determine if you've hit a specific version bug. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md
h
1. Check for Metadata Locks 0 locks 2. Verify Schema Change Status stuck on finished rewriting there is an transaction which is stuck btw @Rocky
r
If your StarRocks schema change is stuck in the
FINISHED_REWRITING
state and blocking Routine Load, it means the data transformation is complete on the Backends (BEs), but the Frontend (FE) cannot finalize the metadata (the "Publish" phase). This usually happens because a transaction is preventing the table version from advancing. Here is how to troubleshoot and resolve this: 1. Identify the Stuck Transaction The
TransactionId: 31495534
in your snippet is the transaction for the schema change itself. If it is stuck, there is likely an older transaction (with a smaller ID) that hasn't finished, blocking this one from becoming visible. Run these commands to find the blocker: 1. Get your Database ID:
Copy code
sql
   SHOW PROC '/transactions';
2. Check for running transactions in that database (replace
<db_id>
):
Copy code
sql
   SHOW PROC '/transactions/<db_id>/running';
* What to look for: Look for any transactions with a smaller ID than
31495534
. If an older transaction is stuck in
PREPARE
or
COMMITTED
but not
VISIBLE
, it will block your schema change from finishing. 2. Check for Version Gaps (Shared-data/Lake Tables) If you are using StarRocks 3.x with Shared-data mode, there is a known issue where a "version discontinuity" can happen. Check your FE leader's
fe.log
for this specific error:
partition.getVisibleVersion() + 1 != version.get(0)
If you see this, the version sequence is broken (e.g., the table is at version 100, and the system is trying to publish version 102, but version 101 is missing). This requires a manual metadata fix or a specific patch depending on your version. 3. Check Routine Load Tasks Since Routine Load is failing, it might be the victim (blocked by the schema change lock) or the cause (an old task holding a version open). * Check the status of current tasks:
Copy code
sql
    SHOW ROUTINE LOAD TASK WHERE JobName = 'your_job_name';
* If you see tasks in
COMMITTED
state for a long time, they are waiting for the schema change to finish. 4. Recommended Actions 1. Kill long-running queries/transactions: If you find an old transaction in
SHOW PROC '/transactions/<db_id>/running'
, try to identify its source (e.g., a hung Stream Load or Insert) and stop it. 2. Restart the FE Leader: If the FE is in a deadlock state or an internal metadata lock is orphaned, restarting the Leader FE node often triggers a cleanup and allows the
FINISHED_REWRITING
job to finally move to
FINISHED
. 3. Check for Disk/IO issues: Ensure your BEs are not under extreme disk pressure, as the final "move" of data files can hang if the filesystem is unresponsive. Can you confirm: 1. Are you using Shared-data (Cloud-native) or Shared-nothing? 2. Did you find any transactions with an ID smaller than 31495534 in the
running
list`running` list? ` list? References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD_TASK.md • sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md
n
@Rocky i am getting this Check for Version Gaps (Shared-data/Lake Tables) If you are using StarRocks 3.x with Shared-data mode, there is a known issue where a "version discontinuity" can happen. Check your FE leader's
fe.log
for this specific error:
partition.getVisibleVersion() + 1 != version.get(0)
my current version is 3.3.17 how can i fix?
r
In StarRocks 3.3.17 (Shared-data mode), the error
partition.getVisibleVersion() + 1 != version.get(0)
indicates a version discontinuity. This happens when a partition's
VisibleVersion
is (for example) 100, but the next transaction arriving is version 102. The system expects version 101 to exist. Because the schema change is stuck in
FINISHED_REWRITING
, it is likely the "missing" transaction that is holding up the entire pipeline. Step 1: Confirm the Version Gap Run the following to see the current state of the table's partitions:
Copy code
sql
SHOW PARTITIONS FROM WORKFLOW_EXECUTION;
Look for: * VisibleVersion: The current version. * NextVersion: The version the system is trying to publish. * Compare these values with the IDs in your
fe.log
error message (
... + 1 != version.get(0) [VisibleVersion] [NextVersion]
). Step 2: Use the Automated Repair Tool For Shared-data (Lake) tables, StarRocks provides an
ADMIN REPAIR
command designed to handle metadata inconsistencies by rolling back to the last consistent state.
Copy code
sql
ADMIN REPAIR TABLE WORKFLOW_EXECUTION 
PROPERTIES ("enforce_consistent_version" = "true");
* What this does: It attempts to find the last valid, continuous version and aligns all tablets to it. * Note: This may cause a small amount of data loss (the data in the "skipped" or "incomplete" versions), but it will unblock the table and the schema change. Step 3: Manual Version Adjustment (High Risk) If
ADMIN REPAIR
does not solve it, you can manually force the partition version to match what the system expects. Use this with caution as it manually overrides metadata.
Copy code
sql
-- Replace <version_number> with the VisibleVersion mentioned in the error log
ADMIN SET TABLE WORKFLOW_EXECUTION PARTITION(WORKFLOW_EXECUTION) VERSION TO <version_number>;
Step 4: Restart the FE Leader The
FINISHED_REWRITING
state is often cached in the Frontend's memory. If the metadata is repaired but the job remains stuck: 1. Identify the FE Leader node. 2. Restart the FE service. This forces the Leader to reload the metadata and the schema change job state from the image and edit logs, which often triggers the completion of a "ready" job. Why did this happen? In StarRocks 3.3.x, this race condition is usually caused by: * Concurrent heavy writes: A Routine Load and a Schema Change competing for the same version sequence. * A failed "Publish" phase: A previous transaction crashed during the publish step, leaving a "hole" in the version history. Recommendation: After resolving this, ensure your
Routine Load
is paused during major schema changes if you notice this happening frequently, or consider upgrading to a later 3.3.x patch (like 3.3.22+) which includes further fixes for Lake table publish stability. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • faq/shared_data_faq.md References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • administration/management/compaction.md
n
@Kavish Manubolu
@Rocky getting this mysql> ADMIN REPAIR TABLE WORKFLOW_NODE_EXECUTION PARTITION (p1789084800000_1786579221279); ERROR 1064 (HY000): Table does not exist or is not OLAP table: WORKFLOW_NODE_EXECUTION
r
The error
Table does not exist or is not OLAP table
usually occurs if the table name is slightly different, the database context is incorrect, or if the system doesn't recognize the "LAKE" engine (used in shared-data mode) for that specific command in your version. Since you are on 3.3.17 (Shared-data mode) and hitting a version discontinuity, here is the refined plan to fix this: 1. Verify Table and Partition Names First, confirm the exact table name and partition name you are targeting. Your first message mentioned
WORKFLOW_EXECUTION
but your command used
WORKFLOW_NODE_EXECUTION
.
Copy code
sql
-- Ensure you are in the correct database
USE your_db_name;

-- Confirm the table exists and check its engine type
SHOW CREATE TABLE WORKFLOW_NODE_EXECUTION;

-- Confirm the partition name exists
SHOW PARTITIONS FROM WORKFLOW_NODE_EXECUTION;
Note: Shared-data tables use
ENGINE=LAKE
. If the table is an external table (like Iceberg or Hive),
ADMIN REPAIR
will not work.
2. Alternative: Use
ADMIN SET PARTITION VERSION
If
ADMIN REPAIR
continues to fail with the "not OLAP" error, you can manually "jump" the version gap using the
ADMIN SET PARTITION VERSION
command. This is often more effective for fixing the specific
getVisibleVersion() + 1 != version.get(0)
error. How to find the correct version: In your
fe.log
, look at the error message:
partition.getVisibleVersion() + 1 != version.get(0) [A] [B] [C]
. * [B] is the current
VisibleVersion
. * [C] is the
NextVersion
the system is trying to jump to. * The "gap" is the version between them. To bridge the gap:
Copy code
sql
-- Manually set the partition to the version it is trying to reach
-- Replace <partition_name> and <version_C> with values from your logs
ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (<partition_name>) VERSION TO <version_C>;
3. Clear the Schema Change Lock If the Schema Change is still stuck in
FINISHED_REWRITING
after fixing the version gap: 1. Cancel the job:
Copy code
sql
    CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
2. Restart the FE Leader: In shared-data mode, the state of a "Finished Rewriting" job is held in the FE's memory. Restarting the FE Leader node forces it to re-evaluate the table's state from the metadata, which often transitions the job to
FINISHED
immediately. Why the "not OLAP table" error? In some older 3.x patch versions, the
ADMIN REPAIR
command had a check that only looked for
ENGINE=OLAP
. Since Cloud-native tables are
ENGINE=LAKE
, the command would mistakenly reject them. If you cannot get
ADMIN REPAIR
to recognize the table, the
ADMIN SET PARTITION VERSION
(Step 2) is the standard workaround. Summary Recommendation: If you can identify the version gap from your logs, use Step 2 to set the version manually. If the schema change is still stuck after that, restart your FE Leader node. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CANCEL_REPAIR.md
n
@Rocky mysql> ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (p1789084800000_1786579221279) VERSION TO 7720; ERROR 5503 (42000): Table 'WORKFLOW_NODE_EXECUTION' is not a OLAP table mysql>
r
The error "Table is not a OLAP table" occurs because, in your StarRocks version (3.3.17), some
ADMIN
commands mistakenly validate for the
ENGINE=OLAP
property and do not yet fully recognize
ENGINE=LAKE
(Shared-data) tables, even though the underlying repair logic is needed. Since you have a version discontinuity (
VisibleVersion + 1 != NextVersion
) and your schema change is stuck in
FINISHED_REWRITING
, you are in a "version gap deadlock." Here is the path to resolve it: 1. Restart the FE Leader (Recommended First Step) In Shared-data mode, the state of a job in
FINISHED_REWRITING
is primarily managed in the Frontend (FE) Leader's memory. * Action: Restart the FE Leader node. * Result: When the FE Leader comes back up, it reloads the metadata and re-evaluates the schema change job. In many cases, it will see that the rewriting is done and successfully transition the job to
FINISHED
, bridging the version gap automatically. 2. Try the "Partition ID" Workaround If the restart doesn't work and you still need to use
ADMIN SET PARTITION VERSION
, try using the Partition ID instead of the name. Sometimes this bypasses the specific validation check that triggers the "not a OLAP table" error. 1. Find the `PartitionId`:
Copy code
sql
    SHOW PARTITIONS FROM WORKFLOW_NODE_EXECUTION;
2. Run the command using the ID (no quotes):
Copy code
sql
    -- Replace <partition_id> with the number from the previous step
    ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (<partition_id>) VERSION TO 7720;
3. Clear the Batch Publish Cache If you see the version gap error repeating in logs, toggle the batch publish setting. This can force the FE to stop trying to "batch" the missing version and move on:
Copy code
sql
-- Disable batch publish temporarily
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Wait 1-2 minutes to see if the job completes, then turn it back on
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Manual Job Cancellation If the job is still stuck in
FINISHED_REWRITING
and blocking everything:
Copy code
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
* Note: If the system says "Job cannot be cancelled in this state," a FE Leader restart is the only way to clear the memory lock. Summary of why this happened In StarRocks 3.3.x shared-data mode, a race condition can occur where a Routine Load (or other write) commits a version (e.g., 7721) while the Schema Change is trying to claim the previous version (7720). Because Lake tables require strict version continuity, the "hole" at 7720 stops the pipeline. Recommendation: After fixing this, consider upgrading to a later 3.3.x patch (like 3.3.20+) where several "gtid mismatch" and "version discontinuity" deadlocks in Shared-data mode have been resolved. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • administration/management/FE_configuration.md • release_notes/release-3.3.md References • faq/operation_maintenance_faq.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • faq/Sql_faq.md
n
@Rocky mysql> ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (p1789084800000_1786579221279) VERSION TO 7720; ERROR 5503 (42000): Table 'WORKFLOW_NODE_EXECUTION' is not a OLAP table mysql> ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (90889287) VERSION TO 7720; ERROR 5503 (42000): Table 'WORKFLOW_NODE_EXECUTION' is not a OLAP table mysql> ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION ('90889287') VERSION TO 7720; ERROR 5503 (42000): Table 'WORKFLOW_NODE_EXECUTION' is not a OLAP table
r
The error "Table is not a OLAP table" occurs because
ADMIN SET PARTITION VERSION
is strictly limited to shared-nothing (Local Storage/OLAP) tables. In StarRocks 3.3.x (Shared-data mode), tables use the
LAKE
engine, and metadata is managed differently. Because you are seeing
partition.getVisibleVersion() + 1 != version.get(0)
, your table's metadata has a "hole" (e.g., it has version 7719 and is trying to jump to 7721, but 7720 is missing). Here are the specific steps to fix this for a Lake table: 1. The Correct Repair Command for Lake Tables In Shared-data mode, you must use
ADMIN REPAIR
with specific properties. If the basic
ADMIN REPAIR
failed with a "not OLAP" error, it is likely a validation bug in v3.3.17. Try running it without the partition name first to see if it targets the whole table:
Copy code
sql
ADMIN REPAIR TABLE WORKFLOW_NODE_EXECUTION 
PROPERTIES ("enforce_consistent_version" = "true");
* Why this works: For Lake tables, this command forces the FE to look at the underlying object storage (S3/OSS), find the last valid continuous version, and roll the metadata back to that state. * Note: This may result in the loss of the data that was in the "skipped" version, but it will unblock the table. 2. Restart the FE Leader (Highly Effective) In StarRocks 3.3.x, the state of a schema change job in
FINISHED_REWRITING
and the version sequence are often cached in the FE Leader's memory. 1. Identify the FE Leader node. 2. Restart the FE service. 3. Upon restart, the FE will reload the image and edit logs. It will re-verify the table's state against the object storage. In many cases, this forces the FE to "discover" the missing version or skip the gap, transitioning the job to
FINISHED
. 3. Clear the Transaction Lock If the version gap is caused by a stuck transaction: 1. Check for any "Running" or "Prepare" transactions on that table:
Copy code
sql
    SHOW PROC '/transactions';
2. If you see a transaction related to
WORKFLOW_NODE_EXECUTION
that is very old, it might be the "missing" version. You may need to wait for it to time out or contact support if it's stuck in a
PREPARE
state indefinitely. 4. Toggle Batch Publish (Workaround) Sometimes the FE gets stuck trying to batch a version that doesn't exist. Disabling batching can force it to process versions one-by-one and potentially clear the error:
Copy code
sql
-- Disable batch publish
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Wait a few minutes to see if the table unblocks, then re-enable
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
Summary Recommendation Since
ADMIN SET PARTITION VERSION
will never work for Lake tables: 1. Restart the FE Leader. This is the safest and most common fix for the
FINISHED_REWRITING
lock in 3.3.17. 2. If the restart fails, use Step 1 (
ADMIN REPAIR
on the whole table) to force a metadata rollback. 3. Upgrade: This specific "version gap" race condition during schema changes was a known issue in early 3.3.x releases and has been significantly improved in 3.3.20+. Consider a patch upgrade to prevent recurrence. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/shared_data_faq.md • administration/management/FE_configuration.md References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • faq/Sql_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
n
@Rocky mysql> ADMIN REPAIR TABLE WORKFLOW_NODE_EXECUTION -> PROPERTIES ("enforce_consistent_version" = "true"); ERROR 1064 (HY000): Getting syntax error at line 2, column 0. Detail message: Unexpected input 'PROPERTIES', the most similar input is {<EOF>, ';'}.
r
The errors you are encountering are due to version-specific limitations in StarRocks 3.3.17 for Shared-data (Lake) mode. 1.
PROPERTIES
Syntax Error:
The
PROPERTIES
clause for
ADMIN REPAIR TABLE
was introduced in v4.1. In v3.3, this command only accepts the table/partition name and does not support the consistency properties you tried. 2. "Not an OLAP table" Error: In v3.3, the
ADMIN SET PARTITION VERSION
command is strictly validated for
ENGINE=OLAP
(Shared-nothing) tables and will reject
ENGINE=LAKE
(Shared-data) tables. To resolve the version gap (
VisibleVersion + 1 != NextVersion
) and unblock your schema change in v3.3.17, follow these steps: 1. Identify the "Missing" Version Run the following to see the status of the transaction that corresponds to the "hole" (e.g., if Visible is 7719 and Next is 7721, look for the transaction that would have produced 7720):
Copy code
sql
-- Check for transactions stuck in PREPARE or COMMITTED state
SHOW PROC '/transactions';

-- Check for stuck compaction tasks that might be consuming a version
SHOW PROC '/compactions';
If you find a transaction in a
COMMITTED
but not
VISIBLE
state, it is likely the one blocking the sequence. 2. Restart the FE Leader (Highly Recommended) In Shared-data mode, the
FINISHED_REWRITING
state and the version publish queue are managed in the FE Leader's memory. * Action: Restart the FE Leader node. * Effect: Upon restart, the FE reloads the metadata and re-scans the transaction states. This often triggers a re-publish that bridges the version gap or clears the "Finished Rewriting" lock, allowing the job to transition to
FINISHED
. 3. Disable Batch Publishing (Workaround) If a specific transaction is failing to publish because it's part of a "batch" that has a hole, you can force the FE to publish versions one-by-one:
Copy code
sql
-- Disable batching to force individual version publishing
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Wait a few minutes to see if the table becomes visible/job finishes.
-- Afterward, re-enable it for performance:
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Manual Job Cancellation If the FE restart does not clear the
FINISHED_REWRITING
state, try to cancel the job manually. If it was truly finished but just failed to publish, a cancel might force a cleanup:
Copy code
sql
CANCEL ALTER TABLE COLUMN FROM WORKFLOW_NODE_EXECUTION;
Summary for v3.3.17 Since the
ADMIN
repair/set commands are not yet fully compatible with Lake tables in your version, a FE Leader restart is the standard operational fix for this specific metadata deadlock. If the issue persists, it may require a patch upgrade to 3.3.20+, which includes several fixes for Lake table version continuity during schema changes. References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/FE_configuration.md • faq/shared_data_faq.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md
n
@Rocky these are my partitions | 90781609 | p1788998400000_1786492829508 | 20015 | 20175 | 20187 | NORMAL | entryTime | [types: [BIGINT]; keys: [1789084800000]; ..types: [BIGINT]; keys: [1789171200000]; ) | executionInstanceId | 5 | 14.7GB | 167628690 | true | false | 321.40 | 322.00 | 322.00 | 16200 | 437583409611538432 | TXN_NORMAL | | 90889287 | p1789084800000_1786579221279 | 7558 | 7718 | 7730 | NORMAL | entryTime | [types: [BIGINT]; keys: [1789171200000]; ..types: [BIGINT]; keys: [1789257600000]; ) | executionInstanceId | 5 | 17.1GB | 191985258 | true | false | 321.80 | 322.00 | 324.00 | 6429 | 437764586286874624 | TXN_NORMAL |
r
The error "Table is not a OLAP table" occurs because, in your StarRocks version (3.3.17), some
ADMIN
commands (like
ADMIN SET PARTITION VERSION
) mistakenly validate for the
ENGINE=OLAP
property and do not yet recognize
ENGINE=LAKE
(Shared-data) tables, even though the underlying logic is needed. Additionally, your partition info shows a version discontinuity: * VisibleVersion: 7718 * NextVersion: 7730 * Gap: There is a hole from version 7719 to 7729. In Shared-data mode, a Schema Change job stuck in
FINISHED_REWRITING
is usually waiting for its specific version (likely 7720) to become "Visible." Because the sequence is broken (the system jumped to 7730), the job is in a "version gap deadlock." To resolve this in v3.3.17, follow these steps: 1. Restart the FE Leader (Highly Recommended) In Shared-data mode, the state of the schema change job and the publish queue are managed in the FE Leader's memory. * Action: Identify and restart the FE Leader node. * Result: When the FE Leader comes back up, it reloads the metadata and re-evaluates the schema change job. In many cases, it will detect that the rewriting is done and transition the job to
FINISHED
(bridging the version gap automatically) or at least allow you to cancel it. 2. Check for Stuck Transactions The "hole" at version 7719 might be caused by a transaction that is
COMMITTED
but not yet
VISIBLE
.
Copy code
sql
SHOW PROC '/transactions';
Look for any transaction with a
TXN_ID
that corresponds to the timeframe when the gap occurred. If it is stuck in
COMMITTED
or
PREPARE
, it is blocking the version sequence. 3. Clear the Batch Publish Cache Sometimes the FE gets stuck trying to "batch" the missing version. Toggling this setting can force it to process individual versions and potentially clear the deadlock:
Copy code
sql
-- Disable batch publish temporarily
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Wait 2-3 minutes to see if the schema change job finishes.
-- If it finishes, turn it back on:
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
4. Try the "Partition ID" Workaround If you still need to force the version and the previous command failed, try using the Partition ID without quotes. Sometimes this bypasses the specific "OLAP" check:
Copy code
sql
-- Use the Partition ID 90889287 directly (no quotes)
ADMIN SET TABLE WORKFLOW_NODE_EXECUTION PARTITION (90889287) VERSION TO 7720;
(Note: If this still gives the "not OLAP" error, the command is simply not supported for Lake tables in your current patch.) Summary & Prevention This "version discontinuity" during schema changes is a known race condition in early 3.3.x releases when concurrent writes occur. * Long-term fix: Upgrade to 3.3.20 or later. Several bug fixes for
LakeTableSchemaChangeJob
version continuity and "GTID mismatch" were specifically added to address this exact deadlock in Shared-data mode. References • faq/shared_data_faq.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • administration/management/FE_configuration.md • release_notes/release-3.3.md References • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md