http://starrocks.io logo
Join Slack
Powered by
I'm hitting a wall on this issue after 2 months of...
# questions-and-troubleshooting
t
I'm hitting a wall on this issue after 2 months of repeated efforts to address the issue. We had an issue where our front end node ran out of memory, we have 61 routine loads currently for 61 different kafka topics feeding into their own tables. We had a misconfiguration on our memory limits that had causes a bunch of OOM issues in our staging starrocks environment which causes a bunch of routine loads to fail, and we addressed this and some of the routine loads we simply deleted and recreated some of them were resumed and carried on correctly. The thing is we now have 83 transactions many of which seems completely stuck for 2 months now. They never timeout, they never recover, and there's seemingly no way to get rid of them. Most of these are routine loads, but some of them are stuck compaction transactions, which also seems to mean that subsequent transactions on many of the effected tables doesn't happen anymore at least for the given partition. I'm not really sure how to resolve this, it doesn't seem like I can kill these transactions because everything related to them can be stopped/killed/removed and the transactions continue to exist. This seems like it has a bunch of negative downstream effects. Right now I've killed ALL my routine loads (this is a staging env) and yet I have 64 transactions which are months old and almost all of them are ROUTINE_LOAD_TASK as their LoadJobSourceType. If I try 
CANCEL LOAD FROM ...
on the stuck tasks it just said the load job doesn't exist. So these are just completely orphaned transactions that never go away. I'd really like a human (ideally a maintainer) to chime in on this, because Rocky hasn't been able to help with this and it seems like it might be a bug.
✅ 1
@Rocky can you assist?
r
Hello! I'm Rocky, your StarRocks assistant. I can help with: * SQL syntax and queries * Table design and data modeling * Data ingestion (pipelines, routines, etc.) * Cluster configuration and performance tuning * Troubleshooting errors How can I help you with your StarRocks project today?
t
@Rocky I'm hitting a wall on this issue after 2 months of repeated efforts to address the issue. We had an issue where our front end node ran out of memory, we have 61 routine loads currently for 61 different kafka topics feeding into their own tables. We had a misconfiguration on our memory limits that had causes a bunch of OOM issues in our staging starrocks environment which causes a bunch of routine loads to fail, and we addressed this and some of the routine loads we simply deleted and recreated some of them were resumed and carried on correctly. The thing is we now have 83 transactions many of which seems completely stuck for 2 months now. They never timeout, they never recover, and there's seemingly no way to get rid of them. Most of these are routine loads, but some of them are stuck compaction transactions, which also seems to mean that subsequent transactions on many of the effected tables doesn't happen anymore at least for the given partition. I'm not really sure how to resolve this, it doesn't seem like I can kill these transactions because everything related to them can be stopped/killed/removed and the transactions continue to exist. This seems like it has a bunch of negative downstream effects. Right now I've killed ALL my routine loads (this is a staging env) and yet I have 64 transactions which are months old and almost all of them are ROUTINE_LOAD_TASK as their LoadJobSourceType. If I try 
CANCEL LOAD FROM ...
on the stuck tasks it just said the load job doesn't exist. So these are just completely orphaned transactions that never go away.
r
It sounds like your StarRocks metadata has entered a state where orphaned transaction records are persisting in the Transaction Manager's memory and Edit Log, even after their associated Routine Load jobs have been deleted. These transactions are likely stuck in a 
COMMITTED
or 
PREPARED
state and are failing to reach 
VISIBLE
(Finish) due to the earlier OOM events or internal thread pool rejections. To resolve these "un-killable" orphaned transactions, follow these steps in order: 1. Identify the Stuck Transaction Details Use the 
SHOW PROC
command to get the exact 
TransactionId
and 
Label
for each stuck transaction. This is more reliable than 
SHOW LOAD
when jobs are missing.
Copy code
sql
-- Replace <db_id> with your database ID (found via SHOW PROC '/transactions')
SHOW PROC '/transactions/<db_id>/running';
Look for transactions where the 
CreateTime
is very old. Note the TransactionId and the Label. 2. Try Canceling by Label (Even if Job is Deleted) Sometimes 
CANCEL LOAD
fails if you reference a job name that no longer exists, but it may still work if you target the specific Label associated with the orphaned transaction.
Copy code
sql
CANCEL LOAD FROM <db_name> WHERE LABEL = "<stuck_label>";
3. Clear Stuck Compaction Transactions If you see transactions with 
LoadJobSourceType
as 
COMPACTION
(often in shared-data/lakehouse clusters or specific versions), you can manually cancel them:
Copy code
sql
-- This command is specifically for compaction tasks
CANCEL COMPACTION WHERE TXN_ID = <txn_id>;
4. Force Label and Transaction Cleanup You can temporarily tune the metadata cleaner to be extremely aggressive. This can sometimes trigger the FE to purge orphaned entries that are past their TTL but haven't been picked up.
Copy code
sql
-- Set the retention time to 0 to force a cleanup of historical and orphaned metadata
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "0");
ADMIN SET FRONTEND CONFIG ("transaction_clean_interval_second" = "1");
Wait a few minutes, then check 
SHOW PROC '/transactions/...'
again. Remember to set these back to their defaults (typically 10800 and 30 respectively) afterward. 5. Restart the Leader FE (Most Effective) If the transactions persist for months and survive configuration changes, they are likely stuck in the FE's memory-resident 
TransactionManager
. * Restart the Leader FE node. * When a new Leader is elected, it reloads metadata from the image and Edit Logs. During this recovery phase, it should re-validate all 
RUNNING
transactions. If the associated job or session is long gone, the FE recovery logic typically aborts these tasks. 6. Destructive Recovery (Last Resort) If the stuck transactions are blocking compaction on a specific partition and you cannot clear them via FE restart: 1. Backup the data (e.g., 
INSERT INTO ... SELECT
to a temporary table). 2. Truncate or Drop/Recreate the Partition: This clears all metadata associated with that partition, including the transaction locks. 3. Use 
FORCE
: If dropping a database/table fails because of these transactions, use 
DROP TABLE <name> FORCE;
. Why this happened? This usually occurs because StarRocks guarantees exactly-once semantics for Routine Load. If the FE crashes (OOM) at the exact moment a transaction is committing but before it's published, the state can become "orphaned." In some versions, if the internal 
PublishVersion
thread pool is full, these transactions stay in 
COMMITTED
indefinitely. Recommended follow-up: Ensure your FE JVM heap is sufficiently large and check 
max_running_txn_num_per_db
(default 100) to ensure you aren't hitting limits that exacerbate the queueing. References • sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md
t
@Rocky transaction_clean_interval_second doesn't exist
r
I apologize for the confusion regarding 
transaction_clean_interval_second
. In StarRocks, that parameter is immutable, meaning it cannot be changed dynamically via 
ADMIN SET FRONTEND CONFIG
. It must be set in your 
fe.conf
file and requires an FE restart to take effect. Since these transactions have persisted for two months despite you deleting the jobs, they are likely "orphaned" in the FE's memory-resident Transaction Manager. Because they aren't in a 
FINISHED
or 
CANCELLED
state, the background cleaner (which uses the intervals mentioned before) ignores them. Here is the step-by-step path to clearing these 83 orphaned transactions: 1. Identify the Exact State First, determine the state of these transactions. This will dictate the next command.
Copy code
sql
-- Replace <db_id> with your database ID
SHOW PROC '/transactions/<db_id>/running';
Look at the 
State
column. It will likely be 
PREPARED
or 
COMMITTED
. 2. Force Abort for Compaction Transactions For the transactions with 
LoadJobSourceType
as 
COMPACTION
, use the dedicated cancellation command (must be run on the Leader FE):
Copy code
sql
CANCEL COMPACTION WHERE TXN_ID = <txn_id>;
Note: If the 
CommitTime
is already set in 
SHOW PROC '/compactions'
, this command may not work, and a restart is required. 3. Attempt Rollback via REST API (for Routine Load/Stream Load) Since 
CANCEL LOAD
is failing because the job is gone, you can try to trigger a rollback directly via the Stream Load transaction interface using the Label found in the 
SHOW PROC
output:
Copy code
bash
curl --location-trusted -u <user>:<pass> \
  -H "label:<stuck_label>" \
  -H "db:<db_name>" \
  -XPOST http://<fe_host>:<fe_http_port>/api/transaction/rollback
If the FE still has the transaction in its memory map, this API call often forces an abort even when the high-level job metadata is missing. 4. The "Nuclear" Option: Restart the Leader FE If the transactions are months old and survive the steps above, they are likely stuck in the FE's internal 
EditLog
or memory and are failing to progress due to a lack of an associated "Job" object to trigger their completion. The most effective way to clear orphaned metadata is to restart the Leader FE. 1. When the FE restarts, it reloads the metadata image. 2. During the "Recovery" phase, the new Leader FE scans all 
RUNNING/PREPARED
transactions. 3. If it finds a transaction whose corresponding Routine Load job no longer exists, the recovery logic should automatically abort it to maintain consistency. 5. Aggressive Label Purge (to clean up the "Job doesn't exist" errors) To clear the metadata of the deleted jobs themselves so you can reuse labels:
Copy code
sql
-- This IS mutable and can be set via SQL
ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "60");
Wait about 5-10 minutes for the background cleaner to run, then check if the orphaned labels are gone. Be sure to set this back to the default (259200, which is 3 days) afterward. Why did this happen? This is typically caused by the FE crashing (OOM) while a transaction was in the 
COMMITTED
state. The transaction was written to the Edit Log but the "Publish Version" task (which makes data visible) was lost or the thread pool was full. Because the Routine Load job was later deleted, there was no "owner" to retry the publish or abort the task. Recommendation: After clearing these, check your FE JVM heap size. For 61 concurrent Kafka topics, ensure you have at least 8GB-16GB of heap to handle the metadata overhead and prevent future OOMs. References • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md • administration/management/compaction.md
t
@Rocky restarting the leader doesn't help
r
To resolve orphaned transactions that persist even after a leader FE restart and the deletion of their parent jobs, you are likely dealing with a known "limbo" state where transactions have reached the 
COMMITTED
state but cannot transition to 
VISIBLE
because the job metadata that would normally trigger the "Publish Version" task is missing. Because StarRocks is designed for exactly-once semantics, transactions in the 
COMMITTED
state typically *cannot be aborted*—the system expects them to finish. If the FE restart didn't clear them, they are persisted in the BDBJE edit logs. Here are the specific ways to force-clear these transactions: 1. Identify the Transaction State Run the following to confirm if they are 
COMMITTED
or `PREPARED`:
Copy code
sql
-- Get your Database ID first
SHOW PROC '/transactions';
-- View running transactions for that DB
SHOW PROC '/transactions/<db_id>/running';
* If `PREPARED`: They should eventually timeout or can be aborted via the REST API. * If `COMMITTED`: They are stuck waiting for a "Publish" signal that isn't coming. 2. Manual Rollback via Stream Load API (Most Successful) Even though these were Routine Load transactions, the 
TransactionManager
handles them similarly to Stream Loads. You can try to force an abort/rollback using the specific Label found in the 
SHOW PROC
output:
Copy code
bash
curl --location-trusted -u <user>:<pass> \
  -H "label:<stuck_label>" \
  -H "db:<db_name>" \
  -XPOST http://<fe_host>:<fe_http_port>/api/transaction/rollback
Try this for one or two labels first to see if the transaction disappears from the 
SHOW PROC
list. 3. Clear Compaction Transactions For those marked as 
COMPACTION
, use the internal cancellation command (must be on the Leader FE):
Copy code
sql
-- Note: Only works if CommitTime is NULL in 'SHOW PROC /compactions'
CANCEL COMPACTION WHERE TXN_ID = <txn_id>;
If this fails, and because you mentioned they are blocking other compactions, you may need to restart all BE/CN nodes. Compaction tasks are partially managed in BE memory; a BE restart will fail the task, allowing the FE to eventually mark the transaction as failed. 4. Force Metadata Cleanup (Staging Env Only) If the REST API and restarts do not work, the transactions are tied to specific `TableId`s or `PartitionId`s. Dropping the table/partition is the only way to break the lock in the metadata manager. 1. Drop the affected Partition/Table with 
FORCE
:
Copy code
sql
    DROP TABLE <table_name> FORCE;
Using 
FORCE
bypasses the Trash bin and should trigger a cleanup of all associated metadata, including orphaned transaction locks. 2. Truncate: If you want to keep the table, try 
TRUNCATE TABLE <table_name>
. While usually blocked by active transactions, in some versions this can force a "reset" of the versioning metadata. 5. Check "Publish Version" Thread Pool If you see a lot of 
COMMITTED
transactions, your FE may have a full task pool. Check your 
fe.log
for: 
Task java.util.concurrent.FutureTask@... rejected from agent-task-pool
If found, increase the pool size in 
fe.conf
and restart:
Copy code
conf
# Add/Update in fe.conf
async_load_task_pool_size = 2048
Why 
CANCEL LOAD
failed The 
CANCEL LOAD
command checks the 
LoadManager
for a job name. Since you deleted the Routine Load jobs, that manager returns "Job not found" before it even looks at the 
TransactionManager
. This is why the transactions appear orphaned. Final Recommendation: If this is a staging environment and the manual rollback API doesn't work, the most reliable path is to Drop the affected tables with the 
FORCE
keyword and recreate them. This will purge the internal metadata IDs that the stuck transactions are clinging to. References • administration/management/compaction.md • loading/SQL_transaction.md
t
@Rocky what is the HTTP fe port?
r
The default port for the StarRocks Frontend (FE) HTTP server is 
8030
. This is controlled by the 
http_port
parameter in your 
fe.conf
file. You can verify which port your specific cluster is using by checking that file or by running:
Copy code
bash
grep "http_port" fe/conf/fe.conf
To use the rollback API I mentioned earlier, the command would look like this (assuming the default port):
Copy code
bash
curl --location-trusted -u <user>:<pass> \
  -H "label:<stuck_label>" \
  -H "db:<db_name>" \
  -XPOST http://<fe_leader_host>:8030/api/transaction/rollback
Other common FE ports for reference: * 9030: MySQL Client Port (
query_port
) * 9020: Thrift RPC Port (
rpc_port
) * 9010: BDBJE Edit Log Port (
edit_log_port
) References • administration/management/FE_configuration.md • deployment/environment_configurations.md • administration/http_interface.md
t
@Rocky what is the default?
r
The default FE HTTP port is 8030. If you haven't explicitly changed it in your 
fe.conf
file, you can access the FE HTTP server at 
http://<fe_host>:8030
. Other default FE ports: * 8030: HTTP Server (
http_port
) * 9030: MySQL Client/Query Port (
query_port
) * 9020: Thrift RPC Port (
rpc_port
) * 9010: BDBJE Edit Log Port (
edit_log_port
) * 8443: HTTPS Server (
https_port
, introduced in v4.0) References • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
t
@Rocky this returns { "Status": "FAILED", "Message": "class com.starrocks.common.StarRocksException: Can not abort COMMITTED transaction 6230171, label is optimus_dbc_firmware_version_routine_load-18868-019eb29e-3455-7dbf-9f30-35bc139e3bb2" }
r
The error message you received—`Can not abort COMMITTED transaction`—is expected behavior in StarRocks. A transaction in the 
COMMITTED
state has already successfully written data to the BE nodes and recorded that success in the FE's metadata. Because StarRocks guarantees exactly-once semantics, it does not allow you to "abort" a transaction that has already been confirmed as committed. It can only move forward to the 
VISIBLE
state. If the transaction is orphaned (stuck 
COMMITTED
after the Routine Load job was deleted) and restarting the Leader FE didn't clear it, follow these steps: 1. Identify the Blockage (FE Logs) The 
PublishVersionDaemon
in the FE is responsible for moving transactions from 
COMMITTED
to 
VISIBLE
. If it's stuck, there is an error in the publish phase. Search your 
fe.log
for the transaction ID:
Copy code
bash
grep "6230171" fe.log
Common issues found here: * 
Task rejected from agent-task-pool
: The FE's thread pool for sending tasks to BEs is full. * 
publish version timeout
: The BEs are not responding to the publish request within the timeout. 2. Restart the BE Nodes Restarting the FE only restarts the manager; however, the actual "Publish Version" tasks are often stuck in the memory of the BE/CN nodes. * Action: Restart all BE/CN nodes in the cluster. This clears the agent task queues on the worker side and forces the FE to re-issue the publish tasks upon the next daemon cycle. 3. Check for Version Gaps If you have multiple stuck transactions, it might be because a lower version transaction is failing, blocking all subsequent versions for that partition. Run this to see the state of tablets for the target table:
Copy code
sql
SHOW TABLET FROM <table_name>;
-- Look for 'Version' and 'LstPublishTime' to see if some tablets are lagging.
4. Version-Specific Fixes * If you are on v4.1+ (Shared-data/Lake mode): StarRocks introduced a specific command to unwedge these exact scenarios:
Copy code
sql
    ADMIN SKIP COMMITTED TRANSACTION <txn_id>;
This forces the system to skip the publish phase for that specific transaction. * If you are on an older version: There is no "skip" command. You must resolve why the BE cannot publish (often by restarting BEs or increasing thread pools). 5. The "Nuclear" Option (Metadata Cleanup) If the transaction is permanently stuck and preventing other operations (like schema changes or compactions), the only way to clear it without the 
ADMIN SKIP
command is to destroy the metadata it is attached to. * Force Drop:
Copy code
sql
    DROP TABLE <table_name> FORCE;
Note: Using the 
FORCE
keyword is critical; it bypasses the Trash bin and immediately purges the table IDs from the metadata manager, which should force-clear any associated orphaned transactions. Recommendation: Try restarting all BEs first. If that fails and you are not on v4.1, check your FE logs for the specific error reason for txn 
6230171
to see if it's a simple thread pool exhaustion issue. References • administration/management/BE_configuration.md • loading/SQL_transaction.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/monitoring/metrics.md
t
@Rocky ADMIN SKIP COMMITTED TRANSACTION succeeds but doesn't clear the transaction
r
If the 
ADMIN SKIP COMMITTED TRANSACTION
command returns "Query OK" but the transaction still appears stuck, it is likely due to how the command works and the sequence of version publishing in StarRocks. Here is what is happening and how to resolve it: 1. It converts the transaction to a "No-Op" When you "skip" a committed transaction, StarRocks does not delete it or mark it as aborted. Instead, it performs a no-op publish. * The Result: The transaction will transition from 
COMMITTED
to 
VISIBLE
. * The Data: All data associated with that transaction ID is discarded. * The Version: The table's version still advances by 1, but no new data is actually added to the tablets. Action: Check the current status of the transaction. It should now (or shortly) show as 
VISIBLE
instead of `COMMITTED`:
Copy code
sql
SHOW TRANSACTION WHERE id = 6230171;
2. Check for Sequential Blockage (The "Version Gap") StarRocks must publish versions in strict sequential order. If you have multiple transactions for the same partition, and a previous (older) transaction is still stuck in the 
COMMITTED
state, skipping the newer one will not clear the queue. * Action: Run the following to see if there are older transactions for the same table:
Copy code
sql
    SHOW PROC '/transactions';
    -- Or check the specific table/partition:
    SHOW TABLET FROM <table_name>;
If there is any transaction with a smaller ID or an older timestamp that is still 
COMMITTED
, you must skip or resolve that one first. 3. Verification of the "Skip" Flag The 
ADMIN SKIP
command flags the transaction in the FE's memory. The 
PublishVersionDaemon
must then pick up this flag and execute the metadata update. * Check FE Logs: Search for the transaction ID to see if the skip task was rejected or if there was an error writing the "no-op" metadata:
Copy code
bash
    grep "6230171" fe.log
* Check Configuration: Ensure this "escape hatch" is actually enabled in your session or global config:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_admin_skip_committed_txn';
(It must be 
true
for the command to take effect). 4. Why it might still be "stuck" If the status is still 
COMMITTED
after several minutes: * Thread Pool Exhaustion: The FE's 
agent-task-pool
might be full, preventing even the "no-op" task from being sent. Restarting the FE Leader can clear the task queue. * Metadata Sync Issue: In shared-data (Cloud Native) mode, the FE might be struggling to write the new metadata file to your object storage (S3/OSS/HDFS). Check your storage health and the FE logs for 
CloudNativeIndex
or 
LakeService
errors. Summary of next steps: 1. Verify if the status changed to 
VISIBLE
using 
SHOW TRANSACTION
. 2. If it's still 
COMMITTED
, check if there is an older transaction for the same table that needs skipping first. 3. Restart the FE Leader to force a refresh of the 
PublishVersionDaemon
. References • loading/SQL_transaction.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • administration/Meta_recovery.md • loading/Stream_Load_transaction_interface.md • administration/management/compaction.md References
t
@Rocky NoOpPublish is now true, but the state never changes from COMMITTED
r
If 
NoOpPublish
is set to 
true
but the transaction remains in the 
COMMITTED
state, the most likely reason is that StarRocks publishes versions in strict sequential order. Even if you have skipped a transaction, it cannot become 
VISIBLE
until every transaction with a smaller ID (or an earlier commit time) for that same partition has also reached the 
VISIBLE
state. Follow these steps to unblock the sequence: 1. Find the "Blocker" Transaction Check if there are older transactions for the same table/partition that are also stuck in 
COMMITTED
.
Copy code
sql
-- Check all transactions for this database
SHOW PROC '/transactions';
Look for any transaction where: * Status is 
COMMITTED
. * Transaction ID is smaller than 
6230171
. * Table/Partition is the same as the one you are trying to fix. Action: If you find an older stuck transaction, you must run 
ADMIN SKIP COMMITTED TRANSACTION <older_txn_id>
on that one first. The "skip" will propagate up the chain once the oldest one is resolved. 2. Check for FE Thread Pool Exhaustion The 
PublishVersionDaemon
in the FE Leader is responsible for moving these transactions to 
VISIBLE
. If its task pool is full, it won't even process the "No-Op" skip. * Search FE Logs: Look for "rejected from agent-task-pool" or "publish version timeout" in your 
fe.log
. * Temporary Fix: Restart the FE Leader node. This clears the internal task queue and forces the daemon to restart the publish sequence for all committed transactions. 3. Verify BE/Storage Health (Lake Mode) Since 
ADMIN SKIP
is a feature for Shared-data (Lake) tables, a "No-Op Publish" still requires writing a new metadata file to your object storage (S3/OSS/HDFS). * If the FE cannot write to the object storage or communicate with the 
lake_service
on the BEs, the version will not advance. * Check BE Logs: Look for errors related to 
lake_service
or 
publish_version
. * Action: Restart the BE nodes to ensure the agent tasks are not hung on the worker side. 4. Forced Sequential Skip If you have a long list of stuck transactions, skipping just the one you care about won't work. You need to skip the oldest one first: 1. Identify the transaction with the minimum ID currently in 
COMMITTED
for that table. 2. Skip it. 3. Wait 15-30 seconds. 4. Check if the next transaction in the list automatically moves to 
VISIBLE
. Summary The state won't change until all preceding transactions for that partition are either successfully published or also skipped. If you have verified there are no older blockers, restarting the FE Leader is the most effective way to "nudge" the metadata manager into processing the skip. References • loading/SQL_transaction.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • administration/Meta_recovery.md • loading/Stream_Load_transaction_interface.md • administration/management/compaction.md • sql-reference/System_variable.md • faq/shared_data_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md
t
@Rocky I can't skip INSERT_STREAMING type transactions
r
The 
ADMIN SKIP COMMITTED TRANSACTION
command has specific limitations that explain why it may not be working for your 
INSERT_STREAMING
(Routine Load) transaction: 1. It only supports Shared-Data (Cloud Native) Tables The 
ADMIN SKIP
command was introduced in v4.1 as an emergency escape hatch specifically for Shared-data (Lake) tables. * If your table is a standard OlapTable (Local Storage): This command is not supported. StarRocks does not currently have a "skip" mechanism for local storage tables because the data is already physically written to the BEs; skipping would cause permanent data inconsistency that the local metadata cannot resolve via "No-Op." * How to check: Run 
SHOW CREATE TABLE <name>;
. If it does not contain a 
PROPERTIES ("storage_volume" = "...")
, it is likely a local table. 2. Transaction Type Support According to the StarRocks v4.1 technical specifications, the skip command currently only supports: * Load transactions (which includes Routine Load/INSERT_STREAMING) * Lake-compaction transactions * It does NOT support 
ALTER
or 
SCHEMA_CHANGE
transactions yet. 3. Why it "Succeeds" but doesn't clear If you are on a Shared-data cluster and the command returned "Query OK" and set 
NoOpPublish: true
, but the state is still `COMMITTED`: * The Sequential Blocker: StarRocks cannot publish version 
N
until version 
N-1
is visible. If you have an older transaction (smaller ID) for the same partition that is also stuck, your 
INSERT_STREAMING
txn will stay 
COMMITTED
indefinitely. You must find the oldest stuck transaction for that table and skip it first. * Routine Load Management: Since your txn is from a Routine Load job (
optimus_dbc_...
), the Job itself might be trying to "re-commit" or "re-publish" the same label. * Action: 
PAUSE
the Routine Load job first, then run the skip command on the txn ID. 4. How to resolve if you can't use "SKIP" If you are on an older version or a local-storage table where 
ADMIN SKIP
isn't an option, use these manual steps: 1. Check FE Thread Pools: The FE can get stuck if the publish task pool is full. Look at 
fe.log
for: 
Task rejected from agent-task-pool
If you see this, restart the FE Leader. This is the most common way to "un-stick" a committed transaction that isn't moving to visible. 2. Restart BEs: Sometimes a BE node is "holding" the transaction in its local memory but failing to report the success back to the FE. Restarting all BEs clears these hung tasks. 3. The Metadata Cleanup (Last Resort): If the transaction is blocking all other loads and you cannot clear it:
Copy code
sql
    DROP TABLE <table_name> FORCE;
This removes the table and all its metadata from the FE, effectively purging the stuck transaction entry. You can then recreate the table and resume the Routine Load. Summary Recommendation: First, verify if you have an older transaction stuck (check 
SHOW PROC '/transactions'
). If not, restart your FE Leader to force the 
PublishVersionDaemon
to retry the state transition. References • faq/loading/Routine_load_faq.md • loading/RoutineLoad.md • sql-reference/sql-statements/loading_unloading/routine_load/SHOW_ROUTINE_LOAD.md
t
these are all S3 stored starrocks tables
@Rocky these are all S3 stored starrocks tables
r
For S3-stored (shared-data) tables, the 
ADMIN SKIP COMMITTED TRANSACTION
command is the correct "emergency" tool, but it often fails to move the state from 
COMMITTED
to 
VISIBLE
if the transaction sequence is broken. Even with 
NoOpPublish: true
, the transaction will stay 
COMMITTED
if there is a version gap. Here is how to resolve it: 1. Check for the "Version Gap" (Most Common) In shared-data mode, StarRocks must publish versions in strict sequential order. If Transaction ID 
100
is stuck, Transaction ID 
101
(your 
INSERT_STREAMING
txn) cannot become visible, even if you mark it as a "Skip/No-Op." * Action: Run this to find the oldest stuck transaction for that database:
Copy code
sql
    SHOW PROC '/transactions';
Sort by 
TransactionId
or 
CommitTime
. If there is any transaction with a smaller ID that is still in the 
COMMITTED
state, you must run the 
ADMIN SKIP
command on that older ID first. The queue will then "unzip" and process your newer skip. 2. Restart the FE Leader The 
PublishVersionDaemon
is a background thread on the FE Leader that processes these skips. If it has encountered an internal error or the task pool is full, it may stop processing the "No-Op" flags. * Action: Restart the FE Leader. This clears the internal task queue and forces the daemon to scan for all 
NoOpPublish=true
transactions and move them to 
VISIBLE
. 3. Pause the Routine Load Job If your 
INSERT_STREAMING
transaction is part of an active Routine Load, the job might be attempting to retry or keep the transaction "active" in a way that conflicts with the skip. * Action: 1. 
PAUSE ROUTINE LOAD FOR <job_name>;
2. Run 
ADMIN SKIP COMMITTED TRANSACTION <txn_id>;
3. Wait for the state to change to 
VISIBLE
. 4. 
RESUME ROUTINE LOAD FOR <job_name>;
4. Check for v4.1.0/v4.1.1 Known Issues If you are on version 4.1.0 or 4.1.1, there were known bugs where certain 
INSERT_STREAMING
(multi-table or high-concurrency) transactions would not transition correctly even after a commit. * Action: Check your version with 
SELECT version();
. If you are on an early 4.1.x release, upgrading to v4.1.3+ is highly recommended as it contains specific fixes for transactions stuck at `COMMITTED`/`PREPARING` in shared-data clusters. 5. Verify Frontend Configuration Ensure the "escape hatch" is globally enabled. If this is 
false
, the command may return "OK" but the FE will ignore the flag during the publish cycle:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'enable_admin_skip_committed_txn';
-- If false, set it to true:
ADMIN SET FRONTEND CONFIG ("enable_admin_skip_committed_txn" = "true");
Summary of the "Hard Reset" If the above fails, the most reliable sequence is: 1. Identify the earliest stuck TxnID for the table. 2. Skip it. 3. Restart the FE Leader. 4. If it remains stuck after 5 minutes, you may need to 
DROP TABLE <name> FORCE
and recreate it, as this is the only way to completely purge the version metadata from S3/for an S3-backed table's lifecycle. References • faq/shared_data_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md
t
@Rocky I'm running 4.1.4
r
Since you are on v4.1.4 and 
NoOpPublish
is already set to 
true
, the system has acknowledged your skip request, but the "Publish" thread is currently unable to finalize the state change. In StarRocks v4.1.x (Shared-data/Lake mode), a transaction stuck in 
COMMITTED
with 
NoOpPublish=true
almost always indicates one of the following two specific blockers: 1. The "Sequential Version" Block (Most Likely) StarRocks cannot publish version $N$ if version $N-1$ is not yet 
VISIBLE
. Because 
INSERT_STREAMING
(Routine Load) generates many high-frequency transactions, it is common for a single failed transaction to create a "traffic jam" behind it. How to verify: Run the following and look at the 
TransactionId
column:
Copy code
sql
SHOW PROC '/transactions';
* Identify your stuck Transaction ID (let's say it's 
1005
). * Look for any transaction with an ID smaller than yours (e.g., 
1003
or 
1004
) for the same table that is still in 
COMMITTED
or 
PREPARING
state. * Action: You must apply 
ADMIN SKIP COMMITTED TRANSACTION
to the oldest stuck transaction first. Once the oldest one moves to 
VISIBLE
, the others will follow like dominoes. 2. FE Agent-Task Pool Exhaustion In v4.1.4, there is a known issue where if the FE's internal 
agent-task-pool
is full, the 
PublishVersionDaemon
cannot submit the task to mark the transaction as visible—even if it's a "No-Op." How to verify: Check your 
fe.log
on the Leader FE for this specific error: 
WARN ... Task ... rejected from agent-task-pool
* Action: If you see this, you must restart the Leader FE. This clears the task queue and restarts the publish cycle. 3. Check for Concurrent Schema Change In Lake mode, if a 
SCHEMA_CHANGE
(Alter table) is running or stuck in 
FINISHED_REWRITING
, it holds a lock on the table's version metadata. This blocks all 
INSERT_STREAMING
transactions from moving to 
VISIBLE
. * Action: Check 
SHOW ALTER TABLE COLUMN;
. If a job is stuck, try to 
CANCEL ALTER TABLE COLUMN FROM <table_name>;
. 4. Special Note on Routine Load (
INSERT_STREAMING
) Routine Load jobs manage their own transaction lifecycle. Sometimes the job metadata on the FE gets out of sync with the Global Transaction Manager (GTM). * Action: 1. 
PAUSE ROUTINE LOAD FOR <job_name>;
2. Check if the txn is still 
COMMITTED
. If it is, and you've already set 
NoOpPublish=true
, restart the FE. 3. Once the transaction is gone or 
VISIBLE
, 
RESUME
the load. Summary Checklist for 4.1.4: 1. Is there a smaller TxnID stuck? If yes, skip that one first. 2. Restart the Leader FE. (This resolves 90% of "stuck in Committed" issues in v4.1.x by clearing the thread pool). 3. Verify Object Storage connectivity. Ensure the FE can still write to S3. If the FE cannot write the new version's metadata file to S3, it will never reach 
VISIBLE
. If the transaction persists after an FE restart and there are no older blockers, it may be a metadata inconsistency. At that point, the most stable path is to 
DROP TABLE <name> FORCE
and recreate it to reset the version sequence. References • loading/SQL_transaction.md • loading/Stream_Load_transaction_interface.md • administration/management/monitoring/metrics.md • faq/shared_data_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md
t
@Rocky I've skipped all the committed transactions that I possibly can and restarted but nothing has cleared or gone into VISIBLE state
r
Since you are on v4.1.4, have already applied the skip command, and restarted the FE without success, the issue is likely a version sequence gap caused by a different type of transaction or a metadata inconsistency on S3. In Shared-data mode, the 
VisibleVersion
cannot advance if there is a "hole" in the version history. If you've skipped all 
INSERT_STREAMING
transactions but the table is still stuck, check these three specific areas: 1. Check for Stuck Compaction Transactions In Lake mode, Compactions are also transactions that consume a version number. If a compaction transaction (type 
LAKE_COMPACTION
) is stuck in 
COMMITTED
or 
PREPARED
, it will block all subsequent 
INSERT_STREAMING
versions from becoming visible. * Action: Run 
SHOW PROC '/compactions';
. * Look for any row where the 
Error
is not NULL or the 
FinishTime
is empty. * If you find a stuck compaction, get its 
TxnID
and run:
Copy code
sql
    -- Note: Skip command also works for lake-compaction in v4.1+
    ADMIN SKIP COMMITTED TRANSACTION <txn_id>;
2. Identify the Version Gap Compare the actual version numbers to see exactly where the "hole" is. * Action: Run 
SHOW PARTITIONS FROM <table_name>;
. * Check the 
VisibleVersion
, 
NextVersion
, and 
CompactVersion
columns. * If 
VisibleVersion
is 
100
and 
NextVersion
is 
105
, there are 4 missing versions (
101, 102, 103, 104
). * Now, find the transactions corresponding to those specific missing versions:
Copy code
sql
    -- Search for the specific database/table in the txn list
    SHOW PROC '/transactions';
Look for any transaction (even those not labeled 
INSERT_STREAMING
, such as 
SCHEMA_CHANGE
or 
COMPACTION
) that falls between the current 
VisibleVersion
and the version of your stuck load. 3. Check for "PREPARED" Transactions The 
ADMIN SKIP
command only works on 
COMMITTED
transactions. If you have an older transaction stuck in the 
PREPARED
state (common if a BE/CN crashed mid-load), it will block the publish queue just as effectively. * Action: Look at 
SHOW PROC '/transactions';
for any rows where 
State
is 
PREPARED
. * Fix: Use 
ROLLBACK TRANSACTION <txn_id>;
for these. You cannot "skip" a prepared transaction; you must abort it. 4. Inspect the Leader FE Logs for "Publish" Errors If the metadata on S3 is inaccessible or corrupted, the FE will keep trying to publish and failing silently in the background. * Action: Search your 
fe.warn.log
or 
fe.log
for the table name or the word 
publish
. * Look for: 
fail to publish version
, 
tablet meta not found
, or 
S3 connection refused
. * If you see 
fail to get tablet meta from remote storage
, it means the data files for a committed transaction were deleted from S3 before they became visible (a rare race condition). Final "Nuclear" Option for Shared-Data If 
SHOW PARTITIONS
shows a version gap that you cannot fill by skipping/aborting and the FE restart didn't help, the metadata sequence for that table is likely broken. 1. Backup the data (if possible via 
INSERT INTO ... SELECT
). 2. Drop the table with FORCE: 
DROP TABLE <table_name> FORCE;
. 3. Recreate the table. In Shared-data mode, this is often faster than surgical metadata repair when the version sequence gets corrupted. Recommended next step: Share the output of 
SHOW PARTITIONS FROM <table_name>
and the top few rows of 
SHOW PROC '/transactions'
(sorted by TxnID) if you need to identify the specific blocker. References • loading/SQL_transaction.md • knowledge/trouble_shooting/fe_deadlock_problems.md
t
@Rocky all stuck transactions are in COMMITTED state
r
If the transactions are still in the 
COMMITTED
state after running the 
SKIP
command and restarting the FE Leader, it means the Publish Version Daemon is unable to advance the version pointer for that table. In StarRocks v4.1.4 (Lake mode), this is almost always caused by a "hidden" transaction or a specific metadata gap on S3. Here is the surgical approach to unstick it: 1. Identify the "Hole" in the Version Chain StarRocks cannot publish version $N$ if version $N-1$ is missing. * Run: 
SHOW PARTITIONS FROM <table_name>;
* Look at the 
VisibleVersion
 and 
NextVersion
. * Example: If 
VisibleVersion
is 500 and the first stuck transaction in 
SHOW PROC '/transactions'
is for version 505, there is a gap of versions 501, 502, 503, and 504. * Action: You must find out what happened to those missing versions. They might be Compaction transactions that don't always appear in your filtered load view. 2. Check for "LAKE_COMPACTION" Transactions In shared-data clusters, compactions are treated as transactions. If a compaction transaction fails or hangs, it blocks all subsequent 
INSERT_STREAMING
data from becoming visible. * Run: 
SHOW PROC '/transactions';
* Look for any row where the 
Label
 starts with 
COMPACTION_
or the 
TransactionId
 is smaller than your stuck load transactions. * Action: If you find a stuck compaction transaction, apply the skip command to it:
Copy code
sql
    ADMIN SKIP COMMITTED TRANSACTION <txn_id>;
3. Verify the FE Leader and Configuration * Ensure you are on the Leader FE: Only the Leader FE processes the Publish queue. Run 
SHOW FRONTENDS;
and check the 
IsLeader
column. Run your commands only on that node. * Check the Skip Toggle: Ensure the "escape hatch" wasn't reset during the restart:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_admin_skip_committed_txn';
    -- Must be true
4. Check for S3 Metadata Errors (The "Missing Log" Issue) If the metadata files (tablet logs) for a transaction were accidentally deleted from S3 (e.g., by an aggressive lifecycle policy or manual cleanup), the FE will hang forever trying to read them to "publish" the version. * Action: Search the FE Leader 
fe.log
or 
fe.warn.log
for the word 
publish
 or your 
TxnID
. * Look for this error: 
fail to get tablet meta from remote storage
or 
No such file or directory
. * Resolution: If the metadata is gone from S3, the version chain is permanently broken. The 
ADMIN SKIP
command should handle this by creating a "No-Op" version, but if it fails to clear after a restart, the table's internal metadata state is likely corrupted. 5. Final Recovery (If the above fails) If 
SHOW PARTITIONS
shows that the 
VisibleVersion
is not advancing despite no other transactions being visible in 
SHOW PROC
, the metadata sequence is likely inconsistent. Option A (Non-Destructive): Force a metadata refresh by manually triggering a small load (e.g., 
INSERT INTO table VALUES (...)
) and see if it also gets stuck. If it does, it confirms the "gate" is closed at a specific version. Option B (The Reliable Fix): Since it's an S3-backed table, the most reliable way to clear a corrupted version chain in v4.1.4 is: 1. 
CREATE TABLE <table_name_new> AS SELECT * FROM <table_name_stuck> WHERE 1=0;
2. 
INSERT INTO <table_name_new> SELECT _ FROM <table_name_stuck>;
(Note: This only works for data that is _already* 
VISIBLE
). 3. 
ALTER TABLE <table_name_stuck> RENAME <table_name_old>;
4. 
ALTER TABLE <table_name_new> RENAME <table_name_stuck>;
5. 
DROP TABLE <table_name_old> FORCE;
Summary recommendation: Run 
SHOW PARTITIONS
and compare the 
VisibleVersion
to the `TxnId`s in 
SHOW PROC
. If there is any ID smaller than your stuck loads, skip that one first. If there are no smaller IDs, check the 
fe.warn.log
for the specific reason the 
PublishVersionDaemon
is rejecting the skip. References
t
@Rocky transactions are still stuck after dropping the table
r
If you have already dropped the table and restarted the FE, but the transactions are still showing as 
COMMITTED
in 
SHOW PROC '/transactions'
, you are likely dealing with "orphaned" transactions in the Global Transaction Manager (GTM). In StarRocks v4.1.x, this happens because the transaction metadata is stored in the FE's memory and edit logs, and it doesn't always automatically purge when the associated table is deleted—especially if the transaction was already in the 
COMMITTED
state (waiting for publish) when the drop occurred. Why they are still there When a table is dropped, the metadata is moved to the Recycle Bin (unless you used 
FORCE
). The GTM keeps these transactions alive in case you 
RECOVER
the table. If you used 
FORCE
, the table is gone, but the GTM may still hold the transaction object because the "Publish" thread is now failing to find the table it's supposed to update, creating a loop. How to clear them 1. Check the Recycle Bin If you didn't use 
FORCE
to drop the table, it is likely still sitting in the FE's recycle bin. The GTM will not abandon transactions for a table that can still be recovered. * Check the bin: 
SHOW PROC '/catalog_recycle_bin';
* Action: If your table is there, you can either: * 
RECOVER TABLE <table_name>;
it, then try the 
ADMIN SKIP
command again now that the table metadata is active. * Wait for the 
catalog_trash_expire_second
(default 24 hours) to pass, after which the FE will physically delete the table and eventually purge the transactions. 2. Manual "Surgical" Abort If the table is already 
FORCE
dropped or you cannot wait for the recycle bin to clear, you need to force the GTM to forget these transactions. Since the table is gone, the 
ADMIN SKIP
command might fail because it can't find the table's version sequence to advance. * Action: Try to manually abort them (even though they are 
COMMITTED
):
Copy code
sql
    -- Try rollback first (sometimes works on orphaned COMMITTED in 4.x)
    ROLLBACK TRANSACTION <txn_id>;
* Note: If 
ROLLBACK
fails with "transaction is already committed", and the table is gone, these are truly orphaned. 3. Clearing the FE Image (The "Last Resort") If the transactions persist through restarts and are cluttering your monitoring, they are persisted in the FE BDBJE edit logs. 1. Check the `TransactionId`s. 2. If they are not blocking new loads into new tables (which they shouldn't, as TxnIDs are global and will just increment), you can technically ignore them. They will eventually be rotated out of the `TransactionMgr`'s history buffer. 3. If they must be removed (e.g., they are causing FE memory pressure or log spam): * You would need to perform a clean metadata restart, but this is highly dangerous and usually only done under support guidance. * Safest approach: Recreate the table with a different name and continue operations. The orphaned transactions for the old table ID will eventually time out and be purged by the FE's 
TransactionCleaner
thread (governed by 
label_keep_max_second
, default is usually 3 days for finished/old txns). Summary Checklist 1. Did you use 
DROP TABLE ... FORCE
? If not, the transactions stay until the table leaves the recycle bin. 2. Are you on the FE Leader? Ensure you are running 
SHOW PROC
and any skip commands on the actual Leader. 3. Check 
fe.log
for the specific TxnIDs. If you see 
Table <id> not found
errors repeating, it confirms they are orphaned. If these transactions are blocking new loads on a new table with the same name, please let me know, as that indicates a different issue with the namespace lock in the FE. References • loading/SQL_transaction.md • faq/shared_data_faq.md
t
@Rocky after waiting the transactions have cleared for those dropped tables, but I still have 8 stuck INSERT_STREAMING transactions
r
If the remaining 8 
INSERT_STREAMING
transactions are still stuck in 
COMMITTED
despite your previous attempts, it means the Version Chain for the associated table(s) is broken. In StarRocks 4.1.x (Shared-data/Lake mode), a transaction cannot move from 
COMMITTED
to 
VISIBLE
if any previous version number is missing or stuck. Follow these steps to diagnose the specific blocker: 1. Identify the "Queue Blocker" Find the oldest Transaction ID among the 8 stuck ones. * Run: 
SHOW PROC '/transactions';
and sort by 
TransactionId
. * Note the 
TransactionId
 and the 
Table
 name of the oldest row. 2. Check for a Version Gap For that specific table, check if there is a gap between what is currently visible and what is trying to be published. * Run: 
SHOW PARTITIONS FROM <table_name>;
* Compare 
VisibleVersion
 and 
NextVersion
. * If 
VisibleVersion
= 100 but your stuck transaction is for Version 105, there are 4 missing versions (101-104) blocking the way. * These missing versions are likely Compaction or Schema Change tasks that failed but didn't clean up their "reservation" of the version number. 3. Search the FE Leader Log for the specific TxnID The FE Leader log will tell you exactly why it is refusing to publish that transaction. * Search your 
fe.log
or 
fe.warn.log
 for the specific 
TxnID
found in Step 1. * Common Errors to look for: * `fail to get tablet meta from remote storage`: A metadata file is missing from S3/Object Storage. * `rejected from agent-task-pool`: The thread pool for publishing is full. * `wait for previous version to be visible`: Confirms a version gap. 4. Check for Metadata Locks In v4.1.x, a metadata lock might be held by a background process (like an aborted Schema Change). * Run:
Copy code
sql
    SELECT * FROM sys.fe_locks WHERE hold_time_ms > 10000;
* If you see a lock on that specific table held for a very long time, it is blocking the 
PublishVersionDaemon
. 5. Surgical Fixes Depending on what you find above, try these in order: * A. Skip the oldest TxnID specifically: Instead of a general skip, target the oldest ID:
Copy code
sql
    -- Ensure this is enabled
    ADMIN SET FRONTEND CONFIG ("enable_admin_skip_committed_txn" = "true");
    -- Skip the specific oldest ID
    ADMIN SKIP COMMITTED TRANSACTION <oldest_txn_id>;
* B. Check for Stuck Compactions: Even if they don't show as 
INSERT_STREAMING
, they consume versions.
Copy code
sql
    SHOW PROC '/compactions';
If any row has an 
Error
or is very old without a 
FinishTime
, use 
ADMIN SKIP
on that compaction's 
TxnID
. * C. Toggle Batch Publish (Temporary): If the FE is struggling to coordinate many small publishes:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
Wait 1 minute. If the transactions clear, you can re-enable it (
true
). Recommendation: If you find the 
TxnID
in the 
fe.warn.log
and it says 
tablet meta not found
, it is a confirmed metadata loss on S3. In that case, 
ADMIN SKIP
on that specific ID is the only way to advance the version. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md • administration/management/compaction.md
m
Hey @Travis (YaaaDingus), nice to meet you — I'm Head of Global Solutions at PhoenixAI. On the stuck transactions: I don't have a clean fix for you. But for what it's worth, COMMITTED transactions that survive a leader FE restart and don't move even with 
NoOpPublish: true
set aren't something you're doing wrong, that looks like a bug. If you haven't already, I'd open it at github.com/StarRocks/StarRocks/issues with your version (4.1.4), the 
SHOW PROC '/transactions'
output, and the fe.log lines around one of the stuck txn IDs. That'll get maintainer eyes on it much faster than this channel will. What I do want to weigh in on is how you got here, because I think there's a bigger fix available. 61 routine load jobs is, to my read, the actual root cause behind the OOM. The data itself goes to the BEs, but all the bookkeeping is on the FE: job state, task scheduling, Kafka offsets, and a transaction per task per batch, in memory and in the edit log. That's what your FEs were carrying when they ran out of memory, and the OOM is what left you with committed-but-never-published transactions. So yes, bigger FEs (memory especially) and you should do that regardless. But it doesn't change the shape of the problem: across the POCs I've run, routine load doesn't scale with the cluster the way people expect. You add BEs and the bottleneck stays on the FE side. What I'd do instead is move those 61 topics onto the Kafka connector: docs.starrocks.io/docs/loading/Kafka-connector-starrocks Same direction of data, different place the work happens. Instead of SR polling Kafka on its own scheduler, Kafka Connect consumes the topics and pushes into SR via Stream Load. You scale by adding Connect workers rather than hoping the FE keeps up, and you get real control over batch size and parallelism per topic — so you're committing far fewer, larger transactions, and none of the job scheduling or offset tracking lives inside SR anymore. Two trade-offs worth knowing before you commit: • you're now operating a Kafka Connect cluster (distributed mode if you want that scaling) • the connector is at-least-once, where routine load gives you exactly-once. With primary key tables and a sensible key that's a non-issue, but it's a real difference. I hope that helps, and if you open an issue paste the link here and I will escalate internally to my R&D
j
Maybe you should find out which is the earliest transaction that blocked the others ,as @Moreno Garcia said, 
SHOW PROC '/transactions'
will help you to find it
🫶 1
t
@Moreno Garcia just seeing this now, this is something I've considered. For my team it's a lot easier to programatically create routine loads rather than manually configure many things by hand and manage the configuration roll out separately. Would scaling FE nodes be a feasible option? If there possibility that this scales better in the future?
2 Views
Open in Slack
PreviousNext
Powered by