@Rocky WHen I have elevated transaction failure/reject rate, how do I check reason? I remember there was a command that lets me see ErrMsg column for the transactions

To check the reason for elevated transaction failures or rejections in StarRocks, you can use several commands depending on the type of operation. The specific command you are likely remembering that features an

ErrorMsg

or

Reason

column is either

SHOW LOAD

or

SHOW TRANSACTION

. 1. For Load Jobs (Broker, Spark, Insert) If you are running load jobs like

INSERT INTO

or Broker Load, use:

sql
SHOW LOAD [FROM db_name] [WHERE LABEL = "your_label"];

* Column to check:

ErrorMsg

. * This column provides the detailed reason (e.g.,

ETL_QUALITY_UNSATISFIED

,

TIMEOUT

, etc.) when a job's state is

CANCELLED

. 2. For Specific Transactions (Stream Load, etc.) If you have a specific Transaction ID (often found in logs or API responses), use:

sql
SHOW TRANSACTION [FROM db_name] WHERE ID = <transaction_id>;

* Column to check:

Reason

. * This field contains the error message explaining why the transaction failed or was aborted. 3. Checking Running/Finished Transactions (System Proc) If you want to see a list of recent transactions for a database to spot rejections or slowdowns, you can use the system

PROC

interface:

sql
-- First, get the Database ID
SHOW PROC '/dbs';

-- Then, view transactions for that DB (replace <db_id> with the actual ID)
SHOW PROC '/transactions/<db_id>/running';
SHOW PROC '/transactions/<db_id>/finished';

* Note: In some versions and specific table types (like Primary Key tables), the

Reason

or

ErrMsg

column in this output will display messages like "Partition's compaction score is larger than 100.0, delay commit," which indicates ingestion pressure or rejection. 4. For Stream Load History If you are using Stream Load and want to query the history via SQL (available in recent versions), you can use:

sql
SHOW STREAM LOAD [FROM db_name] [WHERE LABEL = "your_label"];

* Note: This requires the BE configuration

enable_stream_load_record

to be set to

true

(it is

false

by default in older versions). Summary of Tips: * Check FE/BE Logs: If the SQL commands don't provide enough detail, search the Leader FE

fe.log

or the BE

<http://be.INFO|be.INFO>

logs for the

TransactionId

. Search for the keyword

ABORTED

. * Error URL: For Stream Load failures, the synchronous HTTP response usually contains an

ErrorURL

. You can curl that URL to see the specific rows that caused the rejection. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md

@Rocky

W20260921 19:38:10.825872 22668194330176 agent_task.cpp:388] local tablet migration failed. status: Internal error: could not migration because has unfinished txns., signature: 18924578

What does this indicate

This error indicates that a local tablet migration task failed on a Backend (BE) node. In StarRocks, local migration occurs when the BE tries to move a tablet from one local disk to another (for example, to balance disk usage when one disk is getting full). The specific reason,

could not migration because has unfinished txns

, means the migration was aborted because there are active transactions (e.g., data loading, compactions, or schema changes) currently operating on that tablet. What this means for your cluster: 1. Safety Mechanism: This is a safety check. StarRocks avoids moving tablet data while it is being actively written to or modified to prevent data corruption or lost updates. 2. Ingestion Pressure: If you see this frequently alongside high transaction failure rates, it suggests your tablets are under constant write pressure, leaving no "quiet window" for the BE to perform background maintenance like disk balancing. 3. Transient vs. Persistent: * If it only happens occasionally, it is usually a transient warning; the BE will retry the migration automatically later. * If it is persistent, it can lead to disk skew (one disk filling up while others are empty). How to investigate the "Unfinished Txns": Since you mentioned elevated transaction failures, you should check which transactions are hanging or blocking the system: 1. Find the Tablet ID: The log snippet usually includes a

signature

, which often corresponds to the Tablet ID. You can find which table this belongs to using:

sql
    SHOW TABLET <tablet_id>;

2. Check for Long-Running Transactions: Search for transactions that haven't finished for a long time:

sql
    -- View running transactions for a database
    SHOW PROC '/transactions/<db_id>/running';

Look for transactions with an old

CreateTime

. If a transaction is stuck in the

PREPARE

or

COMMITTED

state but not

VISIBLE

, it might be blocking maintenance tasks. 3. Check Compaction Status: Sometimes "unfinished txns" refer to background compactions. Check if compaction is stuck or falling behind:

sql
    SHOW PROC '/compactions';

4. Check BE Disk Usage: Verify if one of your disks is nearly full, which would explain why the BE is aggressively trying (and failing) to migrate tablets:

sql
    SHOW BACKENDS\G

Look at the

DiskStat

or

Capacity

columns. Recommendation If your transaction rejection rate is high, focus on solving the load failures first. Once the ingestion stabilizes and transactions finish successfully, the "local tablet migration" tasks will eventually succeed on their own during a period of lower activity. References • knowledge/admin_and_operation/shared_data_compaction.md • knowledge/trouble_shooting/fe_deadlock_problems.md

@Rocky Is exclusive_cpu_cores number controlling number of vCPU per node a resource group can use?

Yes,

exclusive_cpu_cores

controls the number of CPU cores (vCPUs) allocated to a resource group per BE node. Here is how it works: 1. Per-Node Limit The value you set for

exclusive_cpu_cores

is applied to each individual Backend (BE) node in your cluster. For example, if you set

exclusive_cpu_cores = 4

for a resource group, that group is allowed to use exactly 4 cores on BE1, 4 cores on BE2, and so on. 2. Hard Limit (Exclusive & Quota) Unlike

cpu_weight

(which is a soft limit/relative weight),

exclusive_cpu_cores

acts as a hard limit: * Exclusive: These cores are reserved specifically for this group. Other groups cannot use them even if this group is idle (unless

enable_resource_group_cpu_borrowing

is set to

true

). * Quota: This group is strictly limited to the number of cores specified. It cannot "burst" to use extra idle capacity from other groups. 3. Constraints * Upper Bound: The sum of

exclusive_cpu_cores

across all resource groups must be less than the total cores on your smallest BE node (specifically

min_be_cpu_cores - 1

). You must leave at least 1 core for the system/shared tasks. * Mutual Exclusion: For a single resource group, you can set either

cpu_weight

OR

exclusive_cpu_cores

, but not both. * Version: Hard limits via

exclusive_cpu_cores

were fully introduced/refined in version 3.3.5+. How to verify You can see how many cores are bound to the threads of an exclusive resource group by querying the system view:

sql
SELECT BE_ID, NAME, BOUND_CPUS 
FROM information_schema.be_threads 
WHERE NAME LIKE '%<resource_group_id>%';

This will show the specific thread pools (

pip_exec

,

pip_scan

) created for that resource group and the number of cores they are restricted to on. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md • best_practices/audit_log_resource_group.md

@Rocky How do I add observer and follower FE nodes to the cluster? Is it similar to adding a backend node?

Adding Follower and Observer FE nodes is conceptually similar to adding a Backend (BE) node, but there are a few critical differences in the startup command and how they handle metadata. 1. The SQL Command Just like you use

ADD BACKEND

, you use

ALTER SYSTEM

to register the new FE. The role you choose determines if the node can participate in elections (Follower) or only serve read-only queries (Observer). * To add a Follower:

sql
    ALTER SYSTEM ADD FOLLOWER "fe_host:9010";

* To add an Observer:

sql
    ALTER SYSTEM ADD OBSERVER "fe_host:9010";

_Note: Use the

edit_log_port

(default 9010), not the query port._ 2. The Startup Command (The Main Difference) The biggest difference from adding a BE is the

--helper

flag. When starting a new FE for the first time, it must "check in" with the current Leader to download the metadata image. * Start the new FE:

bash
    ./fe/bin/start_fe.sh --helper leader_fe_host:9010 --daemon

* Crucial: You only need the

--helper

flag the first time you start the node. For subsequent restarts, just use

./start_fe.sh --daemon

. * BE nodes do not require a helper flag; they just need the FE to be aware of them. 3. Key Differences vs. Adding a BE Feature | FE (Follower/Observer) | BE (Backend) SQL Command |

ALTER SYSTEM ADD FOLLOWER/OBSERVER

|

ALTER SYSTEM ADD BACKEND

Port in SQL |

edit_log_port

(9010) |

heartbeat_service_port

(9050) First Startup | Requires

--helper <leader_ip>:9010

| No helper flag needed Configuration | Needs

meta_dir

in

fe.conf

| Needs

storage_root_path

in

be.conf

HA Impact | Adding Followers increases fault tolerance. | Adding BEs increases storage/compute capacity. 4. Verification After starting the node, check its status to ensure

Alive

is

true

and the

Role

is correct:

sql
SHOW PROC '/frontends';

Important Tips: * Networking: Ensure

priority_networks

is configured in

fe.conf

on the new node, especially if the machine has multiple network interfaces. * Quorum: For a High Availability (HA) cluster, you typically want exactly 3 Followers. Adding more Followers can slow down write performance (since more nodes must acknowledge the edit log), so use Observers if you just need to scale out read capacity. References • deployment/deploy_manually.md • administration/Meta_recovery.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md References • deployment/deploy_manually.md • administration/Meta_recovery.md

@Rocky The architecture is always 1 leader, 1 follower, and then 1 observer right

Not quite. For a production High Availability (HA) setup, the standard architecture is 3 Follower nodes (one of which is elected as the Leader). Here is the breakdown of how the roles actually work: 1. The "Follower" Quorum (Standard: 3 Nodes) To ensure the cluster can survive a node failure, you need a quorum (more than half) of Follower nodes to be alive. * 3 Followers: Can survive 1 node failure ($3 / 2 = 1.5$, so 2 nodes must be up). * 1 Leader + 2 Followers: This is the most common HA setup. All three are technically "Followers," but one is elected "Leader." * Why not 1 Leader + 1 Follower? If you only have 2 Follower nodes total and one goes down, you lose quorum ($2 / 2 = 1$, so you'd need 2 nodes up to elect a leader), and the cluster will become read-only or stop functioning. 2. The "Observer" Role (Optional) Observers are not part of the election quorum. * Purpose: They are used strictly to scale out query (read) performance. * Advantage: You can add as many Observers as you want without slowing down the metadata synchronization (the Leader only waits for a majority of Followers to acknowledge a write, not Observers). * Standard Setup: You usually only add Observers once your 3 Followers can no longer handle the query volume. Summary of Differences Feature | Follower (incl. Leader) | Observer Participates in Election? | Yes | No Counted for Quorum? | Yes | No Can become Leader? | Yes | No Handles Read Queries? | Yes | Yes Handles Write Requests? | Only the Leader | No (forwards them to Leader) Correct mental model for HA: * Minimum HA: 3 Followers (1 becomes Leader). * Scaling Reads: 3 Followers + $N$ Observers. References • deployment/plan_cluster.md • introduction/Architecture.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md References • deployment/plan_cluster.md • deployment/deploy_manually.md • faq/operation_maintenance_faq.md • introduction/Architecture.md

@Rocky How do i remove the observer and add it back as a follower

To change an Observer FE into a Follower FE, you cannot simply "switch" the role in a configuration file while it is running. You must remove it from the cluster and re-add it as a new node. Here is the standard, safest procedure: Step 1: Remove the Observer from the cluster From a MySQL client connected to your Leader FE, drop the observer node:

sql
ALTER SYSTEM DROP OBSERVER "observer_host:9010";

Step 2: Stop and clean the Observer node On the machine where the Observer is running: 1. Stop the FE process:

bash
    ./fe/bin/stop_fe.sh

2. Clear the metadata directory: Locate the

meta_dir

(defined in your

fe/conf/fe.conf

). You must delete or rename this directory to ensure the node starts with a "clean slate" and doesn't try to reuse its old Observer-specific metadata.

bash
    # Example: if your meta_dir is /data/starrocks/fe/meta
    rm -rf /data/starrocks/fe/meta/*

Step 3: Add the node back as a Follower Go back to your MySQL client (connected to the Leader FE) and register the node as a Follower:

sql
ALTER SYSTEM ADD FOLLOWER "host:9010";

Step 4: Restart the node with the Helper flag On the machine you just cleaned, start the FE. Because it is now a "new" node in the eyes of the cluster, you must use the

--helper

flag to synchronize metadata from the Leader:

bash
./fe/bin/start_fe.sh --helper leader_host:9010 --daemon

Step 5: Verify the change Run the following command to ensure the node is now listed with the role

FOLLOWER

and is `Alive`:

sql
SHOW PROC '/frontends';

Why do I have to clear the metadata? The

meta_dir

contains a

ROLE

file and BDB JE environment logs that explicitly identify the node as an Observer. If you don't clear this directory, the FE may fail to start or conflict with the cluster's leader when it tries to join as a Follower. Clearing the directory forces the node to download a fresh, correct metadata image from the Leader. References • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • deployment/deploy_manually.md • administration/Meta_recovery.md References • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md

@Rocky STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin $ ./stop_fe.sh STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin $ ./start_fe.sh --helper 172.30.75.62:9010 --daemon Infered JAVA_HOME=/usr/lib/jvm/java-24-amazon-corretto.x86_64 Frontend running as process 167150. Stop it first. STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin 1 $

The error "Frontend running as process 167150. Stop it first" happens because the

stop_fe.sh

script either hasn't finished shutting down the process yet, or the process is stuck. The StarRocks start script checks for a

.pid

file and verifies if that process is still active in the system. Follow these steps to fix this and ensure the role change (Observer to Follower) works: 1. Force kill the existing process The standard stop script sends a SIGTERM, which can take time. If it's lingering, kill it manually:

bash
# Check if it's still there
ps -ef | grep StarRocksFE

# Kill the specific PID mentioned in your error
kill -9 167150

2. IMPORTANT: Clear the Metadata Since you are changing this node from an Observer to a Follower, you must clear the

meta

directory before starting it again. If you don't, it will try to start using its old identity.

bash
# Find your meta_dir path in fe/conf/fe.conf
# Then clear it (replace path with your actual meta_dir)
rm -rf /opt/starrocks/fe/meta/*

3. Verify the Drop/Add on the Leader Ensure you have already executed the SQL commands on the Leader node:

sql
-- Run these on the Leader FE via MySQL client
ALTER SYSTEM DROP OBSERVER "172.30.67.218:9010";
ALTER SYSTEM ADD FOLLOWER "172.30.67.218:9010";

4. Restart with Helper Now that the process is dead and the metadata is clean, run your start command again:

bash
./start_fe.sh --helper 172.30.75.62:9010 --daemon

Why did this happen? 1. Shutdown lag:

stop_fe.sh

returns control to the terminal quickly, but the Java process often takes 5–10 seconds to flush logs and close the BDB environment. 2. Stale PID file: If the process crashed or was interrupted, a

fe.pid

file might remain in the

bin

or

meta

directory, leading the script to think it's still running. References • deployment/deploy_manually.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md References • deployment/deploy_manually.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md

@Rocky I keep getting Connect time out ErrorMsg when i do

show frontends

A "Connect time out" in the

ErrMsg

column of

SHOW FRONTENDS

means the Leader FE cannot communicate with the new node via its heartbeat/RPC ports. Since you just switched this node from an Observer to a Follower, here is how to troubleshoot this: 1. Verify Connectivity (The #1 Cause) The Leader FE attempts to "ping" the new Follower on its EditLogPort (9010) and RpcPort (9020). If these are blocked, you will see a timeout. Run these commands from the Leader FE machine to see if it can reach the new Follower:

bash
# Test EditLog port
nc -zv 172.30.67.218 9010

# Test RPC port
nc -zv 172.30.67.218 9020

If these fail, check your AWS Security Groups or OS firewall (iptables/firewalld) to ensure the FEs can talk to each other on ports 9010, 9020, 9030, and 8030. 2. Check the FE Logs On the node you are trying to add (

172.30.67.218

), look at the end of the logs to see why it might be failing to join the quorum:

bash
tail -n 100 /opt/starrocks/fe/log/fe.log
# OR
tail -n 100 /opt/starrocks/fe/log/fe.warn.log

Look for these specific errors: * `rep.UnknownMasterException`: The node can't find the Leader. * `priority_networks`: The node is binding to the wrong IP address (common if the machine has multiple network interfaces). * `insufficient memory`: The Java process is crashing immediately. 3. Verify Bind Address (

priority_networks

) If your machine has multiple IPs, the FE might be listening on an interface that the Leader can't reach. 1. Open

fe/conf/fe.conf

. 2. Check if

priority_networks

is set. It should match the subnet of your target IP:

properties
   priority_networks = 172.30.67.0/24

3. If you change this, you must restart the FE. 4. Ensure the Process is Actually Listening On the new Follower node, verify the ports are open and bound to the correct IP:

bash
netstat -ntpl | grep 9010

If you don't see the process listening, it likely crashed during startup. Re-check

fe.log

. 5. Common "Stuck" State Fix If you didn't clear the metadata or if the Leader still thinks the old Observer exists: 1. On Leader:

ALTER SYSTEM DROP FOLLOWER "172.30.67.218:9010";

2. On Follower: Stop FE ->

rm -rf /opt/starrocks/fe/meta/*

3. On Leader:

ALTER SYSTEM ADD FOLLOWER "172.30.67.218:9010";

4. On Follower: Start FE with

--helper

. References • deployment/deploy_manually.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • faq/Others.md • administration/Meta_recovery.md

@Rocky what do you mean priority networks should match subnet of target IP? Is leader IP my target IP?

In this context, "Target IP" refers to the local IP address of the node you are currently configuring. It is not the Leader's IP.

priority_networks

is a setting in

fe.conf

that tells each individual FE node: "If this machine has multiple network interfaces (like a public IP, a private IP, and a docker bridge), only use the IP that matches this pattern." How to set it for your specific node (

172.30.67.218

) On the machine

STARROCKS-CORE-SERVICES-FE-2

(the one you want to be a Follower): 1. Open

/opt/starrocks/fe/conf/fe.conf

. 2. Find the

priority_networks

line. 3. Set it to a CIDR range that covers its own IP (

172.30.67.218

):

properties
    # This tells THIS node to bind to its 172.30.67.x interface
    priority_networks = 172.30.67.0/24

Why this matters If you don't set this and the machine has multiple IPs, StarRocks might randomly pick the "wrong" one (like a local loopback or a virtual bridge IP). If the Follower starts up and thinks its IP is

127.0.0.1

or

172.17.0.1

, the Leader (at

172.30.75.62

) will never be able to connect to it, resulting in the Connect time out error you see. How to verify what IP the node is using On the Follower node, run this command to see which IP it actually grabbed when it started:

bash
grep "IP:" /opt/starrocks/fe/log/fe.log

* If it shows anything other than

172.30.67.218

, your

priority_networks

is either missing or wrong. * If it shows

172.30.67.218

but you still get a timeout, then the issue is likely a firewall/Security Group blocking ports 9010 or 9020 between the two machines. Summary of IPs in

fe.conf

*

priority_networks

: Matches the local IP of the machine you are on. *

--helper

(in start command): Points to the Leader's IP. References • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md

@Rocky # Frontend service configurations frontend_address = 0.0.0.0 priority_networks = 172.30.64.0/19 So this wouldnt work for local ip 172.30.67.218

Actually, that subnet range DOES include your IP. The CIDR

172.30.64.0/19

covers all IP addresses from 172.30.64.0 to 172.30.95.255. Since your IP is

172.30.67.218

, it falls within that range. Why you might still have a "Connect time out" Even if the range is mathematically correct, there are two common reasons for failure: 1. Multiple Matching IPs: If that specific machine has other network interfaces (like a Docker bridge, a VPN, or a secondary ENI) that also start with

172.30.x.x

, StarRocks might be picking one of those instead of the one you want. StarRocks picks the first one it finds that matches the pattern. 2. The

0.0.0.0

setting: While

frontend_address = 0.0.0.0

tells the FE to listen on all interfaces, the Leader FE needs to know exactly which specific IP to use for heartbeats. If the Follower reports the wrong IP to the Leader, the Leader will try to connect to an unreachable address. How to verify and fix it Step 1: Check what IP the FE actually picked Run this on the Follower node:

bash
grep "IP:" /opt/starrocks/fe/log/fe.log

If the output shows anything other than

172.30.67.218

, then

priority_networks

is picking the wrong interface. Step 2: Make the configuration more specific To be safe, narrow the range so only your desired IP can match. A

/24

or even a

/27

is usually better:

properties
# Change this in fe.conf
priority_networks = 172.30.67.0/24

Step 3: Check the Leader's view On your Leader node, run:

sql
SHOW FRONTENDS;

Look at the

IP

column for your new node. If that IP does not match

172.30.67.218

, the Follower joined using the wrong identity. If the IP in

SHOW FRONTENDS

is wrong: 1. Leader:

ALTER SYSTEM DROP FOLLOWER "wrong_ip:9010";

2. Follower: Stop FE ->

rm -rf /opt/starrocks/fe/meta/*

3. Follower: Fix

fe.conf

(

priority_networks = 172.30.67.0/24

). 4. Leader:

ALTER SYSTEM ADD FOLLOWER "172.30.67.218:9010";

5. Follower: Start FE with

--helper

. References • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md

@Rocky STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin 130 $ nc -zv 172.30.67.218 9010 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 172.30.67.218:9010. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin $ nc -zv 172.30.67.218 9020 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 172.30.67.218:9020. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe bin $ tail -n 100 /opt/starrocks/fe/log/fe.log 2026-04-21 211238.141Z ERROR (TableKeeper|132) [SimpleExecutor.executeDDL():150] RepoExecutorexecute DDL error: CREATE TABLE IF NOT EXISTS loads_history (id bigint, label varchar(2048), profile_id varchar(2048), db_name varchar(2048), table_name varchar(2048), user varchar(2048), warehouse varchar(2048), state varchar(2048), progress varchar(2048), type varchar(2048), priority varchar(2048), scan_rows bigint, scan_bytes bigint, filtered_rows bigint, unselected_rows bigint, sink_rows bigint, runtime_details json, create_time datetime, load_start_time datetime, load_commit_time datetime, load_finish_time datetime not null, properties json, error_msg varchar(2048), tracking_sql varchar(2048), rejected_record_path varchar(2048), job_id bigint ) PARTITION BY date_trunc('DAY', load_finish_time) DISTRIBUTED BY HASH(label) BUCKETS 3 PROPERTIES( 'replication_num' = '1', 'partition_live_number' = '30') com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.system.SystemInfoService.checkClusterCapacity(SystemInfoService.java:1309) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:933) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:888) ~[starrocks-fe.jar:?] at com.starrocks.server.MetadataMgr.createTable(MetadataMgr.java:303) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.lambda$visitCreateTableStatement$4(DDLStmtExecutor.java:307) ~[starrocks-fe.jar:?] at com.starrocks.common.ErrorReport.wrapWithRuntimeException(ErrorReport.java:118) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:306) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:212) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.CreateTableStmt.accept(CreateTableStmt.java:344) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:107) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor.execute(DDLStmtExecutor.java:197) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:146) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.createTable(TableKeeper.java:96) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.run(TableKeeper.java:68) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper$TableKeeperDaemon.runAfterCatalogReady(TableKeeper.java:222) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] 2026-04-21 211238.141Z ERROR (TableKeeper|132) [TableKeeper.run():76] error happens in Keeper: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. java.lang.RuntimeException: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:151) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.createTable(TableKeeper.java:96) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.run(TableKeeper.java:68) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper$TableKeeperDaemon.runAfterCatalogReady(TableKeeper.java:222) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] Caused by: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.system.SystemInfoService.checkClusterCapacity(SystemInfoService.java:1309) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:933) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:888) ~[starrocks-fe.jar:?] at com.starrocks.server.MetadataMgr.createTable(MetadataMgr.java:303) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.lambda$visitCreateTableStatement$4(DDLStmtExecutor.java:307) ~[starrocks-fe.jar:?] at com.starrocks.common.ErrorReport.wrapWithRuntimeException(ErrorReport.java:118) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:306) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:212) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.CreateTableStmt.accept(CreateTableStmt.java:344) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:107) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor.execute(DDLStmtExecutor.java:197) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:146) ~[starrocks-fe.jar:?] ... 5 more 2026-04-21 211238.141Z ERROR (TableKeeper|132) [SimpleExecutor.executeDDL():150] RepoExecutorexecute DDL error: CREATE TABLE predicate_columns( fe_id STRING NOT NULL,db_id BIGINT NOT NULL,table_id BIGINT NOT NULL,column_id BIGINT NOT NULL,usage STRING NOT NULL,last_used DATETIME NOT NULL,created DATETIME DEFAULT CURRENT_TIMESTAMP) PRIMARY KEY(fe_id, db_id, table_id, column_id) DISTRIBUTED BY HASH(fe_id, db_id, table_id, column_id) BUCKETS 8 PROPERTIES('replication_num'='1') com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.system.SystemInfoService.checkClusterCapacity(SystemInfoService.java:1309) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:933) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:888) ~[starrocks-fe.jar:?] at com.starrocks.server.MetadataMgr.createTable(MetadataMgr.java:303) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.lambda$visitCreateTableStatement$4(DDLStmtExecutor.java:307) ~[starrocks-fe.jar:?] at com.starrocks.common.ErrorReport.wrapWithRuntimeException(ErrorReport.java:118) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:306) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:212) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.CreateTableStmt.accept(CreateTableStmt.java:344) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:107) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor.execute(DDLStmtExecutor.java:197) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:146) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.createTable(TableKeeper.java:96) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.run(TableKeeper.java:68) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper$TableKeeperDaemon.runAfterCatalogReady(TableKeeper.java:222) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] 2026-04-21 211238.141Z ERROR (TableKeeper|132) [TableKeeper.run():76] error happens in Keeper: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. java.lang.RuntimeException: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:151) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.createTable(TableKeeper.java:96) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper.run(TableKeeper.java:68) ~[starrocks-fe.jar:?] at com.starrocks.scheduler.history.TableKeeper$TableKeeperDaemon.runAfterCatalogReady(TableKeeper.java:222) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] Caused by: com.starrocks.sql.analyzer.SemanticException: Getting analyzing error. Detail message: Cluster has no available capacity. at com.starrocks.system.SystemInfoService.checkClusterCapacity(SystemInfoService.java:1309) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:933) ~[starrocks-fe.jar:?] at com.starrocks.server.LocalMetastore.createTable(LocalMetastore.java:888) ~[starrocks-fe.jar:?] at com.starrocks.server.MetadataMgr.createTable(MetadataMgr.java:303) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.lambda$visitCreateTableStatement$4(DDLStmtExecutor.java:307) ~[starrocks-fe.jar:?] at com.starrocks.common.ErrorReport.wrapWithRuntimeException(ErrorReport.java:118) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:306) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor$StmtExecutorVisitor.visitCreateTableStatement(DDLStmtExecutor.java:212) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.CreateTableStmt.accept(CreateTableStmt.java:344) ~[starrocks-fe.jar:?] at com.starrocks.sql.ast.AstVisitor.visit(AstVisitor.java:107) ~[starrocks-fe.jar:?] at com.starrocks.qe.DDLStmtExecutor.execute(DDLStmtExecutor.java:197) ~[starrocks-fe.jar:?] at com.starrocks.qe.SimpleExecutor.executeDDL(SimpleExecutor.java:146) ~[starrocks-fe.jar:?] ... 5 more Can you diagnose anything

Based on the logs and the

nc

results you provided, there are two distinct issues occurring: one with the FE's internal health and one with the network/heartbeat. 1. The Log Error: "Cluster has no available capacity" This error is the most telling. It means the FE is trying to create internal system tables (like

loads_history

) but it cannot find any active Backends (BEs) to store the data. * The Cause: You likely have no BEs added to the cluster, or all your BEs are currently marked as

Alive: false

. * The Fix: Run

SHOW BACKENDS;

on your Leader FE. * If the list is empty, you must add BEs:

ALTER SYSTEM ADD BACKEND "be_ip:9050";

* If they are all

Alive: false

, check the BE logs (

be.log

) to see why they aren't connecting. 2. The "Connect time out" in

SHOW FRONTENDS

The

nc

test you ran was local (the node connecting to itself). However, the "Connect time out" message in

SHOW FRONTENDS

is reported by the Leader FE. It means the Leader cannot reach this node (

172.30.67.218

). Check these specifically: * The "Leader to Follower" Path: Go to your Leader FE machine (the one at

172.30.75.62

) and try the

nc

command from there:

bash
    # Run THIS on the Leader FE machine
    nc -zv 172.30.67.218 9010
    nc -zv 172.30.67.218 9020

If this fails, your AWS Security Group or firewall is blocking traffic between the FEs. Even if the Follower can "see" itself, the Leader must be able to reach it. * BDBJE Heartbeat Port: Note that FEs communicate metadata over the

EditLogPort

(9010) and heartbeats over the

RpcPort

(9020). Both must be open. 3. Binding to the Wrong IP Even though your

priority_networks

range is technically correct, verify which IP the FE actually chose. Run this on the node `172.30.67.218`:

bash
grep "IP:" /opt/starrocks/fe/log/fe.log

If it shows anything other than

172.30.67.218

(like a Docker IP or the public

54.226.175.90

), then the Leader will never be able to reach it on the private network. Summary Checklist 1. Add/Fix Backends: You cannot successfully initialize an FE's internal metadata state if there are no BEs. Run

SHOW BACKENDS

and ensure at least one is

Alive

. 2. Cross-Node Connectivity: Ensure the Leader can

nc

to the Follower on 9010 and 9020. 3. Confirm Binding: Ensure the FE log says it is using

IP: 172.30.67.218

. If you have no BEs yet, add one first, then restart this FE node. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • faq/Deploy_faq.md • administration/Meta_recovery.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/Meta_recovery.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md

@Rocky Backends all alive on leader node

STARROCKS-CORE-SERVICES-FE-1  54.166.227.30 // (172.30.75.62)  ~  130  $  nc -zv 172.30.67.218 9010
                                                         Ncat: Version 7.93 ( <https://nmap.org/ncat> )                                                                                                                 Ncat: Connected to 172.30.67.218:9010.                                                                                                                       Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                                                                         STARROCKS-CORE-SERVICES-FE-1  54.166.227.30 // (172.30.75.62)  ~  $  nc -zv 172.30.67.218 9020
Ncat: Version 7.93 ( <https://nmap.org/ncat> )                                                                                                                 Ncat: Connected to 172.30.67.218:9020.
Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.                                                                                                         STARROCKS-CORE-SERVICES-FE-1  54.166.227.30 // (172.30.75.62)  ~  $

Leader node able to connect to follower node

grep "IP:" /opt/starrocks/fe/log/fe.log

shows nothing on follower node STARROCKS-CORE-SERVICES-FE-2 54.226.175.90 // (172.30.67.218) opt starrocks fe conf 1 $ journalctl -fu starrocks-fe Sep 22 161248 ip-172-30-67-218.ec2.internal systemd[1]: Started starrocks-fe.service - StarRocks Frontend Service. Sep 22 161248 ip-172-30-67-218.ec2.internal starrocks-fe[167150]: Infered JAVA_HOME=/usr/lib/jvm/java-24-amazon-corretto.x86_64 Sep 22 161607 ip-172-30-67-218.ec2.internal systemd[1]: starrocks-fe.service: Main process exited, code=killed, status=9/KILL Sep 22 161607 ip-172-30-67-218.ec2.internal systemd[1]: starrocks-fe.service: Failed with result 'signal'. Sep 22 161607 ip-172-30-67-218.ec2.internal systemd[1]: starrocks-fe.service: Consumed 16.764s CPU time. Sep 22 161622 ip-172-30-67-218.ec2.internal systemd[1]: starrocks-fe.service: Scheduled restart job, restart counter is at 2. Sep 22 161622 ip-172-30-67-218.ec2.internal systemd[1]: Stopped starrocks-fe.service - StarRocks Frontend Service. Sep 22 161622 ip-172-30-67-218.ec2.internal systemd[1]: starrocks-fe.service: Consumed 16.764s CPU time. Sep 22 161622 ip-172-30-67-218.ec2.internal systemd[1]: Started starrocks-fe.service - StarRocks Frontend Service. Sep 22 161622 ip-172-30-67-218.ec2.internal starrocks-fe[172116]: Infered JAVA_HOME=/usr/lib/jvm/java-24-amazon-corretto.x86_64 Is all I get from system journal ... 5 more 2026-04-21 211240.832Z INFO (statistics meta manager|46) [StatisticsMetaManager.createSampleStatisticsTable():144] create sample statistics table start 2026-04-21 211240.832Z WARN (statistics meta manager|46) [StatisticsMetaManager.createSampleStatisticsTable():165] Failed to create sample statistics, com.starrocks.common.NoAliveBackendException: No alive backend at com.starrocks.common.util.AutoInferUtil.calDefaultReplicationNum(AutoInferUtil.java:31) ~[starrocks-fe.jar:?] at com.starrocks.statistic.StatisticsMetaManager.createSampleStatisticsTable(StatisticsMetaManager.java:148) ~[starrocks-fe.jar:?] at com.starrocks.statistic.StatisticsMetaManager.createTable(StatisticsMetaManager.java:442) ~[starrocks-fe.jar:?] at com.starrocks.statistic.StatisticsMetaManager.refreshStatisticsTable(StatisticsMetaManager.java:526) ~[starrocks-fe.jar:?] at com.starrocks.statistic.StatisticsMetaManager.runAfterCatalogReady(StatisticsMetaManager.java:556) ~[starrocks-fe.jar:?] at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:78) ~[starrocks-fe.jar:?] at com.starrocks.common.util.Daemon.run(Daemon.java:98) ~[starrocks-fe.jar:?] 2026-04-21 211240.832Z WARN (statistics meta manager|46) [StatisticsMetaManager.refreshStatisticsTable():529] create statistics table table_statistic_v1 failed 2026-04-21 211242.051Z WARN (Load history syncer|61) [LoadsHistorySyncer.getLatestFinishTime():164] failed to get db: statistics, table: loads_history 2026-04-21 211242.051Z INFO (TaskCleaner|131) [TaskManager.dropTasks():454] drop tasks:[] 2026-04-21 211242.051Z WARN (PredicateColumnsDaemonThread|149) [PredicateColumnsMgr$DaemonThread.runAfterCatalogReady():232] system table of predicate_columns is still not ready 2026-04-21 211242.637Z WARN (Routine load task scheduler|66) [RoutineLoadTaskScheduler.process():117] no available be slot to scheduler tasks, wait for 10 seconds to scheduler again, you can set max_routine_load_task_num_per_be bigger in fe.conf, current value is 16 2026-04-21 211244.697Z INFO (Thread-80|165) [StarRocksFE.lambda$addShutdownHook$3():504] start to execute shutdown hook 2026-04-21 211244.698Z INFO (Thread-80|165) [StarRocksFE.lambda$addShutdownHook$3():524] shutdown hook end Is sample logs at the end of fe.log

It looks like your FE node is in a very specific failure state. Here is the diagnosis based on the logs you shared: 1. The Log Timestamp Discrepancy Watch out: Your

journalctl

shows September 22, but your

fe.log

shows April 21. * This suggests you might be looking at a stale log file or the system clock on that node is incorrectly set to the past. * Please check the actual modified date of the log file:

ls -l /opt/starrocks/fe/log/fe.log

. If it hasn't been updated today, the errors you're reading aren't from the current crash. 2. The

status=9/KILL

(SIGKILL) The

journalctl

message

Main process exited, code=killed, status=9/KILL

is critical. This means the process did not crash on its own—the OS or a supervisor killed it. There are two likely reasons: * Systemd Timeout: If the FE takes too long to join the cluster (specifically the BDBJE handshake), systemd will eventually kill it. Notice that your FE started at

16:12

and was killed at

16:16

. This 4-minute window is a common systemd timeout. * OOM Killer: If the machine ran out of memory, the Linux kernel might have SIGKILL'd it. _ _Check this with:*

dmesg | grep -i "out of memory"

or

grep -i "oom" /var/log/messages

. 3. "No alive backend" +

TableKeeper

The

fe.log

(from April) shows the FE trying to run

CREATE TABLE

for internal statistics. * This is a red flag: Only the Leader FE should be running these DDL tasks. * If a node you intended to be a Follower is trying to create tables and failing because there are "No alive backends," it means it thinks it is a Leader of its own brand-new, empty cluster. * This happens if the FE was started without the

--helper

flag or if it failed to find the Leader and "re-elected" itself as a standalone master. Recommended Fix Procedure Since your node is currently "Split Brain" (it thinks it's a leader but has no data/BEs) and being killed by the system, follow these steps to reset it: 1. Stop the FE service:

bash
    sudo systemctl stop starrocks-fe

2. Clear the broken metadata: (Since it hasn't joined correctly yet, its current metadata is invalid).

bash
    rm -rf /opt/starrocks/fe/meta/*

3. Check Follower-to-Leader Connectivity: You verified Leader -> Follower. Now check Follower -> Leader (this is where BDBJE usually hangs):

bash
    # Run this on the FOLLOWER node (172.30.67.218)
    nc -zv 172.30.75.62 9010

4. Increase Systemd Timeout: Edit your service file (

/etc/systemd/system/starrocks-fe.service

) and add this under

[Service]

to prevent the `status=9/KILL`:

ini
    TimeoutStartSec=600

Then run

sudo systemctl daemon-reload

. 5. Restart with Helper: The best way to ensure it joins is to start it manually once with the helper flag before letting systemd take over:

bash
    /opt/starrocks/fe/bin/start_fe.sh --helper 172.30.75.62:9010 --daemon

6. Verify via Leader: On the Leader FE, run

SHOW FRONTENDS;

. If

Alive

becomes

true

for

172.30.67.218

, the join was successful. References • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/graceful_exit.md • administration/management/FE_configuration.md • deployment/deploy_manually.md References • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/graceful_exit.md