< Rocky> We see compaction run for roughly five minutes with StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> We see compaction run for roughly f...

Justin

06/01/2026, 4:58 PM

@Rocky We see compaction run for roughly five minutes without much impact on compaction score. Is there a way to force the compaction to continue? What are the relevant compaction configuration settings?

Rocky

06/01/2026, 4:58 PM

In StarRocks, compaction behavior and the "Compaction Score" calculation differ slightly depending on whether you are using a Shared-data (compute-storage separation) or Shared-nothing (standard) architecture. 1. How to Force Compaction To force compaction to continue or trigger it manually: * Shared-data Clusters (v3.1+): You can trigger a manual compaction task using SQL. This is the most effective way to force the system to prioritize specific tables or partitions.

Copy code

sql
    -- For the whole table
    ALTER TABLE <table_name> COMPACT;

    -- For a specific partition
    ALTER TABLE <table_name> COMPACT <partition_name>;

* Shared-nothing Clusters: You must trigger compaction at the tablet level via the BE HTTP API:

Copy code

bash
    curl -X POST "http://<be_ip>:<be_http_port>/api/compact?tablet_id=<tablet_id>&compaction_type=cumulative"

2. Relevant Compaction Configurations If compaction is running but the score isn't dropping, you likely need to tune the concurrency or the number of files merged per task. FE Configurations (Shared-data) *

lake_compaction_score_selector_min_score

(Default: 10): The threshold score that triggers compaction. If your scores are staying just above this, lowering it won't help; instead, you need to speed up execution. *

lake_compaction_max_tasks

(Default: -1): Controls total concurrent tasks in the cluster. Set to a positive integer to increase concurrency if the cluster has spare CPU/IO. BE/CN Configurations (Common) *

compact_threads

(Default: 4): The number of threads on each node dedicated to compaction. It is recommended to set this to 25% of the CPU core count. *

max_cumulative_compaction_num_singleton_deltas

(Default: 1000 or 500): The maximum number of segments/rowsets merged in a single task. Recommendation: Set this to

to make individual tasks finish faster and release resources sooner. *

lake_pk_compaction_max_input_rowsets

(Primary Key tables only): Controls the input limit for PK table compaction. 3. Why the Score Might Not Change A "five-minute run without impact" usually occurs for one of the following reasons: * Size-tiered Logic: In shared-data, rowsets are grouped by size (e.g., 10MB, 100MB). The score is determined by the group with the most files. If compaction merges 50 files in a 10MB group but you still have 100 files in a 100MB group, the

MaxCS

(Max Compaction Score) will not drop because it tracks the largest group. * Task Limits: If you have 1,000 small files and

max_cumulative_compaction_num_singleton_deltas

is set to 100, one task will only reduce the file count by 99. You need multiple successive runs to see a significant score drop. * Stuck Tasks: If a task is "running" but not progressing, check its status:

Copy code

sql
    -- Shared-data status
    SELECT * FROM information_schema.be_cloud_native_compactions;
    -- General status
    SHOW PROC '/compactions';

Recommendation: Check

information_schema.be_cloud_native_compactions

(for shared-data) to see if tasks are failing with errors or if they are simply processing very large amounts of data. If the score is high (e.g., > 100), consider increasing

compact_threads

and reducing

max_cumulative_compaction_num_singleton_deltas

. References • administration/http_interface.md • administration/management/BE_configuration.md • administration/management/compaction.md

Justin

06/01/2026, 5:00 PM

@Rocky is compact_threads per node or the entire cluster?

Rocky

06/01/2026, 5:00 PM

compact_threads

is a per-node (BE or CN) configuration. * compact_threads
(Per Node): This controls the number of concurrent compaction worker threads running on each individual Backend (BE) or Compute Node (CN). If you set this to

and have 10 nodes, your cluster can theoretically execute 80 compaction subtasks simultaneously (provided the FE dispatches enough). * lake_compaction_max_tasks
(Entire Cluster): In shared-data mode, this FE configuration sets the global limit for concurrent compaction tasks across the whole cluster. By default, it is

-1

(adaptive), which usually calculates to

number_of_nodes * 16

. Recommendation In production, it is recommended to set

compact_threads

to 25% of the CPU core count on each node to ensure compaction has enough resources to keep up with ingestion without starving query interference. How to update it Since v3.1.7 (and v3.2.2+), you can update this dynamically on all nodes using SQL:

Copy code

sql
UPDATE information_schema.be_configs SET VALUE = 8 WHERE name = "compact_threads";

References • administration/management/compaction.md • administration/management/BE_configuration.md References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/FE_configuration.md

Justin

06/01/2026, 5:11 PM

@Rocky Our compaction score is 30,080.

Rocky

06/01/2026, 5:11 PM

A compaction score of 30,080 is extremely critical. In StarRocks, a score above 100 is considered unhealthy, and in shared-data clusters, scores exceeding 2,000 typically cause the system to reject new import transactions (

lake_compaction_score_upper_bound

). At 30,000+, you likely have tens of thousands of small segment files. Even if compaction runs for five minutes, it may only merge 100–1,000 files at a time, making the impact on the total score negligible in the short term. 1. How to Force Compaction To force the system to prioritize this table and clear the backlog faster: * Shared-data (v3.1+): Run a manual compaction SQL. This bypasses the standard scheduler and forces a task for the specific table.

Copy code

sql
    ALTER TABLE <table_name> COMPACT;
    -- Or for a specific partition (check scores via SHOW PARTITIONS)
    ALTER TABLE <table_name> COMPACT PARTITION <partition_name>;

* Shared-nothing: Use the BE HTTP API for specific tablets with the highest scores.

Copy code

bash
    curl -X POST "http://<be_ip>:<be_http_port>/api/compact?tablet_id=<tablet_id>&compaction_type=cumulative"

2. Emergency Configuration Tuning To recover from a 30,000+ score, you must increase the throughput of the compaction worker threads. On BE/CN Nodes (via SQL in v3.1.7+): * compact_threads
: Increase this to 25%–50% of your total CPU cores per node. If you have 32 cores, set it to

during this recovery period. * max_cumulative_compaction_num_singleton_deltas
: Reduce this to
100
. _ _Why:* The default (1,000) makes tasks very large and slow. Setting it to 100 allows tasks to finish in seconds, releasing memory and updating the score more frequently, which prevents tasks from looking "stuck."

Copy code

sql
-- Example for all nodes
UPDATE information_schema.be_configs SET VALUE = "16" WHERE name = "compact_threads";
UPDATE information_schema.be_configs SET VALUE = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";

On FE (Shared-data only): * lake_compaction_max_tasks
: Increase the global concurrency limit (e.g., set to

or higher depending on node count) to allow more simultaneous tasks across the cluster.

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "64");

3. Monitoring Progress To see if compaction is actually working or if it is failing with errors: 1. Check Task Status:

Copy code

sql
    -- Shared-data
    SELECT * FROM information_schema.be_cloud_native_compactions WHERE STATUS != 'OK';
    -- General
    SHOW PROC '/compactions';

2. Identify Bottlenecks: Look at

information_schema.partitions_meta

to see if the

MaxCS

(Max Compaction Score) is dropping in specific partitions. Root Cause Warning A score of 30,000 is almost always caused by high-frequency small loads (e.g., Flink/Spark jobs committing every second or

INSERT INTO ... VALUES

statements). * Fix: Increase your ingestion interval to at least 5–10 seconds and increase the batch size to 100MB+ per load. If using Flink, adjust

sink.buffer-flush.max-rows

and

sink.buffer-flush.interval

. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/BE_configuration.md References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • deployment/shared_data/feature-support-shared-data.md

Rocky

06/01/2026, 5:11 PM

• faq/shared_data_faq.md

Francisco Silva

06/02/2026, 9:12 AM

@Rocky I set max_backend_down_time_second = 300 and replication_num=2, with 3 BE nodes but when a node went down, it took 15 minutes for the tablets to be distributed to the remaining 2 nodes. Why?

Rocky

06/02/2026, 9:12 AM

The reason for the 15-minute delay is likely that you are adjusting the wrong parameter for the modern tablet scheduler. While

max_backend_down_time_second

exists, StarRocks (since v2.5.7) uses a specific parameter for the tablet scheduler to determine when to start migrating/repairing replicas from a down node: The Culprit:
tablet_sched_be_down_tolerate_time_s
The default value for this FE configuration is 900 seconds, which is exactly 15 minutes. This matches the behavior you observed. * tablet_sched_be_down_tolerate_time_s
: This is the threshold the Tablet Scheduler uses to decide a BE has been down long enough that its data should be re-replicated elsewhere. * max_backend_down_time_second
: This is an older/broader parameter that often determines when a node is considered "permanently" gone from the cluster's metadata perspective, but the specific logic for tablet migration is usually governed by the threshold above. Recommended Fix To reduce the migration delay to 5 minutes, you should update the FE configuration:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_be_down_tolerate_time_s" = "300");

Additional Factors in Timing Even after setting the threshold to 300 seconds, the total time until all tablets are "distributed" might still be slightly higher than 5 minutes due to: 1. Heartbeat Detection: It takes roughly 15–20 seconds (3 failed heartbeats at 5s intervals) for the FE to mark the BE as "Dead." 2. Repair Delay: There is a setting

tablet_repair_delay_factor_second

(default 60s) which adds a small buffer before the clone tasks are actually scheduled. 3. Scheduling Slots: If you have many tablets, the speed of migration is limited by

tablet_sched_max_scheduling_tablets

(default 10,000) and

tablet_sched_max_balancing_tablets

(default 500). Note: Be careful with very low values in production. If a node restarts for a quick maintenance or a brief network blip, a low threshold will trigger massive unnecessary data movement across your cluster. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md

Francisco Silva

06/02/2026, 9:13 AM

@Rocky can you point me to the docs of max_backend_down_time_second beeing old

Rocky

06/02/2026, 9:14 AM

The behavior you observed is because tablet_sched_be_down_tolerate_time_s
is the primary parameter controlling the tablet repair delay, not

max_backend_down_time_second

. The 15 minutes you experienced matches exactly the default value of

tablet_sched_be_down_tolerate_time_s

(900 seconds). Why your setting didn't work In StarRocks (specifically since version v2.5.7), the logic for node failure was split to provide more granular control: 1. max_backend_down_time_second
(Old/Broad): This is a general cluster-level parameter. It determines when a BE is considered "permanently" gone from the cluster's heartbeat perspective. Even if you set this to 5 minutes, the Tablet Scheduler (the part of the FE that actually moves data) has its own independent timer. 2. tablet_sched_be_down_tolerate_time_s
(Specific/Current): This is the threshold used by the Tablet Scheduler. It defines how long the scheduler waits for a "Dead" BE to come back before it gives up and starts creating new replicas on other nodes. * Default: 900s (15 minutes). * Introduced in: v2.5.7. Proof from the Documentation You can find the definition of the modern parameter in the FE Configuration documentation:

tablet_sched_be_down_tolerate_time_s

* Default: 900

* Description: The maximum duration the scheduler allows for a BE node to remain inactive. After the time threshold is reached, tablets on that BE node will be migrated to other active BE nodes.

* Introduced in: v2.5.7

How to achieve your goal To make tablets migrate after 5 minutes, you must set this specific scheduler parameter:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_be_down_tolerate_time_s" = "300");

If you only change

max_backend_down_time_second

, the FE might mark the node as "Dropped" or "Dead" in the node list, but the Tablet Scheduler will still wait its default 15 minutes before it begins the heavy work of replicating data to the other 2 nodes. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md

Francisco Silva

06/02/2026, 10:55 AM

@Rocky what table does SHOW proc statistic uses to fetch the unhealthy tablet nums

Rocky

06/02/2026, 10:55 AM

The command

SHOW PROC '/statistic'

does not use a physical SQL table. Like most

SHOW PROC

commands, it is a virtual interface that retrieves information directly from the Frontend (FE) node's in-memory metadata and internal components, specifically: 1. GlobalStateMgr (Catalog): To count databases, tables, and total tablets. 2. TabletScheduler: To identify tablets that are currently in an "unhealthy" state (e.g., missing replicas, version mismatches, or currently being cloned). SQL-Queryable Alternatives If you are looking for a way to query similar information using standard SQL (for example, to join with other metadata), you can use the following: * information_schema.be_tablets
: This table provides the state of tablets from the perspective of the Backends (BE). You can filter by the

STATE

column (e.g.,

WHERE STATE != 'NORMAL'

). * ADMIN SHOW REPLICA STATUS FROM <table_name>
: While not a table in

information_schema

, this command returns a tabular result that categorizes tablets into specific unhealthy statuses like

VERSION_ERROR

MISSING

, or

SCHEMA_ERROR

. * SHOW TABLET <tablet_id>
: Provides the

IsSync

flag and a

DetailCmd

to drill down into why a specific tablet is flagged. How "Unhealthy" is Calculated The

UnhealthyTabletNum

/statistic

is an aggregate count maintained by the FE's Tablet Scheduler. A tablet is considered unhealthy if: * The number of healthy replicas is less than the expected

replication_num

. * A replica has a version that falls behind the committed version (Version Error). * The tablet is currently undergoing a

CLONE

task to repair or balance data. For a deep dive into specific unhealthy tablets, the recommended workflow is: 1. Run

SHOW PROC '/statistic/<db_id>'

to get the list of tablet IDs. 2. Run

SHOW TABLET <tablet_id>

to find the specific BE and reason. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/resource_management/Replica.md • sql-reference/information_schema/be_tablets.md References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md

Francisco Silva

06/02/2026, 1:19 PM

@Rocky is there any metric or SQL query we can use to fetch the number of unhealthy tablets?

Rocky

06/02/2026, 1:20 PM

Yes, you can fetch the number of unhealthy tablets using both SQL queries and Prometheus metrics. 1. SQL Query The most accurate way to query this via SQL is using the

information_schema.be_tablets

table. This table tracks the state of every tablet replica across all BEs. To get the total count of unhealthy tablets:

Copy code

sql
SELECT COUNT(*) 
FROM information_schema.be_tablets 
WHERE STATE != 'NORMAL';

_Note: A tablet is usually considered unhealthy if its state is

REPLICA_MISSING

REPLICA_VERSION_MISSING

, or if it is currently

CLONE

-ing._ To get a breakdown by Backend:

Copy code

sql
SELECT BE_ID, STATE, COUNT(*) 
FROM information_schema.be_tablets 
WHERE STATE != 'NORMAL'
GROUP BY BE_ID, STATE;

2. Prometheus Metrics The StarRocks Frontend (FE) exports metrics specifically for the Tablet Scheduler. You can monitor these to see how many tablets are currently being repaired or are waiting for scheduling: * starrocks_fe_scheduled_pending_tablet_num
: Number of tablets waiting to be repaired or balanced. * starrocks_fe_scheduled_running_tablet_num
: Number of tablets currently undergoing repair (cloning) or balancing. If these numbers are high, it indicates the cluster has identified unhealthy tablets and is actively working to fix them. 3. Administrative Commands While not a "queryable" table for joins, the most common way to see the summary is:

Copy code

sql
SHOW PROC '/statistic';

This returns a column

UnhealthyTabletNum

. To find the specific IDs of those tablets, you can drill down into a specific database:

Copy code

sql
SHOW PROC '/statistic/<db_id>';

Summary of "Unhealthy" Types If you use

ADMIN SHOW REPLICA STATUS

, you can filter by specific failure types: * MISSING
: Replicas are lost (e.g., node down). * VERSION_ERROR
: Data is out of sync. * SCHEMA_ERROR
: Metadata mismatch. References • administration/management/monitoring/metrics.md • sql-reference/information_schema/be_tablets.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • sql-reference/information_schema/be_tablets.md References • administration/management/monitoring/metrics.md

Francisco Silva

06/02/2026, 1:23 PM

@Rocky starrocks_fe_scheduled_pending_tablet_num
are not very accurate because what if for some reason the task does not start

Copy code

information_schema.be_tablets

for this the STATE is not NORMAL but running, and I check and once the node goes down the state does not change

Rocky

06/02/2026, 1:23 PM

To accurately fetch the number of unhealthy tablets when standard metrics like

starrocks_fe_scheduled_pending_tablet_num

aren't enough (e.g., because the scheduler hasn't queued a task yet), you can use the following methods: 1. The SQL Query Approach (Most Accurate) Since

information_schema.be_tablets

might still show a state of

NORMAL

for replicas on a node that just went down, you should join it with

information_schema.be_nodes

to identify replicas that are technically "NORMAL" but effectively "MISSING" because their host is dead. Query to count replicas on dead nodes:

Copy code

sql
SELECT count(*) 
FROM information_schema.be_tablets t
JOIN information_schema.be_nodes n ON t.BE_ID = n.BE_ID
WHERE n.ALIVE = 'false';

Query to find tablets with Version Inconsistency or other issues: While

be_tablets

doesn't have a "Health" column, the

ADMIN SHOW REPLICA STATUS

command is the source of truth for "Unhealthy" status (Missing, Version Error, etc.). Unfortunately,

ADMIN SHOW REPLICA STATUS

cannot be queried as a subquery or joined easily. However, you can often identify "Unhealthy" tablets in

be_tablets

by looking for versions that don't match the table's current version, though this requires knowing the expected version. 2. Prometheus Metrics As you noted,

starrocks_fe_scheduled_pending_tablet_num

only counts tablets that the Scheduler has already accepted into its queue. If you want to see the count of tablets that the Checker has identified as unhealthy (which matches the

UnhealthyTabletNum

SHOW PROC '/statistic'

), check for the following metric: * starrocks_fe_tablet_num_unhealthy
(Available in newer StarRocks versions/monitoring templates). * If that is not appearing in your Prometheus, you can monitor the starrocks_fe_tablet_num
grouped by node. If a node's count drops to 0 or its heartbeat fails, you can alert on the delta. 3. Automated Monitoring (The "Proc" Hack) Since

SHOW PROC '/statistic'

is the most reliable source but isn't a standard table, many users use a sidecar script or a simple exporter to run this command and parse the

UnhealthyTabletNum

column. Why
be_tablets
"State" doesn't change The

STATE

column in

be_tablets

(like

NORMAL

CLONE

) refers to the task state of the replica on the BE. If the BE is down, it cannot report a state change. The FE's metadata will keep the last known state (usually

NORMAL

) until the Tablet Scheduler explicitly decides to migrate or delete that replica. This is why joining with the node's

ALIVE

status (as shown in the SQL above) is the only way to detect those unhealthy replicas via SQL. References • sql-reference/information_schema/be_tablets.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md References • faq/operation_maintenance_faq.md • sql-reference/information_schema/be_tablets.md • administration/management/monitoring/metrics.md References • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/information_schema/be_tablets.md

Francisco Silva

06/02/2026, 1:24 PM

@Rocky information_schema.be_nodes does not exist

Rocky

06/02/2026, 1:25 PM

I apologize for the confusion in my previous response. You are correct—*`information_schema.be_nodes` does not exist*. I likely confused it with a naming convention found in other systems. In StarRocks, there is no standard SQL table in

information_schema

that provides the live "Alive" status of nodes for joining. To fetch the number of unhealthy tablets or node status, please use the following verified methods: 1. The Source of Truth:
SHOW BACKENDS
Since there is no queryable SQL table for node health, you must use the administrative command to see which nodes are down:

Copy code

sql
SHOW BACKENDS;

Look at the Alive
column (`true`/`false`). If a node is

false

, all tablets on that node are considered unhealthy/missing until they are repaired. 2. Querying Unhealthy Tablets via SQL If you want to use a SQL query to find tablets that the system has already flagged as problematic (like

REPLICA_MISSING

REPLICA_VERSION_MISSING

), you can use `information_schema.be_tablets`:

Copy code

sql
SELECT STATE, COUNT(*) 
FROM information_schema.be_tablets 
WHERE STATE != 'NORMAL'
GROUP BY STATE;

Why the state might not change immediately: As you observed, if a node goes down, the state in

be_tablets

might not update instantly. This is because this table reflects the FE's last known metadata reported by the BEs. It takes a few minutes for the FE's Tablet Checker to timeout the node and mark those replicas as

REPLICA_MISSING

. 3. The Most Accurate Metric (Prometheus) If you are building a dashboard and find the "Pending/Running" metrics inaccurate, the FE exports a specific metric that matches the

UnhealthyTabletNum

seen in `SHOW PROC '/statistic'`: * Metric Name:

starrocks_fe_tablet_num_unhealthy

* Source: FE Exporter (

http://<FE_IP>:<FE_HTTP_PORT>/metrics

) This metric is the global count of tablets that are currently in an unhealthy state, regardless of whether a repair task has started yet. 4. Alternative: Tablet Scheduler Tasks If you want to see tablets that are actively being repaired because of a node failure, you can query this view:

Copy code

sql
SELECT * FROM information_schema.fe_tablet_schedules WHERE TYPE = 'REPAIR';

This will show you exactly which tablets are being fixed and the reason (e.g.,

REPLICA_MISSING

). Summary * Total Unhealthy Count: Use

SHOW PROC '/statistic'

or the Prometheus metric

starrocks_fe_tablet_num_unhealthy

. * Identify Down Nodes: Use

SHOW BACKENDS

. * Identify Specific Tablets: Use

information_schema.be_tablets

(after the checker timeout) or

information_schema.fe_tablet_schedules

. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/information_schema/be_tablets.md • sql-reference/information_schema/fe_tablet_schedules.md • administration/management/monitoring/metrics.md References • sql-reference/information_schema/fe_tablet_schedules.md • administration/management/FE_configuration.md • sql-reference/information_schema/be_tablets.md • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md

Francisco Silva

06/02/2026, 1:25 PM

@Rocky are you sure this metric exists

starrocks_fe_tablet_num_unhealthy

Rocky

06/02/2026, 1:26 PM

I apologize for that error. You are correct to be skeptical: starrocks_fe_tablet_num_unhealthy
is not a standard, built-in Prometheus metric in the official StarRocks documentation. I likely conflated the name with the "UnhealthyTabletNum" field found in the command line output. If you need to fetch the number of unhealthy tablets, here is the accurate state of affairs: 1. The Prometheus Metric Gap There is currently no single, direct Prometheus metric that perfectly mirrors the

UnhealthyTabletNum

from

SHOW PROC

. The most relevant metrics you can use for alerting are: * starrocks_fe_scheduled_pending_tablet_num
: Tablets waiting to be repaired. * starrocks_fe_scheduled_running_tablet_num
: Tablets actively being repaired. * err_state_metric
: (Added in v2.5+) Specifically tracks the number of tablets in an "error state" for Primary Key tables. 2. Why the metrics feel "inaccurate" As you noted, these metrics only show tablets that the Tablet Scheduler has already recognized and queued for action. * The Delay: When a BE goes down, there is a heartbeat timeout (controlled by

tablet_repair_delay_factor_second

in FE config, default is often 30 mins) before the FE marks those tablets as "missing" and starts counting them as unhealthy. * The Check: If the scheduler task fails to start (e.g., due to resource limits or configuration), it may not show up in the "Pending" metric immediately. 3. Most Reliable "Query" Method Since

information_schema.be_nodes

does not exist and

be_tablets

relies on the same slow-updating metadata, the most reliable way to get a "live" count of unhealthy tablets is to parse the administrative "Proc" command. While you cannot join it like a table, you can check it manually or via a script:

Copy code

sql
SHOW PROC '/statistic';

This returns a summary table where the

UnhealthyTabletNum

column is the aggregate of all tablets the FE currently considers "Missing," "Version Inconsistent," or "Mismatched." Summary Recommendation * For Dashboards/Alerts: Monitor the sum of

starrocks_fe_scheduled_pending_tablet_num

and

starrocks_fe_scheduled_running_tablet_num

. If this remains high for a long time, it indicates a repair backlog. * For Manual Verification: Always use

SHOW PROC '/statistic'

. * For Primary Key Tables: You can safely use the

err_state_metric

. References • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md

Francisco Silva

06/02/2026, 1:27 PM

@Rocky it seems impossible that a ssystem like StarRocks does not have a metric from unhealthy tablet number

Rocky

06/02/2026, 1:27 PM

It is a valid and common frustration. While StarRocks provides many metrics, there is no direct, built-in Prometheus metric that perfectly mirrors the

UnhealthyTabletNum

found in

SHOW PROC '/statistic'

. The system distinguishes between the Checker (which identifies problems) and the Scheduler (which creates tasks to fix them). The metrics you found (

pending_tablet_num

running_tablet_num

) belong to the Scheduler. Why the metrics feel "inaccurate" 1. The Metadata Lag: When a node goes down, the FE does not instantly mark its tablets as

MISSING

. It waits for a heartbeat timeout (controlled by

tablet_repair_delay_factor_second

, often 30+ minutes by default to prevent thrashing during a simple reboot). During this window,

be_tablets

and Prometheus will still reflect the last known state. 2. Scheduler Queueing: A tablet only appears in the

pending

metric once the FE has officially decided to repair it and added it to the task queue. If the queue is full or the scheduler is throttled, the "unhealthy" count in the Checker will be higher than the "pending" count in the Scheduler. How to accurately monitor this Since there is no native gauge for this in the

/metrics

endpoint, here are the three ways engineers handle this in production: 1. The "Source of Truth" (Manual/Scripted) The only place to get the real-time count of tablets that have failed the checker (Missing, Version Error, etc.) is:

Copy code

sql
SHOW PROC '/statistic';

The column

UnhealthyTabletNum

is the aggregate of all health issues. Many users use a "sidecar" or custom exporter to run this SQL and export it as a Prometheus gauge. 2. Using
information_schema
(SQL-based) If you need to query this via SQL for a report, you can use

fe_tablet_schedules

. While this is also "scheduler" based, it provides more detail on why tasks might not be starting:

Copy code

sql
SELECT TYPE, STATE, COUNT(*) 
FROM information_schema.fe_tablet_schedules 
GROUP BY TYPE, STATE;

If you see many tablets in

REPAIR

with a

STATE

PENDING

, the system knows they are broken but hasn't started the work yet. 3. Primary Key Error Metric If you are using Primary Key tables, there is a specific metric that was added to help with this: * err_state_metric
: This tracks the number of Primary Key tablets in an error state. Summary You are correct: the official FE exporter lacks a
starrocks_fe_tablet_num_unhealthy
gauge. To get that number accurately into a dashboard: * Best Practice: Most StarRocks users monitor

starrocks_fe_scheduled_pending_tablet_num

. If it stays above 0 for a long time, it indicates a problem. * Advanced: Use a custom script/exporter to parse the output of

SHOW PROC '/statistic'

. References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • administration/management/monitoring/Monitor_and_Alert.md References • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_CHECK_TABLET.md

Francisco Silva

06/18/2026, 2:37 PM

@Rocky We're seeing queries crash multiple BE nodes with

Copy code

91cf-7d37-b085-42fafc429478, plan_node_id:171
*** Aborted at 1781692179 (unix time) try "date -d @1781692179" if you are using GNU date ***
PC: @          0xd888ee5 starrocks::SegmentIterator::_switch_context(starrocks::SegmentIterator::ScanContext*)
*** SIGSEGV (@0xd00000000) received by PID 7 (TID 0x7f06e0dec640) LWP(534) from PID 0; stack trace: ***
    @     0x7f0773996ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fd1428 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f0774b3b906 PosixSignals::chained_handler(int, siginfo_t*, void*) [clone .part.0]
    @     0x7f0774b3c37e JVM_handle_linux_signal
    @     0x7f077393f520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0xd888ee5 starrocks::SegmentIterator::_switch_context(starrocks::SegmentIterator::ScanContext*)
    @          0xd89a75b starrocks::SegmentIterator::_init_context()
    @          0xd89afbd starrocks::SegmentIterator::_init_internal()
    @          0xd89bf78 starrocks::SegmentIterator::_init()
    @          0xd8a3cb3 starrocks::SegmentIterator::do_get_next(starrocks::Chunk*)
    @          0xde7e398 starrocks::SegmentIteratorWrapper::do_get_next(starrocks::Chunk*)
    @          0xdb63afc starrocks::TimedChunkIterator::do_get_next(starrocks::Chunk*)
    @          0xdb4ffbf starrocks::TabletReader::do_get_next(starrocks::Chunk*)
    @          0xc8a0f5e starrocks::pipeline::OlapChunkSource::_read_chunk_from_storage(starrocks::RuntimeState*, starrocks::Chunk*)
    @          0xc8a1a4f starrocks::pipeline::OlapChunkSource::_read_chunk(starrocks::RuntimeState*, std::shared_ptr<starrocks::Chunk>*)
    @          0xc345fdf starrocks::pipeline::ChunkSource::buffer_next_batch_chunks_blocking(starrocks::RuntimeState*, unsigned long, starrocks::workgroup::WorkGroup const*)
    @          0xb087c60 auto starrocks::pipeline::ScanOperator::_trigger_next_scan(starrocks::RuntimeState*, int)::{lambda(auto:1&)#1}::operator()<starrocks::workgroup::YieldContext>(starrocks::workgroup::YieldContext&) const [clone .constprop.0]
    @          0xc28a97e starrocks::workgroup::ScanExecutor::worker_thread()
    @          0xe44d75e starrocks::ThreadPool::dispatch_thread()
    @          0xe444135 starrocks::thread::supervise_thread(void*)
    @     0x7f0773991ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f0773a22a84 clone
[1781692180.962][thread: 139667519227456] je_mallctl execute purge success
[1781692180.962][thread: 139667519227456] je_mallctl execute dontdump success
start time: Wed Jun 17 10:36:38 UTC 2026, server uptime:  10:36:38 up 4 days, 20:59,  0 users,  load average: 85.05, 275.99, 167.12
Run with JEMALLOC_CONF: 'percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000,metadata_thp:auto,background_thread:true,prof:true,prof_active:false'

this only happens when the cluster is overloaded and resources are scarce - my question is, what setting can we tune to make sure a heavy query doesnt bring the whole cluster down

Rocky

06/18/2026, 2:38 PM

The crash in

starrocks::SegmentIterator::_switch_context

is related to the Late Materialization feature in the StarRocks storage engine. This feature optimizes scans by only reading a subset of columns (those in the

WHERE

clause) first, and then "switching context" to read the remaining columns only for the rows that passed the filter. When a cluster is under extreme resource pressure (high load average, memory exhaustion), the memory allocation or pointer management within this context-switching logic can occasionally trigger a

SIGSEGV

. To prevent heavy queries from destabilizing your cluster, you should implement the following protections: 1. Immediate Workaround: Disable Late Materialization If you need to stop the crashes immediately while you investigate the "overload" root cause, you can disable the specific code path where the crash occurs. * Action: Add/Update this in

be.conf

and restart BEs:

Copy code

properties
    late_materialization_ratio = 0

Note: This might slightly decrease performance for queries with high-selectivity filters, but it will bypass the context-switching logic seen in your stack trace. ────────── 2. Strategic Fix: Resource Groups & Big Query Limits To prevent a single heavy query from monopolizing resources and causing the "scarcity" that triggers these crashes, use StarRocks Resource Groups (Workgroups) to enforce "Circuit Breaking." Create a Resource Group for "Big Queries":

Copy code

sql
CREATE RESOURCE GROUP big_query_group
WITH (
    "type" = "normal",
    "cpu_core_limit" = "10",       -- Limit CPU cores used by this group
    "mem_limit" = "20%",           -- Group-level memory limit
    "big_query_cpu_second_limit" = "100",  -- KILL query if it uses > 100s CPU
    "big_query_scan_rows_limit" = "1000000", -- KILL query if it scans > 1M rows
    "big_query_mem_limit" = "4294967296"     -- KILL query if it uses > 4GB RAM
);

Assign users or specific query patterns to this group:

Copy code

sql
ALTER USER 'power_user' SET ("resource_group" = "big_query_group");

────────── 3. Tuning Query Queues To prevent system-wide overload (the high load average you mentioned), enable Query Queues. This ensures that if the cluster is at capacity, new heavy queries will wait in a queue rather than crashing the nodes. Set these session or global variables: * enable_query_queue = true
: Enables the queuing mechanism. * query_queue_concurrency_limit
: Limits the number of concurrent queries (default is often too high for overloaded clusters; try setting this to a value that matches your node count or CPU cores). * query_queue_mem_used_pct_limit
: Limits queries based on BE memory usage (e.g., set to

0.8

to stop new queries if memory > 80%). 4. Per-Query Memory Limits Ensure that the default per-query memory limit is not set to "unlimited."

Copy code

sql
SET GLOBAL query_mem_limit = 8589934592; -- Limit every query to 8GB by default

Summary of Recommendation 1. Stop the crashes: Set

late_materialization_ratio = 0

be.conf

. 2. Protect the cluster: Implement a Resource Group with

big_query_mem_limit

and

big_query_cpu_second_limit

to automatically kill queries that are too heavy. 3. Manage Load: Enable

enable_query_queue

to prevent the cluster from accepting more work than it can handle. References • administration/management/BE_configuration.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/Query_queues.md • faq/Sql_faq.md References • unloading/Flink_connector.md

Rocky

06/18/2026, 2:38 PM

• administration/Meta_recovery.md • faq/Sql_faq.md • administration/management/monitoring/metrics.md • data_source/datalake_faq.md • administration/management/BE_configuration.md • faq/loading/Loading_faq.md

Francisco Silva

06/18/2026, 2:41 PM

@Rocky how about overall global BE and FE confs for max memory limits

Francisco Silva

06/18/2026, 2:42 PM

@Rocky how can I see what's a normal memory limit per query?

Rocky

06/18/2026, 2:42 PM

To determine a "normal" memory limit for your queries, you need to look at three things: your current settings, your hardware capacity, and your actual query history. 1. How to see the current limit You can check the current memory limit for your session or the global default using SQL:

Copy code

sql
-- Check current session limit
SHOW VARIABLES LIKE 'query_mem_limit';

-- Check global default (applied to all new sessions)
SHOW VARIABLES LIKE 'exec_mem_limit'; -- (Older alias)
SHOW GLOBAL VARIABLES LIKE 'query_mem_limit';

Note: If the value is
0
, it means there is no explicit per-query limit, and a single query can theoretically try to consume the entire BE query pool. 2. How to see what your queries actually use A "normal" limit is best determined by looking at the

memCostBytes

of your successful queries over the last few days. You can query the built-in audit log:

Copy code

sql
SELECT 
    user,
    queryId,
    memCostBytes / 1024 / 1024 AS mem_used_mb,
    cpuCostSeconds,
    sql
FROM starrocks_audit_db__.starrocks_audit_tbl__
WHERE state = 'EOF' -- Only successful queries
ORDER BY memCostBytes DESC
LIMIT 20;

Interpretation: * Total vs. Per-BE: The

memCostBytes

in the audit log is the total memory used across all BE nodes. * Per-BE Limit: The variable

query_mem_limit

is applied per BE node. So, if you have 3 BEs and a query uses 9GB total, its "normal" per-BE usage is roughly 3GB. 3. What is a "Normal" limit? (Rule of Thumb) A safe starting point for

query_mem_limit

is usually calculated based on your BE RAM and expected concurrency: * The "Safety" Limit:

(Total BE RAM * 0.8) / (Expected Concurrent Queries)

* Example: If your BEs have 64GB of RAM and you usually run 4 heavy queries at once: _

64GB _ 0.8 = 51.2GB

(Total Query Pool) *

51.2GB / 4 = 12.8GB

* Recommended Limit: Set

query_mem_limit

8GB

16GB

to prevent one query from causing a crash. 4. Setting the Limit If you've identified that "normal" queries use 4GB but "bad" queries use 40GB and crash the node, set a global safety ceiling:

Copy code

sql
-- Set a global 8GB limit per query per BE
SET GLOBAL query_mem_limit = 8589934592;

Summary of "Normal" Scenario | Normal
query_mem_limit
Small Cluster (16GB RAM) | 2GB - 4GB Medium Cluster (64GB RAM) | 8GB - 16GB Large Cluster (256GB+ RAM) | 32GB - 64GB If you frequently hit these limits, instead of just raising them, consider enabling Spill to Disk so queries finish slower rather than crashing the BE. References • sql-reference/System_variable.md • best_practices/audit_log_resource_group.md • administration/management/resource_management/Memory_management.md References • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md

Francisco Silva

06/18/2026, 2:46 PM

@Rocky is that global persisted? or is it a sesseion variable as well?

Rocky

06/18/2026, 2:46 PM

Yes,

SET GLOBAL

in StarRocks is persisted. Unlike standard MySQL (pre-8.0), where global changes are lost upon restart unless you update the config file, StarRocks stores global variable changes in its internal metadata (the FE edit logs and image). This means the setting will survive a cluster reboot. Here is the breakdown of how the persistence and hierarchy work: 1. Persistence & Scope * SET GLOBAL query_mem_limit = ...;
: * Persistence: Yes, it is saved in the FE metadata and persists across restarts. * Effect: It applies to all future sessions/connections. It does not change the limit for any connection currently active (including the one you used to run the command). * SET query_mem_limit = ...;
(or

SET SESSION ...

): * Persistence: No. It is temporary. * Effect: It applies only to the current connection and disappears once you disconnect. 2. The "Gotcha" (Applying to Current Session) Because

SET GLOBAL

only affects future sessions, a common mistake is running the global command and then wondering why your current query still crashes. If you want it to take effect everywhere immediately, you should run both:

Copy code

sql
-- Apply to this specific window/script right now
SET query_mem_limit = 8589934592;

-- Apply to every new connection from now on (and after restart)
SET GLOBAL query_mem_limit = 8589934592;

3. Variable Hierarchy If multiple limits are set, StarRocks follows this priority (from highest to lowest): 1. SET_VAR
Hint: (e.g.,

SELECT /_+ SET_VAR(query_mem_limit=...) _/ ...

) - applies to one query only. 2. Session Variable: (e.g.,

SET query_mem_limit=...

) - applies to the current connection. 3. User Property: (If you used

ALTER USER ... SET PROPERTIES

) - applies when that specific user logs in. 4. Global Variable: (The

SET GLOBAL

value you set). 5. Default Value: (The hardcoded system default, usually

). Summary * Is it persisted? Yes. * Is it a session variable too? Yes,

query_mem_limit

exists at both levels. References • sql-reference/System_variable.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • administration/management/resource_management/Memory_management.md References • sql-reference/System_variable.md • sql-reference/sql-statements/cluster-management/config_vars/SET.md • administration/management/FE_configuration.md

Rocky

06/18/2026, 2:46 PM

To manage memory effectively and prevent the cluster from crashing during heavy resource scarcity, you should tune the global limits for both the BE (C++ process) and the FE (Java process). 1. Backend (BE) Memory Limits The BE uses a hierarchical memory tracker system. The process-level limit is the most critical. * mem_limit
(be.conf): * Default:

90%

* Description: The absolute maximum memory the BE process can use. * Recommendation: If you have other services on the same node, set this to a literal value (e.g.,

64G

) or a lower percentage. * Behavior: At 80% (Soft Limit), the BE begins aggressive flushing and backpressure. At 90% (Hard Limit), it will start failing memory allocations (or aborting if configured). * query_max_memory_limit_percent
(be.conf): * Default:

* Description: The percentage of

mem_limit

that the Query Pool can consume. * Tuning: If you want to reserve more memory for metadata or loading to ensure cluster stability during heavy queries, lower this to

. * load_process_max_memory_limit_percent
(be.conf): * Default:

* Description: Limits memory used by data loading (MemTables). * Tuning: If the cluster crashes during heavy ingestion, lower this to

. * storage_page_cache_limit
(be.conf): * Default:

20%

(usually) * Description: Caches data pages to speed up repeat scans. * Tuning: Under extreme memory pressure, reducing this can free up "emergency" overhead, though it will slow down queries. ────────── 2. Frontend (FE) Memory Limits The FE is a Java process. Its memory is primarily used for storing cluster metadata (tablet locations, schema, etc.). * JAVA_OPTS
(fe.conf): * Action: Modify

-Xmx

(Max Heap) and

-Xms

(Initial Heap). * Recommendation: Set

-Xmx

and

-Xms

to the same value to avoid JVM resizing pauses. * Sizing Guide: * < 1M tablets: 16GB * 1M - 2M tablets: 32GB * 2M - 5M tablets: 64GB * catalog_trash_expire_second
(fe.conf): * Default:

(24 hours) * Description: How long deleted metadata stays in the "trash" before being purged. * Tuning: If FE memory is consistently high after many table drops/overwrites, lower this to

to reclaim memory faster. ────────── 3. Safety "Kill Switches" To prevent a single query from causing the

SIGSEGV

you saw earlier: * Global Query Limit:

Copy code

sql
    SET GLOBAL query_mem_limit = 8589934592; -- 8GB per query

* Enable Memory-based Queuing: Ensure the cluster stops accepting new queries if memory is already dangerously high:

Copy code

sql
    SET GLOBAL enable_query_queue = true;
    SET GLOBAL query_queue_mem_used_pct_limit = 0.85; -- Stop new queries at 85% BE memory

Summary Table Component | Parameter | Recommended Value BE |

mem_limit

80%

90%

BE |

query_max_memory_limit_percent

(to leave buffer for OS/Meta) BE |

load_process_max_memory_limit_percent

FE |

JAVA_OPTS

(

-Xmx

) |

16G

64G

(based on tablet count) Session |

query_mem_limit

8G

16G

(prevent rogue queries) References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/resource_management/Memory_management.md • faq/Others.md References • administration/management/resource_management/Memory_management.md

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Rocky

06/18/2026, 2:46 PM

Francisco Silva

06/18/2026, 2:47 PM

@Rocky is there a be.conf variable to make it global?

Rocky

06/18/2026, 2:51 PM

For the per-query limit specifically, there is no
be.conf
variable to set it. StarRocks handles that at the SQL level because it needs to be synchronized across the whole cluster. However, there are two

be.conf

variables that act as global caps for the entire Backend process. These are the "guardrails" that prevent any combination of queries or loads from crashing the node: 1. Global Process Limit (
be.conf
) This is the absolute ceiling for the BE process. * mem_limit
* Description: Total memory the BE process is allowed to use. * Configuration:

mem_limit = 80%

mem_limit = 64G

. * How it works: If the process hits this limit, the BE will fail memory allocations or trigger GC/flushing aggressively. 2. Global Query Pool Limit (
be.conf
) This limits how much of the process memory can be used by all queries combined. * query_max_memory_limit_percent
* Default:

(90% of

mem_limit

). * Configuration:

query_max_memory_limit_percent = 70

. * Why use this: If you have heavy data loading or many Primary Key tables (which use significant memory for indexes), lowering this ensures that queries cannot "starve" the rest of the system's memory. ────────── How to apply a per-query limit "Globally" Since you can't put it in

be.conf

, you use the SQL

GLOBAL

scope. Once you run this command, it is stored in the FE metadata and applies to every connection, every user, and every query across the entire cluster, even after a restart:

Copy code

sql
-- Set a cluster-wide per-query limit of 16GB
SET GLOBAL query_mem_limit = 17179869184;

Why the distinction matters Level | Location | Variable | Best Use Case System-wide |

be.conf

mem_limit

| Protect the OS from BE crashing the whole server. Pool-wide |

be.conf

query_max_memory_limit_percent

| Ensure Loading/Metadata always has memory reserved. Query-wide | SQL |

SET GLOBAL query_mem_limit

| Stop a single "rogue" query from eating all memory. Summary: To prevent the crashes you saw during resource scarcity, the most effective "global" change is to set query_max_memory_limit_percent = 70
in

be.conf

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Rocky

06/18/2026, 2:51 PM

Francisco Silva

06/18/2026, 3:03 PM

@Rocky so in short, what configurations do you recommend?

Rocky

06/18/2026, 3:04 PM

To stabilize your cluster and prevent these specific crashes, I recommend the following configuration "Safe Set." 1. Backend Fixes (
be.conf
) These prevent the node-level crash and reserve memory for system stability. * Direct Crash Workaround:

Copy code

properties
    # Bypasses the context-switching logic seen in your SIGSEGV trace
    late_materialization_ratio = 0

* Global Memory Guardrails:

Copy code

properties
    # Absolute ceiling for the BE process (leave 10-20% for OS/other services)
    mem_limit = 80%

    # Combined memory for ALL queries (reserves 20% for Loading/Metadata)
    query_max_memory_limit_percent = 80

2. Frontend Fixes (
fe.conf
) Ensures the FE (Leader) doesn't hang or restart due to metadata pressure. * JVM Sizing:

Copy code

bash
    # Set -Xmx and -Xms to the same value to avoid heap resizing pauses
    # Recommended: 16G (small) to 64G (large clusters)
    JAVA_OPTS="-Xmx32g -Xms32g ..."

* Faster Metadata Reclaim:

Copy code

properties
    # Reduce how long deleted tables stay in memory trash
    catalog_trash_expire_second = 3600

3. Safety "Circuit Breakers" (SQL Global) Run these commands once to apply them cluster-wide. They stop "Big Queries" before they can exhaust node resources. * Set a default per-query ceiling:

Copy code

sql
    -- Limit every query to 16GB. Adjust based on your node RAM.
    SET GLOBAL query_mem_limit = 17179869184;

* Enable Memory-based Queuing:

Copy code

sql
    -- Prevent new queries from starting if the BE is already at 85% memory usage
    SET GLOBAL enable_query_queue = true;
    SET GLOBAL query_queue_mem_used_pct_limit = 0.85;
    SET GLOBAL query_queue_concurrency_limit = 100; -- Cap total concurrent queries

Why these help: 1. late_materialization_ratio = 0
: Directly disables the code path where your crash is occurring (

SegmentIterator::_switch_context

). 2. query_max_memory_limit_percent
: Ensures that even if many queries run, they won't take 100% of the BE's allowed memory, leaving "breathing room" for background tasks. 3. enable_query_queue
: This is your best defense against "resource scarcity." Instead of letting one more query tip the node over into a crash, StarRocks will pause the query in a queue until memory is it safe to run safely. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • administration/management/resource_management/Query_queues.md References • administration/management/BE_configuration.md • sql-reference/System_variable.md • administration/management/FE_configuration.md

Francisco Silva

06/18/2026, 5:12 PM

@Rocky is there a guide or metric I can look at before setting a value on query_mem_limit

Rocky

06/18/2026, 5:13 PM

Before setting a value for

query_mem_limit

, you should analyze your cluster's historical and real-time peak memory usage to ensure you don't set a limit that kills legitimate, necessary queries. Here is a guide on the metrics and tools you should check: 1. Historical Analysis (Best Method) The most reliable way to find a safe limit is to look at your Audit Logs. StarRocks tracks the peak memory usage of every query that has finished. Run this SQL to find the peak memory consumption per query over the last 30 days:

Copy code

sql
SELECT 
    user,
    queryId,
    memCostBytes / (1024 * 1024) AS mem_mb,         -- Peak usage (Cluster-wide)
    (memCostBytes / (1024 * 1024)) / 3 AS mem_per_be_mb, -- Est. usage per BE (e.g., 3 BEs)
    stmt
FROM starrocks_audit_db__.starrocks_audit_tbl__
WHERE state IN ('EOF','OK') 
  AND timestamp >= DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
ORDER BY memCostBytes DESC
LIMIT 20;

* Note:

memCostBytes

is the cluster-wide total. Since

query_mem_limit

is a per-BE limit, divide the

memCostBytes

by your number of BEs to get the approximate value you need to set. 2. Real-Time Monitoring If you want to see what currently running queries are using right now: * SQL Command:

Copy code

sql
    SHOW PROC '/current_queries'; -- Current FE only
    -- OR --
    SHOW PROC '/global_current_queries'; -- All FEs (v3.4+)

Look at the

MemoryUsage

column. This shows the instantaneous memory used by each active query. * BE HTTP Interface: Navigate to

http://<be_ip>:<be_http_port>/mem_tracker?type=query_pool

. This provides a detailed breakdown of the memory currently allocated to the query pool on that specific node. 3. Prometheus/Grafana Metrics If you have Prometheus integrated, monitor these metrics to see overall pressure: * `query_mem_bytes`: Total memory used by all queries on a BE. * `starrocks_be_mem_pool_mem_usage_bytes{type="query_pool"}`: Total query pool usage. Recommendation on Setting the Value 1. Find your "Heavy" queries: Identify the queries in the audit log that use the most memory. 2. Determine if they are valid: If a query uses 50GB but returns 100 million rows for a dashboard, it's valid. If it uses 50GB due to a missing join condition, it's a "rogue" query. 3. Set the ceiling: Set

query_mem_limit

to roughly 20-30% higher than your largest valid query's per-BE usage. _ _Example:* If your largest valid query uses 12GB on a single BE, set

query_mem_limit = 17179869184

(16GB). Important: If you set this value and a query hits it, it will fail with:

Memory limit exceeded: <Reason>... exec_mem_limit: 17179869184

. References • administration/management/monitoring/metrics.md • best_practices/audit_log_resource_group.md • using_starrocks/running_queries.md • administration/management/resource_management/Memory_management.md • sql-reference/System_variable.md References • administration/management/resource_management/Memory_management.md • sql-reference/sql-functions/meta-functions/inspect_memory_detail.md • administration/management/monitoring/metrics.md • faq/Sql_faq.md • using_starrocks/running_queries.md

Francisco Silva

06/18/2026, 5:47 PM

@Rocky ERROR 1064 (HY000): Build Exec OlapScanNode fail, scan info is invalid

Rocky

06/18/2026, 5:47 PM

The error

Build Exec OlapScanNode fail, scan info is invalid

is a generic StarRocks error indicating that the Frontend (FE) cannot find a healthy or valid data replica to read from when building the execution plan. Essentially, the "scan info" is invalid because there are no available Backends (BEs) that can provide the data for the tablets involved in your query. Common Causes 1. Backends are Down: One or more BE nodes are offline, and they held the only available replicas for your data. 2. Unhealthy Tablets: Data replicas are marked as "Bad" or are in an inconsistent state (e.g.,

VERSION_ERROR

CLONE

) due to a previous crash or failed load. 3. Metadata Inconsistency: The FE thinks a replica exists on a specific BE, but that BE has either been dropped or the replica was lost. 4. Single Replica Issues: If your table was created with

replication_num = 1

, any single node failure or local disk issue will immediately trigger this error for those tablets. ────────── How to Troubleshoot and Fix Step 1: Check Backend Health First, ensure all your BE nodes are

Alive

Copy code

sql
SHOW PROC '/backends';

* Fix: If any nodes are

Alive = false

, restart the BE process on those servers. Step 2: Identify the "Bad" Tablets The error message sometimes includes a specific

tablet_id

. If not, find unhealthy tablets across the cluster:

Copy code

sql
SHOW PROC '/statistic';

Look for any non-zero values in the

UnhealthyTablets

column. If you find the problematic tablet ID (e.g.,

), inspect it:

Copy code

sql
SHOW TABLET 12345;

Step 3: Trigger a Manual Repair If you have multiple replicas but some are corrupted, you can force StarRocks to re-synchronize them:

Copy code

sql
ADMIN REPAIR TABLE your_db.your_table;

This triggers the FE to check all tablets for that table and schedule clones for any that are missing or unhealthy. Step 4: Check for Metadata Staleness If you recently dropped a node or had a cluster-wide crash, the FE's metadata cache might be stale. * Short-term fix: Try running

DESC your_table

ANALYZE TABLE your_table

to see if it forces a metadata refresh. * Last Resort: Restarting the Leader FE will clear the memory-based metadata state and force a fresh check of all BE reports. Summary Checklist * Are all BEs

Alive

? * Is

replication_num

at least 2? (If it's 1, data loss on a single node is likely the cause). * Does

SHOW PROC '/statistic'

show unhealthy tablets? * Have you tried

ADMIN REPAIR TABLE

? References • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/administration/ADMIN_REPAIR_TABLE.md • administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 9:06 AM

@Rocky can I set tablet_sched_be_down_tolerate_time_s dynamically?

Rocky

06/19/2026, 9:06 AM

Yes, you can set

tablet_sched_be_down_tolerate_time_s

dynamically without restarting your Frontend (FE) nodes. In the StarRocks documentation, this parameter is marked as "Is mutable: Yes", which means it can be updated at runtime. How to set it Use the

ADMIN SET CONFIG

command from any MySQL client connected to the FE:

Copy code

sql
-- Example: Set the tolerance time to 30 minutes (1800 seconds)
ADMIN SET CONFIG ("tablet_sched_be_down_tolerate_time_s" = "1800");

Why you might use this * During Maintenance: If you know a Backend (BE) will be down for more than the default 15 minutes (900s) for a planned OS update or hardware swap, you can temporarily increase this value to prevent StarRocks from unnecessarily triggering massive tablet redistributions (cloning). * Emergency Recovery: If a node is flapping or you are experiencing network instability, increasing this value prevents the cluster from entering a "rebalancing storm" while you stabilize the infrastructure. Important Notes * Persistence: Changes made with

ADMIN SET CONFIG

are applied to the running FE process and are also persistent (they are written to the FE's internal metadata), so they will survive an FE restart. * Scope: This is a cluster-wide setting that controls how the Leader FE handles BE heartbeats. References • administration/management/FE_configuration.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 9:06 AM

@Rocky there is not ADMIN SET CONFIG

Rocky

06/19/2026, 9:07 AM

My apologies, the command requires the keyword FRONTEND
in the middle. The full syntax to set FE configurations dynamically is:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_be_down_tolerate_time_s" = "1800");

Why you might see an error: 1. Missing
FRONTEND
keyword: As noted above, the command is

ADMIN SET FRONTEND CONFIG

. 2. Privileges: This operation requires the SYSTEM-level OPERATE privilege. If you are not logged in as

root

or a user with administrative rights, the command will fail. 3. Variable Type: Ensure you are setting a configuration parameter (found in

fe.conf

) and not a system variable.

tablet_sched_be_down_tolerate_time_s

is definitely an FE configuration. Alternative: Using the HTTP API If you cannot use the SQL interface, you can also update this setting via a

curl

command to the FE's internal API:

Copy code

bash
curl --location-trusted -u <username>:<password> \
"http://<FE_IP>:<FE_HTTP_PORT>/api/_set_config?tablet_sched_be_down_tolerate_time_s=1800"

_(The default

FE_HTTP_PORT

)._ How to verify the change To confirm the new value has been applied, run:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_sched_be_down_tolerate_time_s%';

Note: Changes made via

ADMIN SET FRONTEND CONFIG

are temporary and will be lost if the FE restarts. To make the change permanent, you must also update the value in your

fe.conf

file on each FE node. References • administration/management/FE_configuration.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • faq/Deploy_faq.md • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/nodes/SHOW_FRONTEND_CONFIG.md References • sql-reference/sql-statements/cluster-management/config_vars/SET.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md • sql-reference/System_variable.md • administration/management/timezone.md

Francisco Silva

06/19/2026, 9:22 AM

@Rocky W20260619 091915.798573 139621583596928 memory_lock.cpp:46] mlock failed for 147415040-357412864 (209997824 bytes): Cannot allocate memory W20260619 092019.944952 139621044315712 kv_store.cpp:327] rocksdb iterate timeout: 63161, limit: 60000 ^C

Rocky

06/19/2026, 9:22 AM

The logs you provided point to two different issues: one related to operating system limits and the other related to disk performance or metadata volume. 1.
mlock failed: Cannot allocate memory
This warning occurs because StarRocks (specifically the Backend or Compute Node) is trying to lock memory pages to prevent them from being swapped to disk, but the operating system is denying the request. * The Cause: Your OS has a limit on how much memory a single process can lock. The default is often very low (e.g., 64KB). * The Fix: You need to increase the

memlock

resource limit to

unlimited

. 1. Temporary check: Run

ulimit -l

on the BE server. If it's not

unlimited

, that is the issue. 2. Permanent Fix: Add the following lines to

/etc/security/limits.conf

(you will need root access):

Copy code

text
        * soft memlock unlimited
        * hard memlock unlimited

3. Note: If you are running in Docker/Kubernetes, you must set the

ulimits

in the container runtime configuration or the Helm chart. ────────── 2.
rocksdb iterate timeout: 63161, limit: 60000
StarRocks uses an internal RocksDB (the

kv_store

) on each Backend to store tablet metadata. This warning means that a scan of this metadata took ~63 seconds, exceeding the internal 60-second safety limit. * The Cause: This usually happens during BE startup or when the FE requests a full tablet report. It indicates that the BE is struggling to read its own metadata from disk quickly enough. * Potential Root Causes: * High Disk I/O: The disk where your

storage_root_path

is located is under extremely high load. * Slow Storage: You are using HDDs (Hard Drives) instead of SSDs/NVMe for the metadata storage. StarRocks metadata operations require fast random I/O. * Too Many Tablets: If a single BE is hosting an excessive number of tablets (e.g., >50,000), scanning the metadata can exceed the timeout. * Troubleshooting Steps: 1. Check disk latency and utilization using

iostat -x 1

. 2. Check the number of tablets on this specific BE by running

SHOW PROC '/backends';

in your SQL client and looking at the

TabletNum

column. 3. Ensure the

storage_root_path

is on an SSD. If your data is on HDDs, try to at least put the metadata on a faster disk if possible. Summary The

mlock

error is a configuration issue that should be fixed to ensure stability. The

rocksdb iterate timeout

is a performance warning—if it happens frequently, it can cause the BE to be marked as "Dead" by the FE because it fails to report its status in time. References • deployment/environment_configurations.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md References • administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 9:26 AM

@Rocky should we set ulimits when running starrocks in a container?

Rocky

06/19/2026, 9:26 AM

Yes, you must set ulimits when running StarRocks in a container. Containers inherit limits from the host's Docker/Containerd daemon by default, which are often too low for a high-performance database. Failure to set these will lead to the exact

mlock

and

rocksdb

errors you encountered, as well as "Too many open files" crashes during heavy loads. Recommended Ulimit Values Based on StarRocks documentation, these are the target values for each BE (Backend) container: Limit | Recommended Value | Why? memlock
|

unlimited

| Prevents memory from being swapped, avoiding your

mlock failed

error. nofile
|

| StarRocks opens many files for tablets, segments, and network sockets. nproc
|

| Handles high-concurrency threads for queries and compaction. stack
|

unlimited

| Required for complex query plan execution. ────────── How to set them 1. In Docker Use the

--ulimit

flag in your

docker run

command:

Copy code

bash
docker run -d \
  --ulimit memlock=-1:-1 \
  --ulimit nofile=655350:655350 \
  --ulimit nproc=65535:65535 \
  --name starrocks-be ...

2. In Docker Compose Add a

ulimits

section to your service:

Copy code

yaml
services:
  starrocks-be:
    image: starrocks/be-ubuntu:latest
    ulimits:
      memlock: -1
      nofile:
        soft: 655350
        hard: 655350
      nproc: 65535

3. In Kubernetes In Kubernetes, some ulimits (like

memlock

and

nofile

) are often restricted. You have two main options: * Option A: StarRocks Operator (Recommended) The StarRocks Operator typically handles these configurations or allows you to specify them in the

spec

. * Option B: Init Container (Manual) Since some ulimits cannot be set directly in a standard

securityContext

, you can use a privileged init container to set them on the node or container level:

Copy code

yaml
    initContainers:
    • name: init-limit
      image: busybox
      command: ["sh", "-c", "ulimit -n 655350 && ulimit -l unlimited"]
      securityContext:
        privileged: true

Important:
vm.max_map_count
In addition to ulimits, you must also set the host's

vm.max_map_count

. This cannot be set inside a standard container; it must be set on the host machine:

Copy code

bash
# On the host machine
sudo sysctl -w vm.max_map_count=262144

(To make it permanent, add

vm.max_map_count=262144

/etc/sysctl.conf

). References • deployment/environment_configurations.md • faq/operation_maintenance_faq.md • administration/management/resource_management/Memory_management.md References • deployment/environment_configurations.md • developers/build-starrocks/Build_in_docker.md • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • administration/management/resource_management/Query_management.md • faq/Others.md

Francisco Silva

06/19/2026, 10:25 AM

@Rocky what are the most important configuration, such as the ulimits, memory and others to make sure StarRocks runs properly in a docker environment specifically?

Rocky

06/19/2026, 10:25 AM

To ensure StarRocks runs reliably and at peak performance in a Docker environment, you must configure both the Host OS (since containers share the host kernel) and the Container Runtime. 1. Host OS Configurations (Crucial) These must be set on the physical or virtual machine running the Docker engine. They cannot be set from inside a standard container. * vm.max_map_count
: StarRocks (via RocksDB) creates many memory map areas. If this is too low, the BE will crash or fail to start.

Copy code

bash
    sysctl -w vm.max_map_count=262144
    # Make it permanent:
    echo "vm.max_map_count=262144" >> /etc/sysctl.conf

* Swap: For database performance, it is highly recommended to disable swap or set

vm.swappiness

. Swapping database memory to disk causes massive latency spikes.

Copy code

bash
    sysctl -w vm.swappiness=0

* Overcommit Memory: Ensure the kernel allows memory allocation requests to succeed.

Copy code

bash
    sysctl -w vm.overcommit_memory=1

────────── 2. Docker Container Ulimits In Docker, you must explicitly pass these limits via the

--ulimit

flag or a

docker-compose.yml

file. Limit | Recommended Value | Why? memlock
|

unlimited

(

-1

) | Prevents memory from being swapped out (fixes your

mlock

error). nofile
|

| StarRocks handles thousands of data segments and network connections. nproc
|

| Required for the high number of threads used for parallel execution. stack
|

unlimited

| Important for deep expression trees in complex SQL queries. Example Docker Compose snippet:

Copy code

yaml
ulimits:
  memlock: -1
  nofile:
    soft: 655350
    hard: 655350
  nproc: 65535

────────── 3. Memory & CPU Configuration StarRocks is designed to use all available resources. In a containerized environment, you must prevent the Docker OOM (Out Of Memory) killer from killing the process. * mem_limit
(in
be.conf
): By default, the BE tries to use 90% of the detected physical memory. If you set a Docker memory limit (e.g.,

--memory=16g

), the BE (in recent versions) will respect the container limit. However, to be safe, manually set this in your environment variables or `be.conf`:

Copy code

bash
    # Inside be.conf or passed as an environment variable
    mem_limit = 80%

Recommendation: Leave ~20% of the container's memory for the OS/Cgroups to avoid the container being killed if the process spikes slightly. * CPU Shares: Do not heavily throttle StarRocks CPUs. The BE uses a pipeline engine that expects high CPU availability. If using Kubernetes/Docker, ensure your

cpu requests

are close to your

cpu limits

. ────────── 4. Networking (Docker Specific) * Host Networking: For the best performance and to avoid issues with internal/external IP mapping, use

--net=host

. * Static IPs/Hostnames: If not using host networking, ensure the FEs and BEs can resolve each other by a persistent hostname or IP. If a container restarts and gets a new IP, the cluster will mark it as

DEAD

. Summary Checklist 1. Host: Set

vm.max_map_count=262144

. 2. Container: Set

ulimit memlock unlimited

and

nofile 655350

. 3. Config: Set

mem_limit

to 80%-90% of the container's RAM. 4. Storage: Mount the

storage

and

log

directories to high-performance local SSD volumes (avoid slow network mounts for metadata). References • deployment/environment_configurations.md • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • faq/operation_maintenance_faq.md References

Rocky

06/19/2026, 10:25 AM

• deployment/environment_configurations.md • developers/build-starrocks/Build_in_docker.md • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md • quick_start/shared-nothing.md

Francisco Silva

06/19/2026, 10:41 AM

@Rocky whenever I restart a BE node, it makes more than 20minutes to become healthy. Is this expected?

Rocky

06/19/2026, 10:41 AM

A 20-minute startup time is not expected for a typical StarRocks cluster, but it is a known behavior in specific scenarios involving high metadata volume or intensive recovery. Normally, a BE should become healthy (status

ALIVE

) within 1–3 minutes. If it takes 20 minutes, it usually points to one of the following three bottlenecks: 1. Massive Tablet Metadata Loading When a BE starts, it must scan its internal metadata store (RocksDB) to load the "header" of every tablet it hosts. * The Cause: If your BE hosts a very high number of tablets (e.g., >100,000 tablets per BE) or if the storage used for metadata is slow (HDD instead of SSD), this process takes a long time. * The Evidence: Look in your

<http://be.INFO|be.INFO>

log for messages like

rocksdb iterate timeout

finished to load tablets

. * The Fix: * Ensure the BE

storage_root_path

is on high-performance SSDs/NVMe. * Consolidate your data: reduce the number of partitions or buckets if you have too many small tablets (target size is ~1GB per tablet). 2. Primary Key Index Rebuilding If you are using the Primary Key table engine, the BE must load or rebuild the Primary Key index in memory during startup. * The Cause: If the BE was shut down ungracefully (e.g.,

kill -9

or an OOM crash), it may need to rebuild the index from the underlying data files, which is extremely I/O and CPU intensive. * The Evidence: Check the logs for

loading primary index

rebuilding primary index

. * The Fix: Always stop StarRocks BEs gracefully (using the

stop_be.sh

script) to allow them to checkpoint the index to disk. 3. Tablet Reporting Delay Even after the BE process is technically "up," it is not considered "Healthy" by the FrontEnd (FE) until it completes its first Full Tablet Report. * The Cause: The BE sends a report of every tablet version it has to the FE. If the FE is overloaded or if the report is massive, the FE may take a long time to process it and update the BE status to

ALIVE

. * The Fix: Check the FE leader's

fe.log

for any "tablet report" related warnings or long processing times. You can check the current status in SQL using:

Copy code

sql
    SHOW PROC '/backends';

Look at the

LastStartTime

vs. the

Alive

status. Summary Checklist * Disk Speed: Are you using SSDs? (Critical for metadata). * Tablet Count: Run

SELECT count(*) FROM information_schema.be_tablets;

— if a single BE has over 100k-200k tablets, startup will be slow. * Shutdown Method: Are you using

stop_be.sh

? Graceful shutdowns drastically speed up the next restart. References • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/management/monitoring/metrics.md • table_design/table_types/primary_key_table.md References • faq/Deploy_faq.md • administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 10:42 AM

@Rocky im using NVMEs, and I have like less than 1000 tablets per be node

Rocky

06/19/2026, 10:42 AM

Even with NVMe storage and a low tablet count (<1000), a 20-minute delay for a BE to become healthy is highly abnormal. In a Docker environment, this usually stems from networking/discovery issues or metadata synchronization delays rather than disk I/O. Here are the most likely causes and how to troubleshoot them: 1. Incorrect IP Registration (
priority_networks
) This is the #1 cause of health status delays in Docker. * The Problem: By default, a StarRocks BE looks at the container's network interfaces and picks the first one to register its IP with the FE. In Docker, there are often multiple interfaces (e.g.,

eth0

docker0

br-xxxx

). If the BE registers an internal Docker IP that the FE cannot reach (or vice versa), the FE will mark it as

DEAD

until a long heartbeat timeout occurs and it finally resolves. * The Fix: Explicitly set the

priority_networks

in your

be.conf

using CIDR notation to force the BE to use the correct network.

Copy code

properties
    # Example: If your Docker network is 172.17.0.x
    priority_networks = 172.17.0.0/16

2. Primary Key Index "Cold Start" (Persistent Index) Even with only 1000 tablets, if those tablets contain millions of rows and you are using the Primary Key engine: * The Problem: If

enable_persistent_index

false

(the default in some older versions), StarRocks must load the entire primary key index into RAM on startup. If the index wasn't saved gracefully, it may trigger a deep scan of the data files. * The Fix: Enable the persistent index in your table properties to allow the BE to load the index from disk (NVMe) instantly instead of rebuilding it.

Copy code

sql
    PROPERTIES (
      "enable_persistent_index" = "true"
    );

3. Graceful vs. Forced Shutdowns If your Docker container is being "killed" rather than stopped gracefully: * The Problem: Docker's default

stop

timeout is often 10 seconds. If the BE needs more time to flush metadata to disk (RocksDB) and doesn't finish, it will perform a "recovery" scan on the next boot. For a database, this recovery can take significantly longer than a clean start. * The Fix: Increase the Docker stop timeout or ensure you are using the

stop_be.sh

script inside the container before it exits.

Copy code

bash
    docker stop -t 120 starrocks-be

4. FE "Tablet Report" Processing Queue A BE is not "Healthy" until the FE processes its initial report. * The Problem: If your FE is under high CPU load or has a massive metadata backlog (BDBJE), it might be slow to process the "Full Tablet Report" sent by the BE upon startup. * The Fix: Check your

fe.log

. Look for

receive tablet report from BE

and see if there is a gap between the BE starting and the FE acknowledging the report. 5. DNS Resolution Delays * The Problem: If you configured your BEs/FEs using hostnames (common in Docker Compose/K8s) rather than IPs, and the Docker internal DNS is slow or has a search domain loop, every internal RPC call may wait for a DNS timeout. * The Fix: Use

net=host

for testing. If the delay disappears, the issue is Docker's internal bridge networking or DNS. Recommended Log Search To find the exact bottleneck, run these commands inside your BE container: 1.

grep "finished to load tablets" storage/log/be.INFO

— See how long the local disk load took. 2.

grep "report_tablet" storage/log/be.INFO

— Check if the BE is failing to send its report to the FE. 3. Check the FE leader log (

fe.log

) for

Backend [10001] is dead

is alive

to see when the FE finally changes its mind. References • deployment/environment_configurations.md • administration/management/BE_configuration.md • faq/Deploy_faq.md

Rocky

06/19/2026, 10:42 AM

• table_design/table_types/primary_key_table.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md

Francisco Silva

06/19/2026, 10:48 AM

@Rocky for graceful shutdowns in docker, do you recommended increasing the timout?

Rocky

06/19/2026, 10:48 AM

Yes, *increasing the timeout is highly recommended*—and in many cases, necessary—for StarRocks to run correctly in Docker. The default Docker stop timeout is only 10 seconds. For a database like StarRocks, this is almost always too short, leading to the 20-minute startup delays you are experiencing. Why 10 Seconds is Not Enough When you run

docker stop

, Docker sends a

SIGTERM

to the process. StarRocks BE then begins its "Graceful Exit" sequence: 1. Stop accepting tasks: It stops taking new queries. 2. Drain in-flight queries: It waits for existing query fragments to finish (the default wait for this alone is 20 seconds). 3. Flush Data: It flushes memory tables (memtables) to disk and checkpoints the Primary Key index and Metadata (RocksDB). If this sequence takes longer than 10 seconds, Docker sends a

SIGKILL

(force kill). This causes the BE to skip the final metadata checkpoint. On the next boot, the BE sees an "unclean shutdown" and must perform a deep scan of its data files to recover, which causes the 20-minute startup time you are seeing. ────────── Recommended Configurations 1. Docker Compose / Docker Run You should increase the grace period to at least 120 seconds (or more if you have very large Primary Key tables). * Docker Compose:

Copy code

yaml
    services:
      starrocks-be:
        image: starrocks/be-ubuntu:latest
        stop_grace_period: 2m  # Set to 2 minutes or more

* Docker Run:

Copy code

bash
    docker stop --time=120 <container_id>

2. BE Configurations (
be.conf
) To coordinate better with the FrontEnd (especially in versions 3.4+), set these parameters to ensure the cluster knows the node is intentionally shutting down: * loop_count_wait_fragments_finish
: This is how long the BE waits for queries to finish (Value * 10 seconds). The default is

(20s). If you have long-running queries, you might increase this, but ensure your Docker timeout is always higher than this value. * graceful_exit_wait_for_frontend_heartbeat
: Set to

true

(available in v3.4.5+). This forces the BE to wait until the FE acknowledges that the node is "SHUTDOWN" before it actually exits, preventing the FE from trying to send new queries to a dying node. How to verify it worked After a restart, check your

<http://be.INFO|be.INFO>

log for these lines: * Graceful: "Receive signal 15 (SIGTERM), exiting..." followed by "BE exited cleanly." * Forced (Bad): If the log simply ends abruptly without an "exited" message, it was killed by Docker, and the next startup will be slow. References • administration/management/graceful_exit.md • administration/management/BE_configuration.md • faq/Deploy_faq.md References • administration/management/graceful_exit.md • administration/management/BE_configuration.md • faq/Deploy_faq.md References • administration/management/graceful_exit.md • developers/build-starrocks/Build_in_docker.md • administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 11:19 AM

@Rocky how long is it expected to take to startup?

Rocky

06/19/2026, 11:19 AM

For your configuration (NVMe and <1,000 tablets), a StarRocks BE node is expected to become healthy within 30 seconds to 2 minutes. A 20-minute startup is roughly 10x to 40x longer than normal. Here is the typical breakdown of a healthy startup for your scale: Normal Startup Timeline (Expected) * 0–5 seconds: Process initializes, logs are created, and internal RPC/HTTP services start. * 5–15 seconds: BE scans the local metadata (RocksDB) on your NVMe to load tablet headers. With only 1,000 tablets, this should be nearly instantaneous. * 15–30 seconds: BE sends its first "Heartbeat" to the FE and its first "Full Tablet Report." * 30–60 seconds: FE processes the report and marks the BE as

Alive: true

. Why yours is taking 20 minutes Since your tablet count is low and disks are fast, the delay is almost certainly not caused by reading data. It is likely one of these "stuck" states: 1. Index Rebuilding (Primary Key Tables): If you use Primary Key tables and the BE was force-killed (due to the Docker 10s timeout), the BE must rebuild the index in memory. For a large dataset (even with few tablets), this can take 15+ minutes of 100% CPU and heavy I/O. * Check: Look for

rebuild index

<http://be.INFO|be.INFO>

. 2. Network/DNS Timeout Loop: In Docker, if the BE is trying to reach the FE at an old IP or a hostname that isn't resolving, it may retry with long backoffs. * Check: Look for

Master FE not found

RPC timeout

<http://be.INFO|be.INFO>

. 3. FE Metadata Lock: If your FE is busy with a massive load job or a schema change, it may put the BE's "Tablet Report" into a queue. The BE process is "up," but the FE hasn't "accepted" it yet. * Check: Run

SHOW PROC '/backends'\G

and look at the

LastStartTime

. If the start time was 19 minutes ago but it's still not

Alive

, the FE is the bottleneck. How to get to <2 minute startups: 1. Graceful Shutdown: Increase your Docker

stop_grace_period

to 2m or 5m. This ensures the BE saves its Primary Key index to the NVMe, so it can load it in seconds on the next boot instead of rebuilding it for 20 minutes. 2. Persistent Index: If using Primary Key tables, ensure

"enable_persistent_index" = "true"

is set in your table properties. This stores the index on your NVMe. 3. Priority Networks: Ensure

priority_networks

is set in

be.conf

so the BE doesn't waste time trying to register incorrect Docker bridge IPs. Summary: For your hardware and scale, anything over 3 minutes is a sign of an unclean shutdown or a network configuration error. References • faq/Deploy_faq.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md

Francisco Silva

06/19/2026, 11:37 AM

@Rocky is there a way to one-time force the garbage collection in a BE node?

Rocky

06/19/2026, 11:38 AM

There is no single "one-click" command to trigger a global garbage collection (GC) for a BE node. However, depending on what you are trying to "clean up" (orphaned files, disk space, or memory), you can force a cleanup using the following methods: 1. Force Disk Space Cleanup (Clear
trash
directory) When you drop tables or partitions, StarRocks moves the data to a

trash

directory for a safety period (default 24 hours). * The "Hack" to Force it: You can temporarily change the expiration config to

second using the BE HTTP API. This forces the background sweep thread to delete everything in the trash on its next pass.

Copy code

bash
    # Set expiration to 1 second
    curl -X POST http://<be_ip>:<be_http_port>/api/update_config?trash_file_expire_time_sec=1

    # Wait a minute, then set it back to the default (86400)
    curl -X POST http://<be_ip>:<be_http_port>/api/update_config?trash_file_expire_time_sec=86400

* The Manual Way: You can safely run

rm -rf

on the

trash

subdirectory inside your BE storage path (e.g.,

/data/starrocks/storage/trash/*

). 2. Force Data Compaction (Merge versions) If "GC" refers to reclaiming space from deleted rows or old versions within a table, you must trigger Compaction. * For the whole table (Shared-data):

Copy code

sql
    ALTER TABLE <table_name> COMPACT;

* For a specific tablet (Shared-nothing/Local): You can trigger a "Base Compaction" (the most thorough cleanup) via the HTTP API:

Copy code

bash
    curl -X POST "http://<be_ip>:<be_http_port>/api/compact?tablet_id=<tablet_id>&compaction_type=base"

3. Force Orphaned File Scan The BE runs a background thread called

path_gc_check

to find files on disk that are no longer in the metadata. This usually runs once every 24 hours. * To force this more frequently, you can dynamically update the sweep intervals:

Copy code

bash
    # Reduce the max interval to force a run soon
    curl -X POST http://<be_ip>:<be_http_port>/api/update_config?max_garbage_sweep_interval=60

4. Memory "GC" Since StarRocks BE is written in C++ and uses Jemalloc, there is no JVM-style "Full GC." * Memory is managed by Jemalloc's "dirty page" release mechanism. * If you see high memory usage that isn't dropping, it is usually because Jemalloc is holding onto "cached" memory. You can inspect this via the BE's

/metrics

/pprof/heap

endpoints, but there is no manual "release all" command other than restarting the process. Summary of Recommended "One-Time" Force: If your goal is immediate disk reclamation, the most effective one-time action is: 1. Check the

trash

directory size. 2. If large, run the

trash_file_expire_time_sec=1

API call mentioned above. 3. If space is taken by many rowset versions, run

ALTER TABLE ... COMPACT

. References • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • administration/http_interface.md • administration/management/compaction.md • faq/Others.md References • administration/management/BE_configuration.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/compaction.md • faq/operation_maintenance_faq.md

Francisco Silva

06/19/2026, 12:24 PM

@Rocky BE node takes more than 10 minutes to become healthy and I see no CPU load, no memory load, no disk load, nothing all I get is E20260619 121302.284268 139849067027840 starrocks_be.cpp:329] BE exit step 13: cache env destroy successfully W20260619 121552.566252 22848871564864 kv_store.cpp:327] rocksdb iterate timeout: 66341, limit: 60000

Rocky

06/19/2026, 12:25 PM

The logs you are seeing, specifically the rocksdb iterate timeout
, provide a critical clue. This confirms that the BE is not "doing nothing"; it is stuck trying to read its own internal metadata from the local disk. Even with only 1,000 tablets and an NVMe drive, a 60-second timeout (

limit: 60000

) for a metadata scan is extreme. Here is why this is happening and why you aren't seeing CPU or Disk "load": 1. The "Hidden" Metadata Bloat The

rocksdb iterate timeout

occurs when the BE scans its local

KVStore

(located in your storage path under

/meta/

). * The Cause: If you have had many frequent small loads or many deleted tables/partitions, the metadata store can accumulate a massive number of "stale" keys or rowset versions. * The Symptom: When the BE starts, it must iterate through every key to rebuild its memory state. If there are millions of tombstoned or stale keys, the scan takes forever. * Why no load? This is often a Single-Threaded operation. One CPU core might be busy, but on a 16 or 32-core machine, your total CPU usage will look like 3-5%. Because it's a "seek-and-read" pattern on metadata, it may not push enough MB/s to show up as "Disk Load" in basic monitors, but the disk latency for those specific metadata reads is what's killing the startup. 2. Interaction with the Docker Shutdown The log

BE exit step 13: cache env destroy successfully

shows the BE was in the middle of a shutdown sequence. If Docker killed the process (the 10s timeout we discussed earlier) while RocksDB was performing a "compaction" of this metadata, the RocksDB state is left "unclean." On the next boot: 1. RocksDB must replay the WAL (Write Ahead Log). 2. It must then perform the heavy iteration that is now timing out. Diagnostic Steps To confirm and fix this, please check the following: 1. Check the Meta Directory Size: Go into your BE storage directory and check the size of the

meta

folder:

Copy code

bash
    du -sh <storage_root_path>/meta

For 1,000 tablets, this should be under 1GB. If it is 10GB+, your metadata is severely bloated. 2. Look for "Rebuild" or "Load" logs: Search your

<http://be.INFO|be.INFO>

for these specific phrases around the time of the timeout: *

loading tablet meta...

rebuild index

(if using Primary Key tables) *

RocksDB error

3. Check for Disk Latency (iowait): While the BE is starting, run

iostat -x 1

top

. Look at the wa
(iowait) percentage. If it is high even with low throughput, your NVMe is struggling with the specific metadata access pattern (likely due to filesystem overhead or Docker volume driver issues). How to Fix 1. Increase Docker `stop_grace_period`: As mentioned before, set this to 5m. This allows the BE to finish its metadata checkpointing, which significantly speeds up the next startup's RocksDB scan. 2. Metadata Cleanup: If the metadata is bloated, the only "natural" way to fix it is to let the BE successfully finish a startup and stay running. It will eventually run a "Meta Compaction" in the background. 3. The "Nuclear" Option: If the BE simply won't start after 20+ minutes and keeps timing out, you may have metadata corruption. In this case, you would need to: * Stop the BE. * Delete the

storage

directory contents (WARNING: This deletes local data replicas). * Start the BE. It will register as a "new" node and the FE will automatically re-replicate the data from other nodes to it. (Only do this if you have a replication factor > 1). Summary: The BE is stuck in a single-threaded metadata scan. This is almost always caused by an unclean shutdown in a previous session. Increasing the Docker stop timeout is the primary preventative measure. References • administration/management/BE_configuration.md • faq/Deploy_faq.md

Rocky

06/19/2026, 12:25 PM

• faq/operation_maintenance_faq.md • administration/management/graceful_exit.md -measure. References • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • sql-reference/System_variable.md

Francisco Silva

06/19/2026, 12:26 PM

@Rocky where can I see the normal size of metadata and how to reduce it?

Rocky

06/19/2026, 12:26 PM

In StarRocks, the BE metadata (internal KVStore) is stored in the meta
directory inside each of your

storage_root_path

locations. This metadata tracks tablet headers, rowset versions, and transaction states. 1. Where to see the size To check the current size of your BE metadata, log into your BE server and run:

Copy code

bash
# Replace /data/starrocks/storage with your actual storage_root_path
du -sh /data/starrocks/storage/meta

What is "Normal"? * For 1,000 tablets: The

meta

directory should typically be between 100MB and 1GB. * Abnormal: If it is 5GB or larger for only 1,000 tablets, you have "Metadata Bloat." This happens when you have a high frequency of small loads (creating thousands of "Rowset" entries) or if the node is frequently killed before it can compact its internal KVStore. ────────── 2. How to reduce metadata size There is no direct SQL command to "compact" the metadata KVStore, but you can trigger the processes that lead to its reduction: A. Trigger "Base Compaction" on Large Tables The metadata size is directly proportional to the number of Rowsets (versions) per tablet. 1. Identify tables with high version counts:

SHOW PROC '/backends';

(Look at the

TabletNum

and compare with query latency). 2. Force a full merge of versions:

Copy code

sql
    -- This merges many rowset metadata entries into one,
    -- which significantly shrinks the KVStore size.
    ALTER TABLE <table_name> COMPACT;

B. Adjust the "Checkpoint" frequency You can tune the BE to commit metadata to disk more aggressively so that it doesn't have to scan as much during startup. Add/update these in `be.conf`: * tablet_meta_checkpoint_min_new_rowsets
(Default: 10): This determines how many new rowset versions are allowed before the BE checkpoints metadata. Reducing this (e.g., to

) makes the BE save its state more often. C. Clean the "Trash" (Reclaim Disk, not Meta) While this doesn't shrink the

meta

directory itself, it clears the

trash

directory which often holds the actual data files associated with old metadata.

Copy code

bash
curl -X POST http://<be_ip>:<be_http_port>/api/update_config?trash_file_expire_time_sec=1
# Wait 10 minutes, then set back to 86400

D. The "Nuclear" Option (Metadata Reset) If your BE is perpetually timing out at 20 minutes and the

meta

folder is huge, the most efficient "fix" is to let the FE re-build the node. Requirement: Your data must have at least 2 or 3 replicas. 1. Stop the BE process. 2. Delete the entire

storage

directory (or just the

meta

and

data

subdirectories). 3. Start the BE. 4. The BE will register as "empty." The FE will notice the missing replicas and automatically stream the data (and clean metadata) from other healthy nodes to this BE. This is often faster than waiting for a bloated RocksDB to iterate. Summary Recommendation Since you are seeing a

rocksdb iterate timeout

, your BE is struggling to read a massive log of metadata changes. Increase your Docker
stop_grace_period
to 5 minutes immediately. This prevents the bloat from happening again by allowing the BE to finish "checkpointing" its metadata during shutdown. References • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • administration/management/compaction.md • administration/Meta_recovery.md References • faq/Others.md • administration/Meta_recovery.md • administration/management/FE_configuration.md

Rocky

06/19/2026, 12:26 PM

• administration/management/BE_configuration.md

Francisco Silva

06/19/2026, 12:27 PM

@Rocky my question is where are you getting that 1GB value from exactly - be concise and direct

Rocky

06/19/2026, 12:27 PM

The "1GB" value for 1,000 tablets is an empirical rule of thumb rather than a hard-coded limit in the documentation. It is derived from the following technical factors: 1. Metadata Overhead per Tablet: Each tablet in StarRocks has a metadata header (typically ~10–100 KB). 1,000 tablets base overhead is ~100 MB. 2. Rowset Versions: The bulk of metadata growth comes from "Rowsets" (data versions). If you have 1,000 tablets and each has 50–100 active rowset versions (typical for healthy ingestion), the metadata folder grows toward 1 GB. 3. RocksDB Write Buffer: In StarRocks v3.5.0+, the configuration

rocksdb_max_write_buffer_memory_bytes

defaults to 1GB. This is the memory limit for the metadata RocksDB instance before it must flush to disk, reflecting what the system considers a "large" amount of metadata to hold at once. 4. The "Abnormal" Threshold: In troubleshooting scenarios, if a node with only 1,000 tablets has a

meta

directory significantly larger than 1–5 GB, it is a definitive sign of metadata bloat (usually caused by too many small versions or failed compactions). Direct Answer: There is no single doc line stating this value; it is a diagnostic heuristic used by StarRocks support and engineers to determine if a BE's internal metadata store is healthy or bloated. If it has accumulated too many stale, uncompacted entries that cause the 60-second RocksDB scan is failing to process. References • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • administration/management/compaction.md • administration/Meta_recovery.md References • faq/Others.md • administration/Meta_recovery.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Open in Slack

Previous Next