http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> in a shared-nothing architecture, c...
# questions-and-troubleshooting
f
@Rocky in a shared-nothing architecture, could there discrepancy in disk usage across BE nodes? StarRocks reports almost same DataUsedCapacity for all BE nodes, but the size of the storage folder in the actual disk differ by 50G in some nodes
r
Yes, it is common and expected to see a discrepancy between the DataUsedCapacity reported by StarRocks and the actual physical size of the storage folder on disk (measured via 
du
). In a shared-nothing architecture, a difference of 50GB across BE nodes is typically not an error but a result of how StarRocks manages data lifecycle and metadata. Here are the primary reasons why your physical disk usage might be higher than the reported `DataUsedCapacity`: 1. The 
trash
Directory When you drop a table, partition, or when StarRocks cleans up old tablet versions, the files are not immediately deleted from the disk. Instead, they are moved to a 
trash
directory within your storage root path. * Reporting: 
DataUsedCapacity
(and 
SHOW DATA
) only accounts for "live" data. It does not include files in the trash. * Retention: Files stay in the BE trash for a default of 24 hours (controlled by the BE config 
trash_file_expire_time_sec
). * Check: Run 
du -sh <storage_root_path>/trash
on the BE nodes to see if this accounts for the 50GB difference. 2. Persistent Indexes (Primary Key Tables) If you use Primary Key tables with 
enable_persistent_index = true
(the default), StarRocks stores the index mapping on disk to save memory. * Storage: These indexes are stored in the 
persist
folder or within tablet sub-directories. * Impact: For large tables, persistent indexes can consume significant space—roughly 
(key size + 8 bytes) _ row count _ 0.5
. * Discrepancy: High-level metrics often prioritize segment data (the actual table rows) over these auxiliary index structures, leading to a gap between reported data and physical usage. 3. Compaction and Versioning StarRocks is based on an LSM-tree structure. When data is updated or imported: * New "versions" of data are created. * Old versions remain on disk until a Compaction task merges them and a subsequent Garbage Collection (GC) cycle removes the obsolete files. * One node might be slightly behind in compaction or GC compared to others, or it might have been assigned tablets that were more "write-heavy," leading to more temporary version bloat. 4. Other Non-Data Files Several other types of files occupy space in the storage folder but aren't counted as "Table Data": * Error Logs: Failed load tasks store sample error data in 
storage/error_log
. * Snapshots: If you performed a backup or a manual snapshot, these files reside in a 
snapshot
directory. * Segment Index Files: In addition to the 
.dat
(data) files, StarRocks creates 
.idx
(index) and footer files. How to Investigate To pinpoint where the 50GB is going, run the following command inside your BE storage directory:
Copy code
bash
du -h --max-depth=1
Commonly, you will find the "missing" space in: 1. 
/trash/
(Safe to clear manually with 
rm -rf
if you don't need to 
RECOVER
data). 2. 
/persist/
(Persistent indexes for PK tables). 3. Individual tablet directories (compaction leftovers). Summary: If 
DataUsedCapacity
is balanced, your logical data distribution is healthy. The 50GB physical difference is likely transient "garbage" in the trash or structural overhead like persistent indexes. References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md • administration/management/BE_configuration.md
f
@Rocky the trash directory is empty on all nodes
r
If your 
trash
directory is empty, the 50GB discrepancy is most likely caused by metadata overhead, auxiliary files, or transient compaction data that StarRocks does not count toward the logical 
DataUsedCapacity
. In StarRocks, 
DataUsedCapacity
(and the output of 
SHOW DATA
) primarily represents the size of the active segment files (the actual table data). Physical disk usage (as seen by 
du
) includes several other components. 1. Persistent Indexes (
/persist
directory) This is the most common reason for a large gap on disk. If you are using Primary Key tables with 
enable_persistent_index = true
(the default), StarRocks stores index mappings on disk to avoid exhausting RAM. * Location: 
${storage_root_path}/persist/
* Behavior: These files can grow significantly (roughly 20-30 bytes per row per replica). They are not included in the table's logical 
DataUsedCapacity
because they are considered index overhead, not raw data. * Check: Run 
du -sh <storage_root_path>/persist
to see if this accounts for the space. 2. Compaction & Versioning "Bloat" StarRocks uses an LSM-tree-like structure. When data is updated or compacted: * Physical: New files are written and old files are marked for deletion. There is a lag between when a file is "logically" replaced and when the BE background thread physically deletes it from the disk. * Metrics: 
DataUsedCapacity
often reflects the Visible Version of the data. If a node is currently under heavy write load or is slow at compaction, it may have many "stale" rowsets that are still taking up space on disk but are no longer "visible" to the system. * Check: Monitor the Compaction Score via 
SHOW PROC '/statistic/compaction_score'
. High scores on specific nodes suggest they are holding onto more intermediate/uncompacted files. 3. Load Error Logs (
/storage/error_log
) If you have had failed 
STREAM LOAD
or 
ROUTINE LOAD
tasks, StarRocks saves samples of the rejected data for debugging. * Location: 
${STARROCKS_HOME}/storage/error_log/
* Retention: These files are kept for 48 hours by default (
load_error_log_reserve_hours
). * Check: 
du -sh <storage_root_path>/error_log
. 4. Snapshots (
/snapshot
) If you have performed manual snapshots or used the 
BACKUP
command recently, temporary copies of tablets are created. * Location: 
${storage_root_path}/snapshot/
* Check: If this directory exists and contains data, it means a backup process or manual snapshot was initiated and perhaps not cleaned up. Recommended Troubleshooting To identify exactly where the 50GB is located, run this command from your BE storage root:
Copy code
bash
du -h --max-depth=1
If the bulk of the 50GB is in a folder named 
data
, it is likely compaction/versioning lag. If it is in 
persist
, it is your Primary Key indexes. Summary: A 50GB difference is not unusual for a large cluster. As long as your 
DataUsedCapacity
is balanced, your data distribution is correct; the difference is purely physical overhead managed by the BE' by the BE. References • introduction/Architecture.md • administration/management/logs.md • table_design/table_design.md • faq/shared_data_faq.md
f
@Rocky whats a good metric to determine if a BE node that went down and now is starting up, how is the replication
r
When a BE node returns after being down, the StarRocks Frontend (FE) automatically checks the version and health of the tablets on that node. If the node was down for a significant time, its data will be "stale" (behind on versions) or "missing" (if the cluster already started repairing elsewhere). The best way to monitor this recovery (replication/repair) is through a combination of 
SHOW PROC
commands and Prometheus metrics. 1. High-Level Summary: 
SHOW PROC '/cluster_balance'
 This provides a bird's-eye view of how many tablets are currently being repaired or are waiting to be processed.
Copy code
sql
SHOW PROC '/cluster_balance';
Look for: * 
pending_tablets
: Tablets waiting for a repair/clone task to be scheduled. * 
running_tablets
: Tablets currently being synchronized (replicated) to or from backends. 2. Active Progress: 
SHOW PROC '/cluster_balance/running_tablets'
 To see exactly what is happening in real-time, including which BE is the source and which is the destination:
Copy code
sql
SHOW PROC '/cluster_balance/running_tablets';
* 
TabletId
: The specific tablet being worked on. * 
DestBe
: If this matches your restarted BE node ID, it means data is being copied to it. * 
Rate
: The speed of the data clone/transfer. 3. Node-Level View: 
SHOW BACKENDS
 You can monitor the gradual recovery of the specific node by watching its tablet count and storage usage:
Copy code
sql
SHOW BACKENDS;
* 
TabletNum
: Compare this to other nodes. If it was down for a long time, this number might start low and slowly climb as the FE re-assigns or repairs tablets to it. * 
DataUsedCapacity
: Monitor this to see the physical data size increasing as clones complete. 4. Monitoring (Prometheus/Grafana) If you use Prometheus, these metrics are the most effective for determining if replication is active: * 
starrocks_be_clone_task_copy_bytes
: (BE metric) The total amount of data being copied by clone tasks. A rising value indicates active replication. * 
starrocks_fe_clone_task_total
: (FE metric) The total number of clone tasks scheduled by the FE. * 
starrocks_be_clone_task_copy_duration_ms
: (BE metric) Helps you see if clones are finishing quickly or getting stuck. Summary of States * If 
pending_tablets
> 0: The FE knows data is missing/stale and is waiting for a "slot" to start the repair. * If 
running_tablets
> 0: The node is actively synchronizing. * If both are 0 and 
TabletNum
is stable: The node is fully synchronized and healthy. Tip: If you want to speed up the recovery, you can increase the 
tablet_repair_delay_factor_sec
in the FE configuration, though the default is usually sufficient for most restarts. References • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md
f
@Rocky I mean prometheus metrics
r
To monitor the replication (cloning/repair) progress of a BE node that has just restarted, you should look at FE metrics for the scheduling status and BE metrics for the actual data transfer. 1. The Scheduling Queue (FE Metrics) These metrics tell you how many tablets the cluster has identified as needing repair and how many are currently being worked on. * 
starrocks_fe_scheduled_pending_tablet_num
: The number of tablets waiting in the queue to be repaired or balanced. If this is high, the FE is aware of the stale data and is waiting for a slot to start the clone. * 
starrocks_fe_scheduled_running_tablet_num
: The number of clone tasks currently in progress across the cluster. * 
starrocks_fe_clone_task_total
: The cumulative count of all clone tasks ever scheduled. * 
starrocks_fe_clone_task_success
: The cumulative count of successfully completed clone tasks. Comparing this to the total count helps you identify if cloning is failing repeatedly. 2. Data Transfer Progress (BE Metrics) These metrics are per-BE node. You should check them on the restarted BE node (which is receiving data) and the source BE nodes (which are providing data). * 
starrocks_be_clone_task_copy_bytes
: The total bytes copied by clone tasks. On the restarting node, a rising value indicates that data is actively being downloaded from other nodes. * 
starrocks_be_clone_task_copy_duration_ms
: The total time spent on copying data. You can use this to calculate the average replication speed (
bytes / duration
). 3. Cluster Health (FE Metrics) To see the "big picture" of how much work is left before the cluster is fully healthy: * 
starrocks_fe_tablet_unhealthy_num
: (Available in most StarRocks versions) The total count of tablets that do not have the required number of healthy replicas. This number should decrease as replication progresses. * 
starrocks_fe_tablet_missing_num
: Specifically counts tablets where a replica is completely missing (likely because the node was down long enough for the FE to start rebuilding them elsewhere). Summary Dashboard View If you are building a Grafana panel, the most useful visualization is: 1. A Time Series of 
running_tablet_num
: To see if the repair is active. 2. A Counter of 
clone_task_copy_bytes
: To see the data throughput. 3. A Gauge of 
unhealthy_num
: To see how close you are to 100% health. Note: If you want to check this via SQL instead of Prometheus, use:
Copy code
sql
SHOW PROC '/cluster_balance/running_tablets';
This will show the specific 
TabletId
and the 
Rate
(copy speed) for every active replication tasks currently running on your BE-to-BE. References • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • faq/operation_maintenance_faq.md • administration/management/resource_management/Replica.md
f
@Rocky if a BE node is down for 15 minutes, writes still ocurr, so when its back up it needs to replicate - I want a metric to see that
r
When a BE node is down for 15 minutes and writes continue, it becomes "stale" because it missed several version updates. Once it restarts, the StarRocks Frontend (FE) detects the version mismatch and schedules Incremental Clone (repair) tasks to bring the node up to date. To monitor this catch-up progress in Prometheus/Grafana, use these specific metrics: 1. The "Workload" Metrics (Are we syncing?) These tell you how many tablets are currently in the process of catching up. * 
starrocks_fe_scheduled_running_tablet_num
: This is the most important gauge. It shows the number of active clone/repair tasks. When the BE restarts, this will spike as the FE starts repairing the stale replicas. * 
starrocks_fe_scheduled_pending_tablet_num
: Shows how many tablets are in the queue waiting for a "slot" to start their catch-up. 2. The "Throughput" Metrics (How fast is it?) These metrics are cumulative counters. You should check them on the restarted BE node to see data moving into it. * 
starrocks_be_clone_task_copy_bytes
: Monitor the rate of change (
irate
or 
rate
) of this metric on the restarted BE. A positive rate means the node is actively downloading the missing rowsets (versions) from other replicas. * 
starrocks_be_clone_task_copy_duration_ms
: Helps you calculate the average speed of the catch-up process. 3. The "Health" Metric (Are we done yet?) * 
starrocks_fe_tablet_unhealthy_num
: This indicates the total number of tablets that do not have the required number of healthy/up-to-date replicas. * Behavior: When the BE is down for 15 minutes, this count will be high. * Recovery: As the BE catches up on versions, this number will decrease toward 0. Quick SQL Verification If you want to see the progress more granularly via SQL while watching the Prometheus graphs:
Copy code
sql
-- See how many tablets are currently "Running" or "Pending" repair
SHOW PROC '/cluster_balance';

-- See the exact copy speed for each tablet currently catching up
SHOW PROC '/cluster_balance/running_tablets';
Summary for your Dashboard: * Gauge: 
starrocks_fe_scheduled_running_tablet_num
(Spike = Active Catchup) * Rate: 
rate(starrocks_be_clone_task_copy_bytes[1m])
(MB/s = Sync Speed) * Status: 
starrocks_fe_tablet_unhealthy_num
(Should trend to 0) References • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/monitoring/metrics.md
f
@Rocky 
starrocks_fe_tablet_unhealthy_num
you sure this exists?
r
You are right to question that—`starrocks_fe_tablet_unhealthy_num` is not a standard Prometheus metric in StarRocks. While the term "Unhealthy Tablet" is frequently used in StarRocks documentation and SQL commands, it is not exported directly as a single Prometheus gauge. If you are looking for that specific name in your Prometheus/Grafana environment, you won't find it. Instead, the "Unhealthy" status is monitored through these two methods: 1. The Prometheus Way (Proxies for Health) Since "unhealthy" typically means a tablet is missing a replica or has a stale version, you monitor the repair process that the FE triggers to fix them: * 
starrocks_fe_scheduled_pending_tablet_num
: This is the best indicator of "work to be done." It counts tablets the FE has identified as needing repair (or balance) but hasn't started yet. * 
starrocks_fe_scheduled_running_tablet_num
: This shows how many "unhealthy" or "stale" tablets are actively being synchronized right now. * 
starrocks_be_clone_task_copy_bytes
: Check this on the restarted BE. If it's increasing, the node is actively receiving the data it missed while it was down. 2. The SQL Way (The Exact Number) If you need the literal "Unhealthy" count that you see in many guides, you must query it via SQL, as it is calculated dynamically by the FE:
Copy code
sql
-- Shows the cluster-wide UnhealthyTabletNum
SHOW PROC '/statistic';

-- Shows which specific tablets are unhealthy in a database
SHOW PROC '/statistic/<db_id>';
Summary for your Dashboard To determine if replication is finished after a 15-minute outage, watch these in Grafana: 1. 
starrocks_fe_scheduled_pending_tablet_num
: Should spike when the node comes back and then trend to 0. 2. 
starrocks_fe_scheduled_running_tablet_num
: Should be greater than 0 while the BE is "catching up." 3. 
starrocks_be_clone_task_copy_bytes
: The throughput (rate) will tell you how fast the data is being replicated. When both 
pending
and 
running
tablet numbers reach 0, the FE considers the cluster healthy and replication complete. References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/monitoring/metrics.md • sql-reference/information_schema/fe_metrics.md
f
@Rocky what metric should I use to detect, for example, if a disk failed on a BE node and the balance/replication tasks dont work
r
To detect a disk failure and the subsequent failure of replication or balancing tasks, you should monitor these specific metrics: 1. Detecting Disk Failure StarRocks monitors the health of each data directory (disk). If a disk becomes read-only or physically fails, the BE will mark it as unhealthy. * 
starrocks_be_disks_state
: This is the most direct metric. * 
1
: Disk is healthy and in use. * 
0
: Disk is unhealthy, disconnected, or failed. _ _Alert Rule Suggestion:* 
starrocks_be_disks_state == 0
* 
starrocks_be_disks_total
& 
starrocks_be_disks_avail
: You can monitor these for sudden drops in total capacity or a disk reaching 100% usage (which often causes "failures" that look like crashes). 2. Detecting Replication/Balance Failures When a disk fails, StarRocks tries to move (clone) the data to other healthy disks or nodes. If these tasks are failing (e.g., due to network issues or no available space), use these metrics: * 
starrocks_be_engine_requests_total{type="clone", status="failed"}
: This is the standard metric for tracking replication errors. * If the 
irate
of this metric is above 0, it means the BE is trying to replicate data but the tasks are crashing or being rejected. * 
starrocks_fe_clone_task_total
vs. 
starrocks_fe_clone_task_success
: * The FE schedules the work. If the 
total
is increasing but the 
success
count is flat, your cluster is stuck in a loop of failing replication tasks. * 
starrocks_fe_scheduled_pending_tablet_num
: * If this number stays high and never decreases, it indicates that the FE is unable to find a valid destination for the replicas (often because other disks are too full or nodes are down). Summary Dashboard Setup Scenario | Prometheus Metric | Condition Dead Disk | 
starrocks_be_disks_state
| 
VALUE == 0
Failed Replication | 
rate(starrocks_be_engine_requests_total{type="clone", status="failed"}[5m])
| 
VALUE > 0
Stuck Balance | 
starrocks_fe_scheduled_pending_tablet_num
| 
VALUE > 0
for long duration Troubleshooting via SQL If the metrics show a failure, you can confirm the exact disk status and error messages using:
Copy code
sql
-- Check disk health/path status
SHOW BACKENDS\G

-- Check why clone tasks are failing (look for 'ErrorMsg' column)
SHOW PROC '/cluster_balance/history_tablets';
References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md
f
@Rocky lets imagine a scenario that for some reason, a BE node comes back up but with a faulty disk, netowkr, something that blocks the replication/clone - meaning the cluster is "yellow" still working but some action is needed What metric/s should I rely on for that type of scenario?
r
In this "yellow" scenario—where the BE is alive but "stuck" or "failing" to catch up—you need to look for a divergence between what the FE wants to do and what the BE is actually accomplishing. Here are the specific metrics to rely on for this scenario: 1. The "Stuck Progress" Metrics (FE Perspective) When replication is blocked, the FE will keep trying to schedule repairs. You will see these numbers stay high instead of trending toward zero: * 
starrocks_fe_scheduled_pending_tablet_num
: This is your primary "Yellow Cluster" indicator. If this stays high for more than a few minutes after a BE has rejoined, it means the FE has tasks queued but they aren't finishing. * 
starrocks_fe_scheduled_running_tablet_num
: If this is 0 but 
pending
is high, the FE can't even find a way to start the tasks (possibly due to "No alive backend" with enough space or compatible versions). 2. The "Failure Rate" Metrics (BE Perspective) If the BE is trying to replicate but failing (due to a faulty disk or network), these counters will increment: * 
starrocks_be_engine_requests_total{type="clone", status="failed"}
: This is the "smoking gun." An increase in this metric means the BE is actively receiving clone instructions but the actual execution is failing. * 
starrocks_fe_clone_task_total
vs. 
starrocks_fe_clone_task_success
: If the gap between these two cumulative counters keeps widening, your repair tasks are failing globally. 3. The "Ghost Disk" Metric If a BE is up but its disk is "faulty" (read-only or corrupted), the BE process might be healthy, but the disk is not: * 
starrocks_be_disks_state
: Check this for any value of 
0
. A BE can be "Alive" in 
SHOW BACKENDS
while having a failed disk. 4. Detecting Network Issues during Clone If the network is the bottleneck or is dropping packets: * 
starrocks_be_clone_task_copy_bytes
: If this remains at 
0
(or the rate is extremely low) while 
starrocks_fe_scheduled_running_tablet_num
is high, it indicates the node is "trying" to copy but no data is moving. ────────── How to Alert on this "Yellow" State You can create a Prometheus alert for a "Stuck Replication" scenario using this logic:
Copy code
promql
# Alert if there are pending tablets AND no successful clones in the last 5 minutes
(starrocks_fe_scheduled_pending_tablet_num > 0) 
AND 
(increase(starrocks_fe_clone_task_success[5m]) == 0)
Investigating the Blockage When you see these metrics, run this SQL command to see the literal error message (e.g., "timeout," "disk full," or "network reset"):
Copy code
sql
-- This shows why the tasks are failing or stuck
SHOW PROC '/cluster_balance/history_tablets';
Look at the 
ErrorMsg
column in the output; it will tell you if it's a 
Network Error
, 
IO Error (Disk failure)
, or 
No Space Left
. References • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md • faq/operation_maintenance_faq.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md • administration/management/monitoring/Monitor_and_Alert.md • faq/operation_maintenance_faq.md • sql-reference/information_schema/fe_metrics.md
f
@Rocky whats a good metric to check the intra BE nodes transfer when replication_num=3
r
To monitor intra-BE node transfer specifically related to replication and cloning (the "repair" or "balance" work that happens when 
replication_num=3
is being enforced), you should use: 1. The "Replication/Repair" Metric * 
starrocks_be_clone_task_copy_bytes
: This is the most accurate metric for background data movement between nodes. * What it measures: The cumulative bytes copied to a BE node from other nodes (Inter-node) or between disks (Intra-node) during tablet repair or balancing. * Why use it: When a node comes back up or a disk fails, this metric will spike on the destination BEs as they "catch up" to reach the 3-replica requirement. 2. The "Load/Ingestion" Metric (Write Amplification) If you are looking for the transfer that happens during active data loading (where 1 write becomes 3 writes across the network): * 
starrocks_be_load_channel_write_bytes
: While this tracks local writes, comparing the 
rate
of this metric across the cluster against your external input rate (e.g., Stream Load bytes) helps you see the 3x amplification. * Query Profile (OlapTableSink): For a specific load job, the query profile is the best place to see this. Look for: * 
BytesSent
/ 
SerializedBytes
in the Exchange Operator or OlapTableSink. These values represent the actual data sent over the wire from the coordinator BE to the various replica BEs. 3. The "Workload" Metric * 
starrocks_be_engine_requests_total{type="clone"}
: This counts the number of replication tasks. * If 
status="failed"
, you have a network or disk issue blocking the transfer. * If 
status="success"
, the intra-node transfer is progressing. Summary for your Dashboard To see the "Yellow to Green" progress for a cluster with 3 replicas: 1. Plot 
irate(starrocks_be_clone_task_copy_bytes[1m])
: This shows the current network throughput (MB/s) being used just for re-replicating data. 2. Monitor 
starrocks_fe_scheduled_pending_tablet_num
: This tells you how many "transfers" are still waiting to start to reach your 
replication_num=3
goal. Note on "Inter" vs "Intra": In StarRocks metrics, Inter-node transfer is what you care about (BE to BE). Intra-node transfer happens if the FE decides to move a tablet from one disk to another on the same BE. Both are bundled into the 
copy_bytes
metric. References • administration/management/monitoring/metrics.md • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/resource_management/Replica.md References • administration/management/monitoring/metrics.md • best_practices/query_tuning/query_profile_operator_metrics.md
f
@Rocky we had a FE node out of 3 misbheaving and the whole cluster writes and operations to BE nodes stopped - why
r
In a StarRocks cluster with 3 FE nodes, the metadata is managed via a quorum-based replication protocol (using BDBJE). For the cluster to function—especially for writes and operations—a majority (at least 2 out of 3) must be healthy and in communication. Here is why a single misbehaving node likely stopped your entire cluster: 1. The Misbehaving Node was the "Leader" Only the Leader FE can process metadata writes (e.g., starting a load, creating a partition, or committing a transaction). * The Zombie State: If the Leader is "misbehaving" (e.g., suffering from a long Stop-the-World GC pause or heavy I/O wait) but has not yet "died" or timed out its heartbeats, the two Followers will not trigger a new election. * Result: The cluster stays in a "zombie" state where the Leader is technically alive but unresponsive, blocking all BE operations and write requests. 2. Loss of Quorum (Double Failure) In a 3-node FE setup, you can only afford to lose one node. * If your "misbehaving" node was one node, but a second node was already silently unhealthy, disconnected, or had a different metadata version (check 
Join: false
in 
SHOW PROC '/frontends'
), the Leader would no longer have a majority. * Result: Without a majority (2/3), the Leader automatically steps down and metadata writes are disabled to prevent data corruption. 3. BDBJE ACK Blocking (Metadata Sync) When the Leader FE writes a piece of metadata (like a load job status), it must receive an acknowledgement (ACK) from at least one other Follower to reach a majority. * The Bottleneck: If one Follower is "misbehaving" (e.g., extremely slow disk I/O on its 
meta_dir
) and the other healthy Follower is also lagging or having network blips, the Leader may hang while waiting for the ACK. * ACK Timeout: By default, the Leader waits for 
bdbje_replica_ack_timeout_second
(default 10s). If it cannot get an ACK in time, it may fail the write and, in some cases, the FE process will exit to protect the metadata. 4. BE Heartbeat Failure BE nodes receive their instructions and "Heartbeats" from the FE Leader. * If the Leader is stuck or misbehaving, it stops sending heartbeats to the BEs. * After a short period, the BEs will mark themselves as "Alive: false" or refuse to accept new load tasks because they cannot verify the cluster's state with the Leader. What to Check in the Logs To confirm which scenario happened, check the 
fe.log
on all three nodes: * Search for 
InsufficientReplicasException
: This means the Leader couldn't find a second node to reach a majority. * Search for 
REPLICA_ACK_TIMEOUT
: This means the Leader was healthy but the other nodes didn't respond fast enough. * Check 
SHOW PROC '/frontends'
: Look for 
Join
, 
Alive
, and 
Role
columns. Ensure all nodes are 
Join: true
and 
Alive: true
. Recommendation If a node is misbehaving and causing a cluster-wide hang, the fastest recovery is often to force-stop the misbehaving FE node. This usually forces the remaining two healthy nodes to immediately elect a new Leader or allows the existing Leader to ignore the dead node and proceed with the remaining healthy Follower. References • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_deadlock_problems.md
f
@Rocky my point is one failing FE node should not stop writes
r
Your logic is correct: in a 3-node FE cluster, the failure of one node should not stop the cluster because you still have a majority (2 out of 3). However, there is a critical distinction between a node that is dead (process stopped, network down) and a node that is "misbehaving" (process alive but extremely slow, "gray failure"). Here is why a single "misbehaving" node can stop writes in a StarRocks cluster: 1. The ACK Blocking Problem (BDBJE Quorum) StarRocks uses BDBJE for metadata replication. When the Leader FE writes metadata (like committing a load transaction), it must receive an acknowledgement (ACK) from at least one Follower to satisfy the quorum (Leader + 1 Follower = 2/3). * The Trap: If one Follower is dead, the Leader ignores it and gets the ACK from the other healthy Follower. Writes continue instantly. * The Delay: If one Follower is "misbehaving" (e.g., severe disk latency or a "Stop the World" JVM pause), the Leader may hang and wait for that specific node to respond. * The Result: The Leader waits for the 
bdbje_replica_ack_timeout_second
(default is 10 seconds). During these 10 seconds, the Leader's metadata thread is blocked. If every write operation hits this 10s timeout, the cluster effectively stops processing writes. 2. "Gray Failure" and the Leader Election If the Leader is the node misbehaving (e.g., it has a 30-second GC pause): * It is not "dead" enough for the Followers to realize it's gone and start a new election immediately. * The Followers wait for the 
heartbeat_timeout_second
to expire before they "rebel" and elect a new Leader. * Until that timeout happens, the entire cluster is waiting on a Leader that is technically "Alive" but completely unresponsive. 3. The "Insufficient Replicas" Loop If you have 3 nodes and Node A is misbehaving, Node B (Leader) tries to sync with A and C. If Node C has even a minor network blip at the same time, the Leader suddenly sees only 1 healthy node (itself). * BDBJE will throw an 
InsufficientReplicasException
. * To protect data integrity, the Leader will stop accepting writes immediately until it can see at least one other node. How to prevent this in the future A "clean death" is always better than a "slow crawl" in HA systems. To mitigate this: 1. Lower the ACK Timeout: You can reduce 
bdbje_replica_ack_timeout_second
in 
fe.conf
(e.g., to 3 or 5 seconds) so the Leader gives up on a slow node faster. 2. Monitor FE Latency: Alert on 
starrocks_fe_edit_log_write_latency_ms
. If this spikes, a node is likely misbehaving. 3. Kill the Zombie: If you identify a misbehaving FE, kill the process manually. Once the node is dead, the BDBJE protocol will immediately stop trying to talk to it and the cluster will resume normal operation with the remaining 2/3 majority. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • knowledge/trouble_shooting/fe_deadlock_problems.md • administration/management/monitoring/metrics.md ) References • administration/management/FE_configuration.md • administration/Meta_recovery.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md ) References
f
@Rocky whats a good metric to check that writes are still happening
r
To verify that writes are still successfully completing in your StarRocks cluster, you should monitor metrics that track transactions and data ingestion throughput. In a 3-node FE setup, most "write success" metrics are only reported by the Leader FE. If you are looking at a dashboard, ensure you are aggregating these or specifically looking at the Leader. 1. The "Transaction Heartbeat" (Best for overall health) This is the most reliable way to see if the metadata "engine" is actually committing work. * 
starrocks_fe_txn_total_latency_ms_count
: * What it is: A counter that increments every time a transaction (Stream Load, Routine Load, or Insert) finishes. * How to use: Use 
irate(starrocks_fe_txn_total_latency_ms_count[1m])
. If this drops to zero, no transactions are finishing. * Note: This is only reported by the Leader FE. 2. Ingestion Throughput (By Load Type) If you want to ensure specific load methods are moving data, use these counters: * Stream Load: 
streaming_load_bytes
(Total bytes) or 
streaming_load_requests_total
(Total attempts). * Routine Load (Kafka): 
starrocks_fe_routine_load_rows
(Total rows consumed). * Insert Into: 
fe_committed_insert_load_job
(Number of successful INSERT statements). 3. The "Publish" Phase (The Final Step) Sometimes data is "committed" but fails to become "visible" because the BE nodes are busy or the Leader is struggling to talk to them. * 
starrocks_fe_publish_version_daemon_loop_total
: This counter increments every time the FE attempts to tell BEs to make new data visible. If this stops incrementing, the write pipeline is stuck at the final step. * 
starrocks_fe_txn_publish_latency_ms
: If this latency spikes, it means transactions are "committing" but taking a long time to become readable. 4. Backend (BE) Side Verification If you suspect the FE is fine but the BEs are the bottleneck: * 
starrocks_be_load_channel_write_bytes
: This shows the raw bytes being written to disk by the BEs during load tasks. * 
starrocks_be_tablet_writer_count
: High numbers here mean the BE is actively writing data. If it's 0 across all BEs, no data is being written. Summary Checklist for your Dashboard: Metric | Source | Significance 
irate(starrocks_fe_txn_total_latency_ms_count[1m])
| FE (Leader) | Crucial. 0 = Writes are completely stopped. 
starrocks_fe_edit_log_write_latency_ms
| FE (Leader) | High = Metadata sync (BDBJE) is slow/failing. 
irate(streaming_load_bytes[1m])
| BE/FE | Shows actual data flow for Stream Loads. 
starrocks_fe_routine_load_jobs{state="RUNNING"}
| FE | Ensures Kafka consumers haven't crashed/paused. Pro-Tip: If 
starrocks_fe_txn_total_latency_ms_count
stops but your load clients aren't reporting errors yet, they are likely stuck in a "hanging" state waiting for the FE Leader to respond to the commit request. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md References • administration/management/monitoring/metrics.md
f
@Rocky is there an HTTP endpoint or something I cause use as a healthcheck for the docker contianer of the FE and BE nodes?
r
Yes, both the FE and BE nodes provide a dedicated HTTP endpoint for health checks. You can use these in your Docker 
HEALTHCHECK
instruction or in Kubernetes liveness/readiness probes. 1. Health Check Endpoints The standard endpoint for all StarRocks node types (FE, BE, and CN) is: 
GET /api/health
Node Type | Default HTTP Port | Health Check URL FE | 
8030
| 
http://<FE_IP>:8030/api/health
BE | 
8040
| 
http://<BE_IP>:8040/api/health
2. Dockerfile Example You can add a 
HEALTHCHECK
to your Dockerfile or 
docker-compose.yml
like this: For FE:
Copy code
dockerfile
HEALTHCHECK --interval=10s --timeout=5s --retries=3 \
  CMD curl -f <http://localhost:8030/api/health> || exit 1
For BE:
Copy code
dockerfile
HEALTHCHECK --interval=10s --timeout=5s --retries=3 \
  CMD curl -f <http://localhost:8040/api/health> || exit 1
3. What the Response Looks Like A successful health check returns an HTTP 200 OK status. The response body is a JSON object:
Copy code
json
{
    "code": 0,
    "message": "OK",
    "result": null
}
If the node is starting up or has encountered a fatal internal error, the endpoint will either be unreachable (connection refused) or return a non-200 status code. 4. Alternative: Metrics Endpoint If you want a more "heavyweight" health check that also verifies the node's internal metrics system is working, you can use: * FE: 
<http://localhost:8030/metrics>
* BE: 
<http://localhost:8040/metrics>
However, 
/api/health
is recommended for standard container health checks as it is lightweight and specifically designed for this purpose. References • administration/http_interface.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/http_interface.md • administration/management/BE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md
f
@Rocky doesnt that reutrn for all Be nodes type? and not the not we are querying itself
r
No, the 
/api/health
endpoint is node-specific. It does not return the status of the entire cluster or other nodes; it only tells you if the specific process you are querying is alive and healthy. Here is how it behaves for FE and BE: 1. BE Health Check (
:8040/api/health
) * Scope: It only checks the local BE process. * Behavior: If the BE process is up and its internal heartbeats/thread pools are functioning, it returns 
{"code":0,"message":"OK","result":null}
. * Usage: This is what you should use for a Docker 
HEALTHCHECK
or K8s probe for a BE container. It ensures that if one BE hangs, only that container is marked unhealthy. 2. FE Health Check (
:8030/api/health
) * Scope: It only checks the local FE process. * Behavior: It returns a simple "OK" if the FE JVM is running and the HTTP server is responsive. It does not check if the FE is the Leader or if the BEs are connected. * Usage: Use this to ensure the FE process hasn't crashed or entered a "Stop the World" GC pause. If you want "Cluster-Wide" status (all BEs): If you were looking for an endpoint that returns the status of all nodes in the cluster (like 
SHOW BACKENDS
), you wouldn't use the health check endpoint. Instead, you would query the FE Leader using the following: * Via SQL API: 
POST /api/v1/sql
with the body 
{"query": "SHOW BACKENDS"}
. * Via Proc API: 
GET /api/show_proc?path=/backends
(This returns a JSON list of every BE in the cluster and their status). Summary for Docker/K8s: * For the Container Healthcheck: Always use 
GET /api/health
on the local port. This prevents "cascading failures" where one bad node makes every other node's health check fail. * For Monitoring/Alerting: Use the Prometheus 
/metrics
endpoint to track the status of all nodes at once via the FE Leader. References • administration/http_interface.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/http_sql_api.md • administration/http_interface.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md ) References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/http_sql_api.md • data_source/data_cache_observe.md • administration/http_interface/query_detail.md
f
@Rocky one a write with 3 FE nodes and 3 BE nodes - according to these docs https://docs.starrocks.io/docs/administration/management/resource_management/Replica/ it seems the reading of the data is done directly to the BE node from the client. is this true?
r
The behavior depends on whether you are writing (ingesting) data or reading (querying) data. Your understanding is partially correct, but the "direct connection" happens differently for each. 1. For Writes (Data Ingestion via Stream Load) Yes, the data transfer is direct to the BE. When you perform a "Stream Load" (the most common write method): * The Redirect: Your client typically sends the initial HTTP request to an FE. * The Handshake: The FE looks at the cluster status and sends an HTTP 307 Redirect back to the client, telling it which BE should coordinate this specific load. * The Data Flow: Your client then connects directly to that BE and streams the raw data to it. The FE is no longer in the middle of the data transfer. * The Finish: That "Coordinator BE" handles the replication to other BEs and sends the success/fail response back to your client. 2. For Reads (Standard SQL via MySQL/JDBC) No, the client usually communicates through the FE. For standard SQL queries (using a MySQL client, JDBC, or BI tools): * The Connection: Your client connects to the FE (typically on port 9030). * The Planning: The FE parses the SQL and tells the BEs which data to scan. * The Data Flow: The BEs execute the query and send the result sets back to the FE. * The Proxy: The FE gathers those results and sends them over its connection to your client. _ _Note:* In this scenario, the client never talks to the BE directly. This is why you only need to expose the FE ports to your end-users/BI tools. 3. The Exception: Arrow Flight SQL (High-Speed Reads) If you use the Arrow Flight SQL protocol (often used for Data Science/Python to move millions of rows quickly): _ StarRocks _can* allow the client to download data directly from the BEs in parallel to bypass the FE bottleneck. * However, if your BEs are in a private network (like a restricted Docker network), StarRocks also has an "FE Proxy" mode for Arrow Flight so the FE can act as the single entry point. Summary Table Operation | Client talks to... | Path of the Data Write (Stream Load) | FE (initially) $\rightarrow$ BE | Direct to BE (after redirect) Read (MySQL/JDBC) | FE | Via FE (FE acts as a proxy) Read (Arrow Flight) | FE $\rightarrow$ BEs | Direct to BEs (parallel download) What this means for your Docker setup: If your client is outside the Docker network, it must be able to reach the BE IP/Port directly for writes to work, because the FE will tell the client "Go talk to BE at 172.18.0.x". If that IP isn't routable from the client, the write will fail. References • loading/StreamLoad.md • introduction/Architecture.md • unloading/arrow_flight.md References • loading/StreamLoad.md • introduction/Architecture.md • unloading/arrow_flight.md • administration/management/resource_management/Replica.md • best_practices/table_clustering.md • sql-reference/http_sql_api.md References • introduction/Architecture.md • sql-reference/http_sql_api.md • loading/StreamLoad.md • quick_start/shared-nothing.md
f
@Rocky how do I know if it is stream load?
r
You are using Stream Load if your data ingestion process meets these specific criteria: 1. The URL and Method You are making an HTTP PUT request to an endpoint that looks like this: 
http://<FE_IP>:8030/api/<db>/<table>/_stream_load
If your code or script (like a 
curl
command) is hitting a URL ending in 
/_stream_load
, it is a Stream Load. 2. The Tools You Use Stream Load is the "under the hood" engine for several common StarRocks tools: * `curl`: If you are running 
curl --location-trusted -u user:passwd -T data.csv ...
. * Flink/Spark Connectors: Most StarRocks connectors for Flink and Spark use the Stream Load API to send data to the BEs. * StarRocks Routine Load: While you configure it via SQL, Routine Load (for Kafka) actually uses the Stream Load engine internally to move data from the BE to the tables. * `stream-load-sdk`: Any Java/Python application using the official StarRocks load SDKs. 3. The HTTP Headers Stream Load jobs are identified by the use of custom HTTP headers to pass metadata. If you see these in your logs or code, it's a Stream Load: * `label`: A unique ID for the job. * `column_separator`: (e.g., 
,
or 
\t
) * `columns`: Defining the mapping of file columns to table columns. * `format`: Usually 
CSV
or 
JSON
. 4. The Response Format Unlike a SQL query that returns rows, a Stream Load returns a JSON object once the upload is finished. It looks like this:
Copy code
json
{
    "Status": "Success",
    "Message": "OK",
    "NumberTotalRows": 1000,
    "NumberLoadedRows": 1000,
    "LoadBytes": 123456,
    "CommitAndPublishTimeMs": 102
}
5. How to Check via SQL If you aren't sure what is happening in the background, you can query the system metadata. Standard 
SHOW LOAD
commands do not show Stream Loads. Instead, run:
Copy code
sql
SELECT * FROM information_schema.stream_loads;
If you see entries there, those are your Stream Load jobs. Summary: Is it Stream Load? Method | Is it Stream Load? 
INSERT INTO ... VALUES (...)
| No (Standard SQL) 
curl ... /_stream_load
| Yes 
LOAD LABEL ... (Broker Load)
| No (Asynchronous background task) Flink/Spark StarRocks Connector | Yes (Usually) References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/information_schema/stream_loads.md References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/information_schema/stream_loads.md • loading/Loading_intro.md • administration/management/monitoring/metrics.md • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/information_schema/stream_loads.md • loading/Loading_intro.md • administration/management/monitoring/metrics.md ) ) References • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • loading/StreamLoad.md
f
@Rocky the replication_num=1 means only one node contains that tablet/data?
r
Yes, that is correct. When you set 
replication_num = 1
, it means that for every tablet (the physical horizontal slices of your table), StarRocks will store only one copy in the entire cluster. What this means in practice: * One Node: Each specific piece of data (tablet) exists on exactly one BE node. * No Redundancy: If the BE node holding that tablet goes down, that data becomes unavailable for queries immediately. * Data Loss Risk: If the disk on that BE node fails or the data is corrupted, that data is permanently lost because there are no other copies (replicas) to recover from. * No High Availability (HA): In a 3-node BE cluster with 
replication_num=1
, the data will be distributed across all three nodes, but each node will hold unique data. If any one node fails, your cluster will report "Tablets are unavailable," and queries will fail. Why the default is 3: StarRocks defaults to 
replication_num = 3
so that: 1. Fault Tolerance: The cluster can survive the loss of up to 2 BE nodes without losing data or stopping queries. 2. Self-Healing: If one node fails, the FE will automatically notice and command the remaining nodes to clone the missing data to bring the count back up to 3. 3. Read Performance: Queries can read from any of the 3 replicas, allowing for better load balancing. When is 
replication_num = 1
okay? It is generally only used for: * Testing/Development: Where data loss isn't a problem. * Temporary Tables: Data that can be easily re-imported from a source (like S3 or Kafka) if a node fails. * Extreme Storage Constraints: If you absolutely cannot afford the disk space for 3 copies, but this is highly discouraged for production. Note: If you are using StarRocks Shared-data mode (where data is stored on S3/Object Storage), this concept is different because the object storage itself handles the redundancy. The above rules apply specifically to the standard Shared-nothing architecture. References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • faq/Sql_faq.md ) References • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • knowledge/usage/best-practice.md • administration/management/FE_configuration.md • faq/Sql_faq.md • introduction/Architecture.md
• sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/http_interface.md • loading/StreamLoad.md • unloading/arrow_flight.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/information_schema/stream_loads.md • loading/Loading_intro.md • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md • administration/http_interface/query_detail.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/http_sql_api.md • best_practices/table_clustering.md • quick_start/shared-nothing.md References • administration/management/resource_management/Replica.md • administration/management/FE_configuration.md • knowledge/usage/best-practice.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • faq/Sql_faq.md
f
@Rocky if the LOG LEVEL IS warn does the be.INFO.log file contain WARN level logs? how about be.out? the docs dont mention these
r
Yes, the 
be.INFO.log
 file (and the 
<http://be.INFO|be.INFO>
 symlink) will contain WARN, ERROR, and FATAL level logs, even if the system log level is set to 
WARN
. StarRocks' Backend (BE) uses the glog library, which follows a hierarchical logging structure. Here is how it works: 1. 
be.INFO.log
and Log Levels * Hierarchical Rule: In glog, a log file of a specific level contains all messages of that level and every level above it. * 
<http://be.INFO|be.INFO>
contains: INFO + WARN + ERROR + FATAL * 
be.WARNING
contains: WARN + ERROR + FATAL * 
be.ERROR
contains: ERROR + FATAL * If `sys_log_level = WARN`: The system tells the logger to discard all "INFO" severity messages. Because no INFO messages are produced, the 
be.INFO.log
file will only record WARN, ERROR, and FATAL entries. It will essentially look identical to 
be.WARNING.log
. 2. What is 
be.out
? The 
be.out
file is a redirect of the process's stdout (Standard Output) and stderr (Standard Error). It is managed by the 
start_be.sh
script rather than the internal StarRocks logging framework. * Purpose: It captures anything that bypasses the logging system. * Typical Content: * Crashes: If a BE process crashes (e.g., Segmentation Fault), the OS or the crash handler prints the stack trace and Query ID that caused the crash to 
be.out
. * Startup Errors: Failures that happen so early that the logging system hasn't initialized yet. * Third-party logs: Output from some underlying C++ libraries that print directly to the console. * Log Level impact: Changing 
sys_log_level
to 
WARN
has no effect on 
be.out
. It will still capture raw console output as usual. Summary Comparison Log File | Level = 
INFO
(Default) | Level = 
WARN
 
be.INFO.log
 | INFO, WARN, ERROR, FATAL | WARN, ERROR, FATAL 
be.WARNING.log
 | WARN, ERROR, FATAL | WARN, ERROR, FATAL 
be.out
 | Stdout/Stderr (Crashes, etc.) | Stdout/Stderr (Crashes, etc.) Pro-tip: If you are troubleshooting a BE crash, always check 
be.out
first. If you are looking for generic system issues (like compaction failures or connection timeouts), check 
be.WARNING.log
. References • administration/management/logs.md • administration/management/BE_configuration.md • faq/Environment_faq.md References • administration/management/BE_configuration.md • administration/management/logs.md • administration/management/FE_configuration.md
f
@Rocky can I force test a WARN to check? right now I have no be.WARN.log file
r
The reason you don't see a 
be.WARNING.log
file yet is likely because your Backend (BE) hasn't encountered a warning or error since it started. StarRocks (using the glog library) only creates the file the moment the first message of that severity is logged. To "force" a message into the 
be.WARNING.log
(and verify it also appears in 
be.INFO.log
), you can use one of these methods: Method 1: The "Tiny Memory" Trick (Safest) By setting a memory limit that is impossibly low, you can force an "Allocation Failed" error. Since ERROR level logs are also written to the WARNING log file, this will trigger the creation of the file. 1. Open your SQL client and run:
Copy code
sql
    -- Set memory limit to 1 Byte
    SET mem_limit = 1;
    -- Run any query (even a simple one)
    SELECT * FROM information_schema.tables LIMIT 1;
2. The query will fail with a "Memory limit exceeded" error. 3. Check your log directory. You should now see 
be.WARNING.log
(containing the error) and 
be.INFO.log
(also containing the error). Method 2: Invalid Configuration via HTTP You can attempt to set a configuration parameter to an invalid value using the BE's HTTP API. The BE will typically log a warning when it rejects the change. Run this from your terminal (replace 
<BE_IP>
and 
<BE_HTTP_PORT>
usually 8040):
Copy code
bash
curl -X POST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?sys_log_level=NOT_A_LEVEL"
Because 
NOT_A_LEVEL
is invalid, the BE should log a warning about the failed configuration update. Method 3: Dynamic Log Level Switch If you want to test the behavior specifically for the 
WARN
level: 1. Change the log level to 
WARN
dynamically:
Copy code
bash
    curl -X POST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?sys_log_level=WARN"
2. Now, any log that would have been a warning will be the only thing recorded. If you then perform an action that triggers a warning (like Method 1), you will see it in both files. 3. Important: Switch it back to 
INFO
when done so you don't lose visibility:
Copy code
bash
    curl -X POST "http://<BE_IP>:<BE_HTTP_PORT>/api/update_config?sys_log_level=INFO"
How to verify they match? Once the file is created, run:
Copy code
bash
grep "Memory limit exceeded" be.WARNING.log
grep "Memory limit exceeded" be.INFO.log
You will see the exact same log line (including the timestamp and thread ID) in both files, proving that the 
be.INFO.log
is indeed a superset of the warning log. References • administration/management/logs.md • administration/management/BE_configuration.md • faq/Environment_faq.md References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/Sql_faq.md • administration/management/logs.md
f
@Rocky the LOG LEVEL change didnt work
r
If the log level change didn't seem to work, it is usually due to one of three reasons: the StarRocks version, the way the change is being verified, or the specific command syntax. 1. Check your StarRocks Version The 
sys_log_level
parameter only became dynamic (changeable without a restart) in the following versions: * v3.3.0 and later * v3.2.7 and later * v3.1.12 and later If you are on an older version (e.g., v2.5 or v3.1.5), the 
curl
command might return 
{"status": "OK"}
but the change will not actually apply. In those versions, you must modify 
be.conf
and restart the BE. 2. Verify the Change via SQL Instead of checking the log files, check what the BE actually thinks its configuration is. Run this query in your SQL client:
Copy code
sql
SELECT * FROM information_schema.be_configs 
WHERE NAME = 'sys_log_level';
* If the 
VALUE
is still `INFO`: The 
curl
command failed or your version doesn't support dynamic changes. * If the 
VALUE
is `WARN`: The change worked. If you still see INFO logs, they are likely "leftover" logs from before the change, or you are looking at a different BE node. 3. Use the correct 
curl
syntax Ensure you are using the BE's HTTP Port (default 
8040
), not the Heartbeat Port (
9050
) or the FE port.
Copy code
bash
# Correct syntax (note the double quotes for the URL)
curl -X POST "http://<BE_IP>:8040/api/update_config?sys_log_level=WARN"
4. Why you might not see the 
be.WARNING.log
yet If you successfully changed the level to 
WARN
, StarRocks will stop writing INFO logs. However, it will not create the 
be.WARNING.log
file until a real Warning or Error occurs. To prove the system is now ignoring INFO logs: 1. Verify 
sys_log_level
is 
WARN
using the SQL above. 2. Tail your `be.INFO.log`: 
tail -f storage/log/be.INFO.log
3. Perform a normal action (like a small query). You should notice that no new lines appear in the log, because those would have been "INFO" lines which are now suppressed. How to fix if it's "Static" If your version is older and the SQL check shows the level hasn't changed: 1. Edit 
be/conf/be.conf
. 2. Add or update: 
sys_log_level = WARN
. 3. Restart the BE process. References • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md • administration/management/logs.md • faq/Environment_faq.md References • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md
f
@Rocky +---------+---------------+-----------------------------------------+--------+---------+---------+ | BE_ID | NAME | VALUE | TYPE | DEFAULT | MUTABLE | +---------+---------------+-----------------------------------------+--------+---------+---------+ | 2335101 | sys_log_level | WARN | string | INFO | 1 | | 2346265 | sys_log_level | NOT_A_LEVEdasfsadfLgkdsahgndfnguspgnr4a | string | INFO | 1 | | 2315376 | sys_log_level | WARN | string | INFO | 1 | +---------+---------------+-----------------------------------------+--------+---------+---------+
r
The fact that your BE accepted that "garbage" string (
NOT_A_LEVEdasf...
) confirms that the StarRocks 
update_config
API for the Backend does not strictly validate the string before storing it in its internal config map. However, accepting the string is not the same as the logger using it. Here is what is happening and how to fix it: 1. Why the "Garbage" Value was accepted StarRocks BE stores configuration as a key-value map. When you send a 
curl
request, it updates that map. The underlying logging library (glog) expects specific values. If you provide an invalid string: * The Config Table: Will show exactly what you sent (the garbage string). * The Actual Logger: Will typically ignore the invalid update and continue using the last valid log level (likely 
INFO
) or default to 
INFO
. This is why you feel the "change didn't work." 2. How to tell if the change actually worked To verify if a BE is respecting the 
WARN
level, look at the first character of the log lines in `be.INFO.log`: * `I`: INFO level * `W`: WARNING level * `E`: ERROR level The Test: 1. Run 
tail -f storage/log/be.INFO.log
. 2. If you see new lines starting with 
I
, the change failed (the BE is still logging INFO). 3. If you only see lines starting with 
W
 or 
E
, the change succeeded. 3. Fixing the Nodes You should reset all nodes to a valid value using the 
curl
command. The 
information_schema.be_configs
table is a view; you cannot 
UPDATE
it directly via SQL in most versions. You must use the API for each BE: For the node with the garbage value:
Copy code
bash
# Use the IP and Port for the specific BE (ID 2346265)
curl -X POST "http://<BE_IP>:8040/api/update_config?sys_log_level=WARN"
4. Why you still have no 
be.WARNING.log
 Even if 
sys_log_level
is set to 
WARN
correctly: * 
be.INFO.log
 will continue to grow (but it will only contain 
W
and 
E
lines). * 
be.WARNING.log
 will not be created until the very first warning occurs. If your cluster is healthy and idle, this file simply won't exist. Summary of your status * BE 2335101 & 2315376: Are correctly set to 
WARN
. They will stop writing 
I
(INFO) lines to 
be.INFO.log
. * BE 2346265: Has an invalid config and is likely still logging everything (including INFO) because it didn't understand the command. Fix this one first. If you still see 
I
lines on the "WARN" nodes: Your StarRocks version might be older than the fix for dynamic logging (v3.3.0, v3.2.7, or v3.1.12). In that case, you must: 1. Edit 
be/conf/be.conf
on each server. 2. Add 
sys_log_level = WARN
. 3. Restart each BE. References • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md • administration/management/logs.md References • administration/management/FE_configuration.md
f
@Rocky I want to force a WARN log to test the logging
r
To force a 
be.WARNING.log
file to be created, you need to trigger a log event with a severity of WARNING or higher (since 
WARNING.log
includes both Warnings and Errors). Since your cluster is currently healthy, the file doesn't exist yet because the logging library (glog) only creates it upon the first relevant event. Use one of these two methods to force it: Method 1: The "Immutable Config" Warning (Cleanest) The BE will log a WARNING if you try to dynamically update a configuration parameter that is marked as "Static" (not mutable). This is a safe way to trigger a warning without failing any queries. 1. Pick a static configuration like 
sys_log_roll_num
(which cannot be changed without a restart). 2. Run this 
curl
command against one of your BEs (replace 
<BE_IP>
and the port, usually 
8040
):
Copy code
bash
    curl -X POST "http://<BE_IP>:8040/api/update_config?sys_log_roll_num=20"
3. The BE will likely return a success message in the JSON, but it will log a warning internally stating that the configuration is not mutable. 4. Check your log directory; 
be.WARNING.log
should now exist. ────────── Method 2: The "Tiny Memory" Error (Most Reliable) In StarRocks' logging system, ERROR logs are automatically mirrored into the WARNING log file. Triggering a memory limit error is the most reliable way to force the file to appear. 1. In your SQL client, run:
Copy code
sql
    -- Set a memory limit that is impossible to satisfy
    SET mem_limit = 1;
    -- Run a query that requires some memory
    SELECT * FROM information_schema.be_configs;
2. The query will fail with 
Memory limit exceeded
. 3. Because this is an 
ERROR
, it will be written to 
be.INFO.log
, 
be.WARNING.log
, and 
be.ERROR.log
simultaneously. ────────── Why your previous attempt looked strange In your 
be_configs
output, one node has a "garbage" value for 
sys_log_level
. This happened because: 1. No Validation: The BE's API currently accepts any string you send to it and stores it in the config map. 2. Fallback: When the logging library tries to read that garbage string, it doesn't recognize it as a valid level (INFO/WARN/ERROR) and usually defaults back to its previous state (likely 
INFO
). 3. Result: You see the garbage in the SQL table, but the BE continues logging everything at the 
INFO
level. To fix that specific node (BE 2346265): Run the 
curl
command again for that specific BE IP to set it back to a valid value:
Copy code
bash
curl -X POST "http://<THAT_BE_IP>:8040/api/update_config?sys_log_level=INFO"
How to verify the logs are working via SQL You can actually query the logs directly from the database to see if the BE has recorded any warnings, even if you can't find the file:
Copy code
sql
SELECT * FROM information_schema.be_logs 
WHERE LEVEL = 'WARNING' 
ORDER BY TIMESTAMP DESC 
LIMIT 10;
If this table returns results, the file must exist on the BE disk in the directory defined by 
sys_log_dir
(default is 
be/log/
). References • administration/management/logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_logs.md References • administration/management/logs.md • administration/management/BE_configuration.md
f
@Rocky I dont want to set the memory limit cause it can break stuff. Is there anything else I can do?
r
To trigger a 
be.WARNING.log
entry safely without risking cluster stability, you can use one of the following "harmless" methods. These trigger a log event by either attempting a forbidden configuration change or forcing a specific non-critical system behavior. Method 1: Attempt to change a "Static" Config (Easiest & Safest) The StarRocks Backend (BE) will log a WARNING if you try to dynamically change a parameter that is marked as non-mutable (static). This does not change the setting and does not affect query execution. 1. Pick a static parameter like 
sys_log_roll_num
(which requires a restart to change). 2. Run this 
curl
command against a BE (replace 
<BE_IP>
with your actual BE IP):
Copy code
bash
    # Try to change a static config to its current or a new value
    curl -X POST "http://<BE_IP>:8040/api/update_config?sys_log_roll_num=10"
3. Check the BE's log directory. The BE should have logged a warning saying 
config name sys_log_roll_num is not mutable
, which will force the creation of the 
be.WARNING.log
file. Method 2: Check the "Garbage" node logs You already have a "safe" test case running! Node 
2346265
has a garbage value (
NOT_A_LEVEdasfs...
). 1. On the server for BE 2346265, look at the 
be.INFO.log
or 
be.WARNING.log
. 2. The BE likely logged a warning when it failed to parse that string into a valid log level. 3. You can search for it: 
grep -i "level" storage/log/be.INFO.log
. Method 3: Use the SQL Log Table You don't even need to look at the disk files to see if a warning was generated. StarRocks exposes the most recent logs via a system table. If this query returns anything, your BEs are successfully logging warnings:
Copy code
sql
SELECT * FROM information_schema.be_logs 
WHERE LEVEL = 'WARNING' 
ORDER BY TIMESTAMP DESC 
LIMIT 5;
Why Method 1 is the best test * Zero Impact: It doesn't use memory, CPU, or change how data is processed. * Specific: It specifically tests the "Configuration" component of the BE. * Immediate: The log is written the moment the 
curl
command is received. Reminder: After you verify that the logs are working, make sure to fix the node with the "garbage" value by setting it to 
WARN
or 
INFO
properly:
Copy code
bash
curl -X POST "http://<BE_IP_OF_2346265>:8040/api/update_config?sys_log_level=WARN"
References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md • administration/management/FE_configuration.md • faq/Environment_faq.md • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md
f
@Rocky root@amplmkt-gcp-prod-usce1-starrocks-cluster00-be02:/opt/starrocks/logs$ curl -X POST "http://127.0.1:8040/api/update_config?sys_log_roll_num=10" { "status": "BAD", "msg": "set sys_log_roll_num=10 failed, reason: Not supported: 'sys_log_roll_num' is immutable" but still no WARN.log
r
The reason you don't see a 
be.WARNING.log
yet is that the "immutable" error message you received was a synchronous HTTP response from the API handler. In StarRocks, these types of API validation errors are often returned directly to the client without being passed to the internal logger as a 
WARNING
event. To force a 
be.WARNING.log
file to be created safely, you need to trigger an event that the BE Execution Engine or Storage Engine considers a warning. Here are the two safest ways to do this without touching memory limits or risking cluster health: Method 1: Stream Load to a Non-Existent Table (Most Reliable) When you attempt a Stream Load to a table that doesn't exist, the BE task scheduler will fail to find the target. This failure is recorded as a 
WARNING
(not just an INFO) because a task failed to execute. Run this from your BE server (replace 
root:
with your actual credentials if needed):
Copy code
bash
curl --location-trusted -u root: -H "label:force_log_test" \
     -T /dev/null \
     "<http://127.0.0.1:8040/api/demo_db/non_existent_table_test/_stream_load>"
* Why it's safe: It attempts to load zero bytes (
/dev/null
) into a table that doesn't exist. It will return an error immediately. * Result: Check your log directory. This usually triggers a 
W[date] ... fragment execute failed
or 
tablet not found
entry, which creates the 
be.WARNING.log
. ────────── Method 2: Check for "Invalid Level" Warnings You currently have a node (
2346265
) with a "garbage" log level. The BE likely logged a warning when it initially tried to process that string. Instead of looking for a new file, check if that specific node already has a warning buried in its 
be.INFO.log
:
Copy code
bash
grep "W" /opt/starrocks/logs/be.INFO.log | head -n 20
In 
glog
(the logging system StarRocks BE uses), all warnings are also written to the INFO log. If you see lines starting with a 
W
, it means warnings are being generated. If the 
be.WARNING.log
file still doesn't exist, it means the system has literally not encountered a single warning-level event since the last time the logs were cleared. ────────── Why the "Immutable" curl didn't work for your test In the StarRocks source code, many API validation checks look like this: 1. User sends request. 2. Code checks 
if (config.is_mutable == false)
. 3. Code returns 
Status::InternalError("is immutable")
to the HTTP response. 4. No 
LOG(WARNING)
call is made because the error was "caught" before it reached the system's core logic. One Final Tip If you truly have no 
be.WARNING.log
and you want to be 100% sure the system is capable of writing it, you can check the 
be_logs
system table to see what the BE "thinks" it has logged:
Copy code
sql
SELECT * FROM information_schema.be_logs 
WHERE LEVEL = 'WARNING' 
LIMIT 1;
If this query returns 0 rows, then the BE has not generated any warnings, and thus the file has no reason to exist. If it does return a row, then look at the 
sys_log_dir
in 
be_configs
to make sure you are looking in the right folder on the disk. References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md References • administration/management/logs.md • sql-reference/information_schema/be_logs.md • administration/management/BE_configuration.md • sql-reference/information_schema/be_configs.md
3 Views
Open in Slack
PreviousNext
Powered by