@Rocky I didn't enable replication for my primary key table in shared nothing architecture. If one BE node goes down, I understand why writes fail due to zero replication. However, reads still fail with the Exec OlapScanNode error right? How can I just bypass this node's data and still allow queries to alive nodes?

In a StarRocks shared-nothing architecture, if you set

replication_num = 1

, every tablet exists on exactly one Backend (BE). When that BE goes down, its data becomes completely unavailable, causing the

Exec OlapScanNode

error because the FE cannot find a healthy replica to scan. To bypass the unavailable data and allow queries to continue on the remaining alive nodes (with partial results), you can use the following configuration: 1. Enable Empty Tablet Recovery StarRocks has a specific Frontend (FE) configuration that allows queries to proceed even if some tablets are missing or corrupted. By default, this is

false

to ensure data consistency. You can enable it dynamically without restarting the FE:

sql
ADMIN SET FRONTEND CONFIG ("recover_with_empty_tablet" = "true");

* What this does: If a tablet replica is lost (because the only BE holding it is down), StarRocks will treat that tablet as "empty" and allow the query to execute. * Consequence: Your query will succeed, but the results will be incomplete/incorrect because the data from the offline BE is simply skipped. 2. Identify Missing Tablets To see exactly which tablets are causing the failure, you can check the replica status for your table:

sql
ADMIN SHOW REPLICA STATUS FROM your_table_name WHERE STATUS = "DEAD";

This will help you confirm that the missing tablets are indeed tied to the offline

BackendId

. 3. Considerations for Primary Key Tables Since you are using a Primary Key table, there are extra risks: * Metadata Consistency: Primary Key tables rely heavily on the persistent index and metadata. Running with missing tablets can lead to unexpected behavior if you attempt to write or update data that should have existed on the dead node. * Writes: Even with this setting, writes to those specific tablets will still fail because there is no destination for the data. 4. Better Alternatives for the Future * Increase Replication: For production environments,

replication_num

should always be at least 3. This allows for high availability where one or even two nodes can go down without any read/write interruption. * BE Blacklist (v3.3+): If a node is flapping (unstable network), you can manually blacklist it to prevent the FE from attempting to use it for queries:

sql
    ADD BACKEND BLACKLIST "node_ip:heartbeat_port";

_Note: This only works for routing; if the node holds the only copy of the data, you still need

recover_with_empty_tablet = true

to avoid query failure._ Rocky's Recommendation: Use

recover_with_empty_tablet

only as a temporary emergency measure to get partial data out. As soon as possible, restore the BE or rebuild the table with at least 3 replicas to avoid this "zero-replication" pitfalls. References • using_starrocks/caching/query_cache.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • faq/Sql_faq.md • loading/Flink-connector-starrocks.md • administration/management/resource_management/spill_to_disk.md • loading/Spark-connector-starrocks.md • loading/Kafka-connector-starrocks.md

@Rocky Why can't I just the recover_with_empty_tablet = true as default? This always allows us to query even with dead BE nodes, but we get notified of dead nodes and still will try to bring it back a fast as possible. What are the pitfalls

Setting

recover_with_empty_tablet = true

as a global default is generally considered dangerous for production environments. While it solves the immediate "query failure" problem, it introduces several critical pitfalls, especially for Primary Key tables in a shared-nothing architecture. Here are the reasons why you shouldn't use it as a default: 1. Data Inconsistency (The "Silent Failure") The biggest pitfall is that StarRocks will return partially correct results without warning the end-user. * If a BE node holding 20% of your data goes down, your queries will still return

200 OK

, but the counts, sums, and results will be 20% lower than reality. * In a business environment (e.g., financial reporting or billing), a "successful" query with wrong data is often much worse than a failed query that alerts you to a problem. 2. Primary Key Index Corruption Primary Key tables rely on a Persistent Index to handle updates and deletes. * The Conflict: If you "bypass" a dead node by creating an empty tablet, that new tablet starts with a blank index. * Duplicate Keys: If an update comes in for a key that actually exists on the dead BE, the system will treat it as a brand-new insert because it doesn't see the key in the "empty" replacement tablet. * The Nightmare Reconnect: When the dead BE eventually comes back online, the FE will see two versions of the same tablet: one with the old data and one with the new "empty-start" data. Reconciling these without manual

ADMIN REPAIR

or data loss is extremely difficult. 3. Split-Brain/Metadata Divergence With

replication_num = 1

, the tablet on the dead node is the "source of truth." * If you enable

recover_with_empty_tablet

, you are essentially telling the FE to ignore that source of truth and create a new one. * If you perform any writes while the node is down, those writes go into the "empty" tablet. * When the node returns, the system may struggle to decide which data is valid. You risk permanent data loss if the FE decides to overwrite the "returned" node's data with the "empty/new" tablet's metadata to maintain cluster health. 4. Masking Infrastructure Health Using this as a default often "hides" hardware or network issues. * If a BE is flapping (restarting every 10 minutes), your queries might oscillate between correct and incorrect results. * Because queries aren't "failing," monitoring alerts based on error rates won't trigger, leaving the underlying hardware issue undetected until it's too late. 5. Impact on Materialized Views and Downstream If you have Asynchronous Materialized Views or ETL jobs reading from these tables: * The MV might refresh using the "partial" data. * Once the BE node comes back, the MV is now stale and incorrect, but it won't know it needs to re-run unless you manually intervene. Rocky's Recommendation: Instead of bypassing the failure with

recover_with_empty_tablet

, the standard StarRocks philosophy is: 1. Redundancy over Workarounds: Set

replication_num = 3

. This is the only way to get true High Availability where nodes can fail without query interruption or data loss. 2. Fail-Fast: Keep the setting

false

so that your applications know immediately when the data layer is unhealthy. 3. Manual Intervention Only: Only set

recover_with_empty_tablet = true

as a temporary emergency session variable if you absolutely need to extract what's left of a table that has suffered permanent disk failure. To enable it only for your current session (safer than global):

sql
SET recover_with_empty_tablet = true;
-- Run your emergency query
SELECT ...; 
SET recover_with_empty_tablet = false;

1; -- reset to default

## References - [administration/management/BE_configuration.md](https://docs.starrocks.io/docs/administration/management/BE_configuration) - [release_notes/release-2.4.md](https://docs.starrocks.io/releasenotes/release-2.4)

@Rocky Don't all writes fail if the node is down? So there is no risk?

Sorry, I encountered an error while processing your request: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'Deadline expired before operation could complete.', 'status': 'UNAVAILABLE'}}

@Rocky Don't all writes fail if the node is down? So there is no risk?

You are correct that with

replication_num = 1

, writes will still fail if the node is down, regardless of whether

recover_with_empty_tablet

is set to

true

or

false

. This is because there is no physical location (BE) available to receive and acknowledge the new data. However, setting

recover_with_empty_tablet = true

as a default is still strongly discouraged for several critical reasons: 1. The "Silent Failure" Problem (Most Dangerous) This is the primary reason why it isn't enabled by default. * Without the setting: Your query fails with an error. You know your data is incomplete, and you investigate the node. * With the setting: Your query succeeds, but it returns partially correct results. * Example: You run a sum of daily sales. One node is down. The query returns

$800,000

instead of the real

$1,000,000

. * Because the query didn't error out, your dashboard looks "fine," but you are making business decisions based on incorrect data. In many production environments, no data is better than wrong data. 2. Primary Key Index Integrity Primary Key tables rely on a Persistent Index stored on the BE. * When a node is down, the FE "bypasses" that tablet by treating it as empty. * If you leave this setting on, you might be tempted to "fix" the cluster by forcing a replica migration or repair. If you create a "new" empty replica on a different node to get writes working again: * The new replica starts with a blank index. * All your historical Primary Keys that were on the dead node are now "forgotten" by the system. * If you later recover the old data or perform updates, you will end up with duplicate keys or corrupted delete vectors, breaking the fundamental "Primary Key" guarantee. 3. Metadata and Reconciliation When the dead BE node eventually comes back online: * The FE has been treating those tablets as "empty" for the duration of the downtime. * The returning BE has "stale" data from before it crashed. * StarRocks must reconcile the version history. If you performed any metadata-only operations (like

TRUNCATE

or

DROP PARTITION

) while the node was down and the "empty tablet" logic was active, the reconciliation process can become extremely complex and potentially lead to the FE deciding to wipe the "stale" (but actually valid) data from the returning BE. 4. Masking Infrastructure Issues If this setting is a default, it masks hardware flapping or network instability. * A BE might be crashing and restarting every 5 minutes. * Your monitoring tools (which usually check for query success rates) will show 100% success, even though your data is constantly oscillating between "full" and "partial." You won't realize you have a hardware problem until it becomes a total cluster failure. Recommended Approach Instead of making it a global default, use it as a Session Variable for emergency manual data extraction:

sql
-- Use only when you need to extract whatever data is left
SET recover_with_empty_tablet = true;
SELECT ... ; 
SET recover_with_empty_tablet = false;

Rocky's Bottom Line: The "risk" isn't that you'll accidentally write data; the risk is that you will unknowingly trust incorrect data and eventually corrupt the metadata relationship between the FE and the physical tablets. For

replication_num = 1

, the only safe "default" is for the system to fail and demand a query when data is missing. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • table_design/table_types/primary_key_table.md • administration/management/BE_configuration.md

@Rocky Is there a way to tell if a user is hitting their resource limits? Via resource groups cpu_weight allocation

Yes, you can monitor whether a user or resource group is hitting their limits through several built-in tools. However, it is important to distinguish between soft limits (like

cpu_weight

) and hard limits (like

exclusive_cpu_cores

or

concurrency_limit

), as they "hit" the limit in different ways. 1. Real-Time Monitoring (Is it happening now?) To see current resource consumption and identify if queries are being throttled or queued, use the following commands: *

SHOW USAGE RESOURCE GROUPS;

(Available from v3.1.4+) This provides a live snapshot of CPU and memory usage per group across your Backends (BEs). * Check

BEInUseCpuCores

: If this value is consistently near your expected share (based on

cpu_weight

), the group is "saturated." *

SHOW RUNNING QUERIES;

If you have configured

max_cpu_cores

or

concurrency_limit

for the group, queries will move to a

PENDING

state when the limit is hit. * Check

State

: If you see queries in

PENDING

state, the group has hit its hard resource threshold. *

SHOW PROCESSLIST;

Look at the

IsPending

column. If

true

, the query is waiting for resources from its assigned resource group. 2. Historical Analysis (Did it happen?) If you use the AuditLoader plugin, you can query the

starrocks_audit_db*.starrocks_audit_tbl*

table to analyze historical behavior: * Check

PendingTimeMs

: This field records how long a query waited in the queue. If this value is greater than 0, the user hit a resource group limit (usually concurrency or

max_cpu_cores

). * Analyze

cpuCostNs

: You can compare the total CPU time (

cpuCostNs

) against the wall-clock execution time. If the CPU cost is significantly lower than the duration during high-load periods, it suggests the query was competing for CPU cycles due to the

cpu_weight

scheduling. 3. Monitoring Metrics (Prometheus/Grafana) StarRocks exports specific metrics that are ideal for setting up alerts: *

starrocks_be_resource_group_cpu_use_ratio

: The ratio of CPU time used by the group relative to the total. If this hits a plateau while query latency increases, the group is hitting its

cpu_weight

ceiling. *

starrocks_fe_resource_group_query_queue_pending

: A non-zero value here means queries are actively being queued because the group is over its limit. *

starrocks_be_resource_group_cpu_limit_ratio

: Shows the instantaneous quota ratio. 4. Key Difference: Soft vs. Hard Limits *

cpu_weight

(Soft Limit): You don't "hit" this as a hard wall. Instead, if the cluster is under heavy load, the BE scheduler ensures your group gets at least its weighted share. You will notice this through increased query latency rather than an error or a "pending" status. *

max_cpu_cores

/

concurrency_limit

(Hard Limits): These trigger Query Queuing. You can explicitly see these being hit because queries will show up as

PENDING

and

PendingTimeMs

will increase. Rocky's Tip: If you suspect a user is being throttled by

cpu_weight

, run

EXPLAIN ANALYZE <query>

. Look at the Pipeline Execution metrics in the profile. If "CPU time" is high but "Execution time" is much higher, and the BE logs show high CPU utilization, the

cpu_weight

is doing its job of keeping that user within their allocated "slice" of the CPU. References • administration/management/resource_management/resource_group.md • administration/management/resource_management/query_queues.md • best_practices/audit_log_resource_group.md • administration/management/monitoring/metrics.md

@Rocky Is there grafana metric exposed to measure this? So there's isn't really a way to check for cpu_weight because it's a soft limit? How can I set up alerts for this?

Because

cpu_weight

is a soft limit, a user doesn't "hit" it like a wall. Instead, the resource group is throttled relative to others when the system is under contention. You can measure this in Grafana by comparing a resource group's actual usage against its assigned quota. 1. Key Metrics for Monitoring Throttling StarRocks exposes several metrics to track how much CPU a resource group is consuming compared to its theoretical share. Metric Name | Description

starrocks_be_resource_group_cpu_use_ratio

| Actual Usage: The percentage of CPU cycles currently consumed by this group.

starrocks_be_resource_group_cpu_limit_ratio

| Assigned Quota: The percentage of CPU cycles this group is "entitled" to (calculated from its

cpu_weight

).

starrocks_fe_query_resource_group_latency

| Experience: Query latency percentiles (P95, P99) broken down by resource group. 2. How to "See" the Soft Limit in Grafana To visualize if a user is being throttled by their

cpu_weight

, create a Grafana panel with these two metrics: * Query A:

starrocks_be_resource_group_cpu_use_ratio{name="your_group"}

* Query B:

starrocks_be_resource_group_cpu_limit_ratio{name="your_group"}

How to interpret the graph: * Case 1:

use_ratio

<

limit_ratio

: The user is not hitting their limit. They have room to grow. * Case 2:

use_ratio

>

limit_ratio

: The user is using more than their share because the cluster is idle. This is the benefit of a soft limit! * Case 3:

use_ratio

≈

limit_ratio

AND Latency is rising: This is the "Throttling Zone." The user wants more CPU, but StarRocks is capping them at their weight because other groups are also busy. 3. Setting Up Alerts Since you can't alert on a "hard failure," you should alert on Resource Contention + Latency Degradation. Alert Rule Example: Group CPU Saturation * Condition:

(starrocks_be_resource_group_cpu_use_ratio / starrocks_be_resource_group_cpu_limit_ratio) >= 0.95

* For: 5 minutes * Meaning: This group has been stuck at its maximum weighted share for 5 minutes. If latency is also high, it’s time to increase their

cpu_weight

. Alert Rule Example: Specific User Latency Spike * Condition:

starrocks_fe_query_resource_group_latency{type="95_quantile", name="marketing_group"} > 5000

(e.g., 5 seconds) * Meaning: Even if the group isn't at 100% of its quota, their queries are taking too long. This helps you identify if the soft limit is too restrictive for their specific workload. 4. Advanced: Checking for "Pipeline" Wait If you want to see if the BE is actively delaying tasks for a resource group, look at the Profile of a running query. 1. Run

GET TYPE profile

or check the StarRocks Manager UI. 2. Look for

PipelineTaskWaitTime

. 3. If this value is high while the group's

cpu_use_ratio

is at its

limit_ratio

, the StarRocks scheduler is intentionally making that query wait to let other groups run. Rocky’s Recommendation: Don't alert on

cpu_use_ratio

alone. Always pair it with

starrocks_fe_query_resource_group_latency

. A user at their limit who is still getting fast results doesn't need an alert; a user at their limit with slow results does. References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md

@Rocky What about just using the use_ratio/limit_ratio + CPU saturation? So we know for sure not much idle CPU available and if ratio close to 1 (0.95 <= ratio <= 1.05) then it is hitting limits

Your logic is spot on. Since

cpu_weight

is a relative scheduling mechanism, you can only tell if a user is "hitting" it by looking at their actual share vs. their entitlement during a period of system-wide saturation. Here is how to build that specific alert and the logic behind why it works. 1. The "Throttling" Formula You are correct that the ratio of

use_ratio

to

limit_ratio

is the key. *

resource_group_cpu_use_ratio

: The percentage of CPU cycles this group actually consumed relative to all other groups. *

resource_group_cpu_limit_ratio

: The percentage of CPU cycles this group is entitled to (based on its

cpu_weight

). The Alert Logic: If

(use_ratio / limit_ratio)

is close to 1.0 (e.g., $0.95 \le \text{ratio} \le 1.05$) AND the total BE CPU is saturated ($>90\%$), then the group is definitively hitting its soft limit. 2. Why you need "Total CPU Saturation" * Scenario A (Idle Cluster): If a user is the only one running queries, their

use_ratio

will be 1.0 (100% of all activity), even if their

limit_ratio

is only 0.1 (10% entitlement). In this case, the ratio is 10.0. This is not hitting a limit; it’s the user "borrowing" idle capacity. * Scenario B (Saturated Cluster): If the cluster is at 100% CPU and many users are active, StarRocks will force the user down to their

limit_ratio

. Their

use_ratio

will drop until it matches their

limit_ratio

. The ratio becomes 1.0. This is the limit. 3. Recommended Grafana / Alerting Setup A. The "Saturation Ratio" (PromQL) This measures how close a group is to its fair-share ceiling:

promql
label_replace(
  starrocks_be_resource_group_cpu_use_ratio / starrocks_be_resource_group_cpu_limit_ratio,
  "resource_group", "$1", "name", "(.*)"
)

B. The "Hitting Limit" Alert Set an alert with the following combined condition: 1. Group is at its quota:

(resource_group_cpu_use_ratio / resource_group_cpu_limit_ratio) < 1.1

2. BE is saturated:

avg(starrocks_be_cpu{mode="idle"}) < 10

(meaning < 10% idle CPU) 3. Duration:

For 5m

(to ignore brief spikes) 4. Better Metric for v3.1.4+ If you are on version 3.1.4 or higher, use the newer

resource_group_inuse_cpu_cores

metric. It is much more intuitive than the "ratio of ratios." * Logic: Compare

resource_group_inuse_cpu_cores

directly to the

cpu_weight

you assigned. * Alert:

resource_group_inuse_cpu_cores >= (resource_group_cpu_limit_ratio _ total_be_cores _ 0.95)

Summary for your Alerts Ratio Result | Total CPU Status | Conclusion

1.1

| Any | Borrowing: User is getting more than their weight (This is good!). 0.9 – 1.1 | Low (< 70%) | Under-utilized: User is just using what they need; no contention. 0.9 – 1.1 | High (> 90%) | Hitting Limit: User is being throttled to their fair share. Rocky’s Tip: If you see a user consistently hitting this "1.0 ratio" during peak hours and their queries are slowing down, that is your signal to increase their

cpu_weight

or move them to a group with

exclusive_cpu_cores

. References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/alert.md References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md