< Rocky> What are the recommended alerts in FE+CN with verio StarRocks #questions-and-troubleshooting

Join Slack

<@U0A71G31CDV> What are the recommended alerts in ...

# questions-and-troubleshooting

Rumit Jain

06/21/2026, 4:52 AM

@Rocky What are the recommended alerts in FE+CN with verios 4.0.1 shpould be added in promehtues alerts.

Rocky

06/21/2026, 4:52 AM

For StarRocks version 4.0.1 with an FE+CN (Compute Node) architecture, the monitoring focus shifts toward compute efficiency and service availability, especially since CN nodes are typically stateless. The following Prometheus alerts are recommended, categorized by service health, resource usage, and query performance. 1. Service Availability & Status These are critical alerts to ensure the cluster is up and functional. * FE Node Down: Triggers if any Frontend node is unreachable. * PromSQL:

up{group="fe"} == 0

* No Leader FE: Triggers if there is no Leader FE (essential for metadata operations and query coordination). * PromSQL:

sum(starrocks_fe_is_leader) == 0

* CN Node Down: Triggers if a Compute Node is offline. In a CN architecture, scaling is easy, but losing too many nodes impacts concurrency. * PromSQL:

up{group="cn"} == 0

* BE/CN Service Suspension (Internal State): Triggers when the system detects a "dead" node state. * PromSQL:

node_info{type="be_node_num", state="dead"} > 0

2. Resource Utilization (CN & FE) Since CN nodes handle heavy lifting (JOINs, aggregations), CPU and Memory are the primary bottlenecks. * CN CPU High Usage: Alert if CN CPU exceeds 80% for a sustained period. * PromSQL:

100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle", group="cn"}[5m])) * 100) > 80

* CN Memory High Usage: Alert if CN memory usage exceeds 90%. CN nodes are sensitive to OOM (Out Of Memory) during large queries. * PromSQL:

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90

* FE JVM Heap Usage: Alert if FE JVM heap exceeds 80%, which can lead to GC pauses and metadata instability. * PromSQL:

jvm_memory_bytes_used{area="heap", group="fe"} / jvm_memory_bytes_max * 100 > 80

3. Load & Transaction Alerts Even in compute-only nodes, monitoring the data ingestion pipeline is vital. * Loading Failure Rate: Alert if more than 5% of loading transactions fail. * PromSQL:

rate(starrocks_fe_txn_failed[5m]) / rate(starrocks_fe_txn_total[5m]) * 100 > 5

* Routine Load Lag: For Kafka-based ingestion, alert if the consumption lag is too high (e.g., > 100,000 entries). * PromSQL:

starrocks_fe_routine_load_max_lag_of_partition > 100000

* Routine Load Job Paused: Triggers if any Routine Load job enters a

PAUSED

state due to errors. * PromSQL:

starrocks_fe_routine_load_jobs{state="PAUSED"} > 0

4. Query & Performance (FE+CN Specific) In version 4.0.1, monitoring the interaction between FE and CN is key. * High Query Error Rate: Triggers if query errors increase suddenly. * PromSQL:

rate(starrocks_fe_query_err[5m]) > 1

(Adjust threshold based on your baseline). * Connection Limit: Alert if the number of FE connections approaches the system limit. * PromSQL:

starrocks_fe_connection_total / 2000 * 100 > 80

(Assuming a 2000 connection limit). * Shared-Data Cache Performance (if applicable): If you use shared-storage, monitor the Data Cache hit rate on CN nodes. A low hit rate (e.g., < 50%) indicates high S3/Object Storage latency. * PromSQL:

starrocks_be_data_cache_hit_ratio < 0.5

5. Infrastructure Alerts * FE Metadata Disk Space: FE metadata is small but critical. Alert if the disk space on the

meta_dir

is low. * PromSQL:

node_filesystem_free_bytes{mountpoint="/path/to/meta"} / 1024^3 < 10

(Alert if < 10GB). * CN Disk Space (Log/Temp): Although CN is stateless, it needs space for logs and spill-to-disk (if enabled). * PromSQL:

node_filesystem_avail_bytes{group="cn"} / node_filesystem_size_bytes < 0.1

Deployment Tip

Rocky

06/21/2026, 4:52 AM

For version 4.0+, it is highly recommended to use the official StarRocks Grafana Dashboard which includes these PromSQL expressions pre-configured. You can find them in the StarRocks GitHub repository or the monitoring section of the documentation. References • administration/management/monitoring/alert.md

Rumit Jain

06/21/2026, 4:56 AM

@Rocky I dont see

starrocks_be_data_cache_hit_ratio

metrics in prometheus. Is this same starrocks_be_datacache_mem_bytes? If not which metrics should i measure. I am on v 4.0.1 FE+CN in EKS.

Rocky

06/21/2026, 4:56 AM

The metric

starrocks_be_data_cache_hit_ratio

is not a standard Prometheus metric exported directly by StarRocks. Instead, it is usually a calculated value in Grafana dashboards. Furthermore,

starrocks_be_datacache_mem_bytes

(or its likely actual name

starrocks_be_datacache_mem_used_bytes

) is not the same as a hit ratio; it measures the amount of memory currently occupied by the Data Cache on the CN node. To measure the Data Cache hit ratio in version 4.0.1 (FE+CN on EKS), you should use the following metrics and methods: 1. Recommended Prometheus Metrics The CN node exports byte-level counters for cache activity. You can calculate the hit ratio using these: * `starrocks_be_block_cache_hit_bytes`: Total bytes served from the local cache. * `starrocks_be_block_cache_miss_bytes`: Total bytes fetched from remote storage (S3/EKS storage) because they weren't in the cache. 2. How to Calculate Hit Ratio (PromQL) To get the hit ratio over the last 5 minutes, use this formula in Prometheus/Grafana:

Copy code

promql
rate(starrocks_be_block_cache_hit_bytes[5m]) 
/ 
(rate(starrocks_be_block_cache_hit_bytes[5m]) + rate(starrocks_be_block_cache_miss_bytes[5m]))

3. Memory Usage vs. Hit Rate * Memory Usage: Use

starrocks_be_datacache_mem_used_bytes

to monitor how much RAM the cache is consuming. * Disk Usage: Use

starrocks_be_datacache_disk_used_bytes

if you have configured disk-based caching (standard for CN nodes in EKS). 4. Real-time Observation (API) If you want to see the pre-calculated hit rate without using PromQL, you can call the following API directly on a CN node:

Copy code

bash
# From within your EKS cluster
curl http://<CN_IP>:8040/api/datacache/app_stat

This returns a JSON object containing

block_cache_hit_rate

and

block_cache_hit_rate_last_minute

. Summary of Differences Metric Name | Purpose

starrocks_be_datacache_mem_used_bytes

| Capacity: How much memory is currently used.

starrocks_be_block_cache_hit_bytes

| Performance: Counter used to calculate the hit ratio.

starrocks_be_block_cache_hit_rate

| API only: Not typically exported as a Prometheus gauge. Note: Since you are on EKS, ensure your Prometheus scrape configuration is targeting the CN port

to collect these metrics. If you don't see any

starrocks_be_block_cache_*

metrics, check if Data Cache is actually enabled in your CN configuration (

conf/cn.conf

) using

block_cache_enable = true

. References • using_starrocks/caching/block_cache.md • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md

Rumit Jain

06/21/2026, 5:09 AM

@Rocky Whta should be the runbook for

node_filesystem_avail_bytes{group="cn"} / node_filesystem_size_bytes < 0.1

even if I enabled logging in AWS EKS stdout and i dont have spill volume. Only strage0 and storage1

Rocky

06/21/2026, 5:10 AM

In a StarRocks FE+CN (Shared-data) architecture on EKS, even if logs are sent to

stdout

and you haven't enabled "spill-to-disk," your CN nodes still use local storage (

storage0

storage1

) for two critical functions: Data Cache and Primary Key Persistent Indexes. Because the Data Cache uses an LRU (Least Recently Used) mechanism that marks blocks as overwritable rather than deleting files, your Prometheus disk alert (

< 0.1

or > 90% used) will often trigger and stay triggered "by design." Runbook: CN Disk Capacity Alert 1. Identify the Space Consumer Exec into the affected CN pod and check which directory is consuming the space:

Copy code

bash
# Check the storage root paths (storage0, storage1)
du -sh /path/to/storage0/*
du -sh /path/to/storage1/*

* starlet_cache/
: This is the Data Cache. It caches data from S3 to speed up queries. It is normal for this to reach its configured limit (default is 80% of disk) and stay there. * persist/
: This stores Persistent Primary Key Indexes. If you have large Primary Key tables, this directory will grow as your data grows. * core
or
core.<pid>
: If these exist, the CN has crashed recently. Delete them after investigation. * log/
: Even with stdout enabled, some internal or "mini" logs might still be written here. 2. Handle Data Cache (Most Likely Cause) The Data Cache fills the disk until it reaches a threshold, then replaces old data. However, the OS still sees the disk as "full" because the files are not unlinked. * Check actual cache usage: Run

SHOW BACKENDS\G

in MySQL and look at

DataCacheMetrics

. * Action: If the alert is too sensitive (e.g., triggering at 90% while the cache limit is 80%), either: * Increase Disk Capacity: Expand the EBS volumes in EKS. * Lower the Cache Limit: Reduce the threshold so the alert doesn't fire. Set this in

cn.conf

(requires restart) or via SQL (runtime only):

Copy code

sql
        -- Example: Limit cache to 70% of disk to leave headroom for other files
        UPDATE be_configs SET VALUE="70" WHERE NAME="starlet_star_cache_disk_size_percent";

3. Handle Primary Key Indexes If

persist/

is the main consumer: * Check Table Properties: Check if your tables use

enable_persistent_index = true

. * Action: In StarRocks 3.3.2+ (including 4.0.x), you can set

"persistent_index_type" = "CLOUD_NATIVE"

for your Primary Key tables. This moves the bulk of the index to S3, using the local disk only as a small cache. 4. Clean up "Ghost" Files If

du

shows less space than

df

, there may be deleted files still held open by the CN process. * Action:

Copy code

bash
    lsof +L1 # List deleted files held open

If large deleted logs or temp files are held open, you must restart the CN pod to release the space. 5. Adjust Alert Threshold If the CN is performing correctly and the usage is confirmed to be the Data Cache, the alert threshold of 10% remaining may be too aggressive for a cache-heavy workload. * Recommendation: Change the alert to

0.05

(5% remaining) or ensure your

starlet_star_cache_disk_size_percent

is at least 15-20% lower than your alert threshold. Summary of Key Configs Parameter | Default | Purpose

starlet_star_cache_disk_size_percent

| 80 | % of disk used for Data Cache.

storage_root_path

${STARROCKS_HOME}/storage

| Where

starlet_cache

and

persist

live.

datacache_disk_size

| 0 (unlimited) | Hard limit for cache size (e.g., "100G"). *G*"). | References • deployment/shared_data/feature-support-shared-data.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md

Rocky

06/21/2026, 5:10 AM

• administration/management/BE_configuration.md • quick_start/shared-data.md

Rumit Jain

06/21/2026, 5:19 AM

@Rocky What this metrics use for starrocks_be_process_mem_bytes. Can this be used in place of

(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes * 100 > 90

Rocky

06/21/2026, 5:19 AM

The metric

starrocks_be_process_mem_bytes

represents the total physical memory currently being used by the CN/BE process, as tracked by StarRocks' internal memory manager (which includes the memory used by queries, metadata, loading, and the cache). Can it be used in place of the
node_memory
alert? Yes, and in many cases, it is better for StarRocks-specific alerting, especially in an EKS environment. Here is why: 1. Process vs. Host:

node_memory_MemTotal_bytes

measures the memory of the entire EC2 instance. If you have multiple pods or other services on that node, the

node_memory

alert might trigger because of a different process.

starrocks_be_process_mem_bytes

isolates the StarRocks CN node. 2. Internal Limits: StarRocks has its own internal "soft" and "hard" limits (defined by the

mem_limit

parameter in

cn.conf

, usually 90% by default). StarRocks will start failing queries or triggering self-protection based on its internal process memory tracker before the OS actually runs out of memory. 3. EKS Precision: In EKS, a pod's memory is restricted by a Kubernetes limit (cgroups).

node_exporter

often reports the host's total RAM, which might be much larger than your pod's limit, leading to "false negatives" where the pod is about to be OOM-killed by Kubernetes but the node still looks healthy. Recommended Alert Rule for CN Memory Instead of monitoring the node's total memory, you should monitor the CN process memory relative to its specific limit. In StarRocks 4.0.1, you can use:

Copy code

promql
starrocks_be_process_mem_bytes / starrocks_be_mem_limit_bytes > 0.9

_Note: If

starrocks_be_mem_limit_bytes

is not appearing in your Prometheus, you can substitute the denominator with the fixed byte value of your pod's memory limit or the

mem_limit

you set in

cn.conf

._ Which one should you use? Use Case | Recommended Metric "Is the CN process about to crash/fail queries?" |

starrocks_be_process_mem_bytes

"Is the EKS Pod about to be OOMKilled?" |

container_memory_working_set_bytes

(from cAdvisor) "Is the EC2 instance (host) running out of RAM?" |

node_memory_MemAvailable_bytes

(from node_exporter) Summary Runbook Tip If

starrocks_be_process_mem_bytes

is high: 1. Check Query Pool: Use

starrocks_be_query_pool_mem_usage_bytes

to see if a massive query is the cause. 2. Check Metadata: Large numbers of tablets/segments can drive up

starrocks_be_table_meta_mem_usage_bytes

. 3. Manual Check: You can see the detailed breakdown by visiting the CN's internal health page:

http://<CN_IP>:8040/mem_tracker

. Recommendation: Use

starrocks_be_process_mem_bytes

for your primary StarRocks alert, but keep a low-priority alert on

node_memory

to detect if other processes (like the EKS kubelet or log agents) are starving the node resources from the node. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/alert.md

Rumit Jain

06/22/2026, 4:29 AM

@Rocky In place of this : rate(starrocks_fe_query_err[5m]) > 1 (Adjust threshold based on your baseline). Is this metrics not better: starrocks_fe_query_err_rate What should be the runbook when we get this alert.

Rocky

06/22/2026, 4:29 AM

In StarRocks v4.0.1, starrocks_fe_query_err_rate
is generally the better metric to use for alerting because it is a pre-calculated gauge provided by the StarRocks exporter specifically for monitoring. Comparison: Which is better? * rate(starrocks_fe_query_err[5m])
: This uses a Counter (raw total of errors since startup). Prometheus calculates the rate. It is very reliable because it handles process restarts (counter resets) gracefully. * starrocks_fe_query_err_rate
: This is a Gauge pre-calculated by the FE. It represents the failure rate per second (usually over a 1-minute window). * Advantage: It is simpler to use in alerts and matches the "Query Error" dashboard items in official StarRocks Grafana templates. * Note: In your version (4.0.1), this metric is fully supported. Recommendation: Use

starrocks_fe_query_err_rate

for simplicity in dashboards, but for critical alerting,

rate(starrocks_fe_query_err[1m]) > 0.1

(1 error every 10 seconds) is the industry standard for robustness. ────────── Runbook: High Query Error Rate When this alert triggers, follow these steps to diagnose and resolve the issue: 1. Identify the Error Type (Immediate Actions) The most common causes are syntax errors, timeouts, or backend (CN) connectivity issues. * Check the Audit Log: This is the fastest way to see the actual SQL and error message.

Copy code

bash
    # Exec into the FE pod
    grep 'State=ERR' fe/log/fe.audit.log | tail -n 20

Look for the
ErrorCode
and
ErrorMsg
columns at the end of the line. * Use SQL to Query Audit (If AuditLoader is enabled):

Copy code

sql
    SELECT query_id, stmt, error_code, error_msg
    FROM starrocks_audit_db*.starrocks_audit_tbl*
    WHERE state = 'ERR'
    ORDER BY timestamp DESC LIMIT 20;

2. Categorize the Failure * Case A:
ErrorCode: 1064 (Syntax Error)
* Action: This is usually a client-side issue. Contact the application team. If it suddenly started, check for recent app deployments. * Case B:
ErrorCode: 5067 (Timeout)
* Action: Check if the cluster is under heavy load (CPU/Memory). Consider increasing

query_timeout

for specific sessions or globally:

SET GLOBAL query_timeout = 300;

. * Case C:
Internal Error
or
Connection Refused
* Action: This indicates a communication failure between FE and CN. 1. Check CN health:

SHOW BACKENDS;

(Look for

Alive: false

). 2. Check CN logs:

grep -E 'ERROR|WARN' cn.INFO.log

. 3. Check if CNs are being OOMKilled in EKS:

kubectl get pods -n <namespace>

. 3. Deep Diagnosis (Kernel/Planner Issues) If the audit log doesn't explain the error (e.g., "Internal Error" with no detail): * Check
fe.log
: Search for the

query_id

found in the audit log to see the Java stack trace. * Enable Query Dump: If a specific query consistently fails, capture its context:

Copy code

sql
    SET enable_query_dump = true;
    -- Rerun the failing query

The details will be saved in

fe/log/fe.dump.log

. 4. Resolution Steps * Resource Exhaustion: If errors are due to memory/CPU, scale your CN deployment in EKS. * Metadata Locks: If queries are stuck in

PENDING

and then failing, check for long-running

ALTER TABLE

INSERT

jobs that might be locking metadata. * EKS Networking: If "Connection Refused," verify that the EKS security groups and pod-to-pod networking (CN port 8060/9060) are not being blocked. References • administration/management/monitoring/alert.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/logs.md • faq/Sql_faq.md References

Rocky

06/22/2026, 4:29 AM

• administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/alert.md

Rumit Jain

06/22/2026, 4:43 AM

@Rocky What is this metrics

starrocks_fe_query_resource_group_err

and how it is different than starrocks_fe_query_err_rate

Rocky

06/22/2026, 4:43 AM

The key difference between these two metrics lies in their granularity and labels. While both track query failures, they serve different roles in your monitoring strategy. 1.
starrocks_fe_query_err_rate
* What it is: A global metric (Gauge) representing the number of failed queries per second across the entire FE node, typically calculated over a 1-minute window. * Purpose: High-level "Cluster Health" monitoring. It tells you that queries are failing, but not who or which group is affected. * Labels: Usually has no labels or only basic labels like

instance

(the FE host). * Best for: General alerting (e.g., "Alert if any query fails"). 2.
starrocks_fe_query_resource_group_err
* What it is: A multidimensional metric (Counter) that tracks the cumulative number of failed queries broken down by Resource Group. * Purpose: Performance isolation and multi-tenancy troubleshooting. It allows you to see if failures are concentrated in a specific resource group (e.g., the

analytics_team

group is failing while

realtime_ingestion

is healthy). * Labels: It includes a resource_group
label (e.g.,

resource_group="default_group"

resource_group="heavy_queries"

). * Best for: Identifying "noisy neighbors" or group-specific issues. ────────── Comparison Table Feature | starrocks_fe_query_err_rate
| starrocks_fe_query_resource_group_err
Scope | Global (Entire FE) | Segmented (Per Resource Group) Type | Gauge (Pre-calculated rate) | Counter (Total count since startup) Main Label | None/Instance |

resource_group="<name>"

Use Case | P0 Alert: "The system is broken." | Debugging: "Is the Data Science group hitting its limit or failing?" Why use
starrocks_fe_query_resource_group_err
? If you see an alert for

starrocks_fe_query_err_rate

, your first question will be "Which users are affected?" You would then look at

starrocks_fe_query_resource_group_err

in Grafana to see which group's counter is climbing. Common "Resource Group" specific errors included in this metric: * Queries killed because they exceeded the resource group's

cpu_core_limit

mem_limit

. * Queries rejected because the resource group's concurrency limit or query queue was full. * Normal SQL syntax or timeout errors that happened to be executed by a user in that group. Recommended Alerting Strategy 1. Primary Alert (Global): Use

starrocks_fe_query_err_rate > 0.1

to catch any system-wide issues. 2. SLA Alert (Per Group): If you have a critical production group (e.g.,

api_backend

), use:

rate(starrocks_fe_query_resource_group_err{resource_group="api_backend"}[5m]) > 0

This ensures you are notified if your most important "tenant" is experiencing issues, even if the total cluster error rate is low. References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/Monitor_and_Alert.md

Rumit Jain

06/22/2026, 4:47 AM

@Rocky But there is no resource group label in this metrics: starrocks_fe_query_resource_group_err{app_kubernetes_io_component="fe", app_starrocks_ownerreference_name="kube-starrocks", container="fe", endpoint="http", instance="10.0.0.0:8030", job="kube-starrocks-fe-service", name="default_wg", namespace="ABC", pod="kube-starrocks-fe-2", service="kube-starrocks-fe-service"}

Rocky

06/22/2026, 4:47 AM

You are correct that the label name in the metric you see is name
rather than

resource_group

. In StarRocks FE metrics (specifically those starting with

starrocks_fe_query_resource_group_*

), the label identifying the resource group is name
. Why is it
name
? StarRocks internally refers to resource groups as Workload Groups. In the Prometheus exporter for the Frontend (FE), the "name" of that group is exported via the

name

label. * name="default_wg"
: This refers to the Default Workload Group. If you haven't explicitly assigned a query to a custom resource group using a classifier or session variable, it automatically falls into

default_wg

. * name="default_mv_wg"
: You might also see this, which is the default group for Materialized View refresh tasks. How to use this for Alerting Since the label is

name

, your Prometheus query for a specific group should look like this:

Copy code

promql
# Rate of errors for the default resource group
rate(starrocks_fe_query_resource_group_err{name="default_wg"}[5m])

If you want to compare errors across all resource groups, you can sum by that label:

Copy code

promql
# Errors per second across all groups
sum by (name) (rate(starrocks_fe_query_resource_group_err[5m]))

Key Differences Recap Metric | Purpose | Identifying Label

starrocks_fe_query_err_rate

| Global error rate for the entire FE. | (None /

instance

)

starrocks_fe_query_resource_group_err

| Error count broken down by group. | name
Summary: Don't worry that the label isn't literally called `resource_group`; the

name

label is exactly where that information lives. If you see

name="default_wg"

, you are looking at the errors for your primary query group of regular queries. References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/alert.md References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md

Rumit Jain

06/22/2026, 4:50 AM

@Rocky If there is only one name always in this metrics, what is the point of using this other than

starrocks_fe_query_err_rate metrics?

Rocky

06/22/2026, 4:50 AM

If you only see one name (like

default_wg

), it means your cluster is currently using the default configuration without custom Resource Groups. However, there are three main reasons why

starrocks_fe_query_resource_group_err

is often used instead of (or alongside) `starrocks_fe_query_err_rate`: 1. Robustness of Counters vs. Gauges * starrocks_fe_query_resource_group_err
is a Cumulative Counter. In Prometheus, counters are the gold standard because the

rate()

function handles FE restarts (counter resets) automatically without creating "fake" spikes or drops in your graphs. * starrocks_fe_query_err_rate
is an Instantaneous Gauge (a pre-calculated rate from the FE). If the FE process restarts or the network jitters, the gauge value might be lost or misleading. 2. Monitoring Multi-Tenancy (Future Proofing) Even if you only have one group now, StarRocks uses different groups for different types of internal work. As soon as you perform certain actions, you will see more "names": * default_wg
: Regular user queries. * default_mv_wg
: Asynchronous Materialized View refresh tasks. * Custom Groups: If you eventually create groups for "ETL" vs. "Reporting" to prevent one from starving the other, this metric is the only way to see which specific business unit is experiencing errors. 3. Query Flexibility Using the counter allows you to calculate the error rate over any time window in Prometheus, whereas the

_err_rate

metric is hard-coded by the FE (usually to a 1-minute average): * Last 5 minutes:

rate(starrocks_fe_query_resource_group_err[5m])

* Last 1 hour (for daily reports):

increase(starrocks_fe_query_resource_group_err[1h])

Summary Recommendation * Use
starrocks_fe_query_err_rate
for a quick "smoke test" dashboard to see if anything is wrong right now. * Use
starrocks_fe_query_resource_group_err
for your production alerts. It is more accurate across restarts and allows you to distinguish between a failed user query and a failed Materialized View refresh (which have different levels of urgency). References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md • sql-reference/sql-statements/cluster-management/resource_group/CREATE_RESOURCE_GROUP.md

Rumit Jain

06/22/2026, 5:07 AM

@Rocky So which one should be used in as alert and what should be the threashold and its runbook among these 3?

Rocky

06/22/2026, 5:07 AM

For production monitoring, I recommend using these three metrics together to distinguish between "noisy users" and "system instability." 1. Recommendation: Which one for Alerting? For your primary P0 alert, you should use starrocks_fe_query_internal_err
. * Why? Unlike the other two, this metric excludes user errors (like SQL syntax errors or typos). It only increments when a query fails due to a system issue (memory, BE crash, or bug). This prevents "alert fatigue" caused by users writing bad SQL. Metric | Best Used For starrocks_fe_query_internal_err
| P0 Alert: Catching real system bugs or infrastructure failures. starrocks_fe_query_resource_group_err
| Tenant Alert: Catching if a specific team is failing/timed out. starrocks_fe_query_err_rate
| Dashboard Only: Quick visual of the overall failure trend. ────────── 2. Suggested Thresholds Metric | Recommended Alert Rule (PromQL) | Threshold Internal Errors |

increase(starrocks_fe_query_internal_err[1m]) > 0

| Any > 0: Internal errors should ideally never happen. Global Error Rate |

starrocks_fe_query_err_rate > 0.05

| 0.05: This equals roughly 3 failures per minute. Resource Group |

increase(starrocks_fe_query_resource_group_err{name="prod"}[5m]) > 10

| Adjustable: Based on your expected SLA for that group. ────────── 3. The Runbook (What to do when alerted) When one of these alerts triggers, follow these steps in order: Step 1: Identify the failure type Check if the failures are "User Errors" (Syntax/Permissions) or "System Errors" (Timeout/Memory). * Command:

grep 'State=ERR' fe.audit.log | tail -n 20

* What to look for: Look at the

ErrorCode

column. * Error 1064: Syntax error (User's fault). * Error 5021: BE node offline (System issue). * Error 5024: Query Timeout (Resource issue). Step 2: Check for "Noisy Neighbors" Use the Resource Group metric to see if errors are localized. * Action: Look at your Grafana panel for

starrocks_fe_query_resource_group_err

. If only

name="analytics_wg"

is failing but

name="api_wg"

is healthy, you know the issue is specific to those queries, not the whole cluster. Step 3: Check Cluster Health If failures are widespread: * Check BEs:

SHOW BACKENDS;

(Check if

Alive

false

). * Check Memory: Look for "Mem usage has exceeded limit" in

fe.log

<http://be.INFO|be.INFO>

. * Find Big Queries: Run

SHOW PROC '/current_queries';

to see if one massive query is hogging resources and causing others to fail. Step 4: Emergency Mitigation * Kill hung queries:

KILL QUERY <query_id>;

* Scale Timeout: If failures are just timeouts due to high load, temporarily increase the limit:

SET GLOBAL query_timeout = 600;

References • administration/management/monitoring/alert.md • administration/management/monitoring/Monitor_and_Alert.md • sql-reference/Error_code.md • faq/Sql_faq.md References • faq/Sql_faq.md • administration/management/monitoring/alert.md • sql-reference/Error_code.md

Rumit Jain

06/22/2026, 10:03 AM

@Ro

Rumit Jain

06/22/2026, 10:03 AM

@Rocky If we make 2 CN pods and both on spot. How the query will get affected. Like if one spot down and there is a qeury running. Will it transfer to another pod or there will be error

Rocky

06/22/2026, 10:03 AM

If you use 2 CN (Compute Node) pods on Spot instances and one is reclaimed (goes down) while a query is running, the behavior depends on your configuration, but here is the standard breakdown: 1. Will the query transfer or error? By default, the running query will error. StarRocks uses an MPP (Massively Parallel Processing) architecture. When you run a query, the FE (Frontend) splits it into "fragments" and distributes them across all available nodes (the 2 CNs). These fragments communicate with each other in real-time. * The "Pipeline" Breaks: If one CN pod disappears, the network connections between the fragments are broken instantly. Because the intermediate data sitting in the memory of the lost pod is gone, the query cannot simply "hand over" its work to the remaining pod. * The Result: The user will typically see an error like

RPC error

Connection refused

, or

Fragment terminal

. 2. How to achieve "Automatic Retry" While StarRocks cannot resume a query from where it left off, it can automatically restart the query from the beginning on the surviving nodes. You can enable this using the following FE configuration: * max_query_retry_time
: (Default is usually 2). This tells the FE that if a query fails due to a node connection issue, it should try to plan and run the query again from scratch. * How it works with 2 pods: 1. Query starts on CN-1 and CN-2. 2. CN-1 is reclaimed by the cloud provider. 3. Query fails. 4. FE detects CN-1 is down, marks it as "Dead/Blacklisted," and immediately retries the query using only CN-2. 5. The user sees a slightly longer execution time (the time of the first failure + the time of the successful retry), but they do not see an error message. 3. Graceful vs. Abrupt Preemption Spot instances usually provide a warning (e.g., 2-minute notice on AWS, 30 seconds on GCP). * Abrupt Shutdown: If the pod is killed instantly, active queries fail and rely on the

max_query_retry_time

mentioned above. * Graceful Exit (Recommended): You can use Kubernetes PreStop Hooks to trigger a "Graceful Exit." * Since v3.3, StarRocks supports a Graceful Exit mode (

stop_cn.sh -g

). * When triggered, the CN tells the FE: "I am leaving." * The FE stops sending new queries to that CN but allows existing queries to finish during the shutdown window. * This can significantly reduce errors during spot rotations if your queries are shorter than the spot warning period. Summary & Recommendation * Result: Without retries, you get an error. With retries, you get a "latency spike" but a successful result. * Best Practice for Spot CNs: 1. Ensure

max_query_retry_time

is at least

in your

fe.conf

. 2. Use at least 2 pods (as you planned) so that a retry has a node to run on. 3. If possible, use Shared-Data mode (Compute-Storage Separation). CNs are stateless in this mode, making them much faster to scale and safer to lose than traditional BEs that store data. References • administration/management/graceful_exit.md • administration/management/FE_configuration.md • deployment/plan_cluster.md • administration/management/BE_blacklist.md References • administration/management/BE_blacklist.md • deployment/plan_cluster.md • administration/management/graceful_exit.md • introduction/Architecture.md

Rumit Jain

06/23/2026, 4:19 AM

@Rocky In grafana, I want to show number of queries time series per pod in fe+CN, which metrics should I consider?

Rocky

06/23/2026, 4:19 AM

To show the number of queries per pod for both Frontends (FE) and Compute Nodes (CN), you should use the following metrics depending on whether you want to see throughput (how many queries per second) or concurrency (how many queries are active right now). 1. For Frontend (FE) Pods The FE is the entry point for queries. These metrics show queries arriving from users. * Metric Name:

starrocks_fe_query_total

(Cumulative Counter) * Best for: Query Throughput (QPS). * Grafana Query:

Copy code

promql
    sum(irate(starrocks_fe_query_total[5m])) by (instance)

(Note: In Kubernetes,
instance
usually corresponds to the Pod IP/Name.) * Metric Name:

starrocks_fe_unfinished_query

(Instantaneous Gauge) * Best for: Seeing how many queries are currently running on each FE. * Grafana Query:

Copy code

promql
    starrocks_fe_unfinished_query

────────── 2. For Compute Node (CN) Pods CNs do not "own" a query; they process "fragments" of a query sent by the FE. * Metric Name:

starrocks_be_pip_query_ctx_cnt

(Instantaneous Gauge) * Best for: Number of active queries currently being processed by this specific CN pod. * Grafana Query:

Copy code

promql
    starrocks_be_pip_query_ctx_cnt

* Metric Name:

starrocks_be_fragment_requests_total

(Cumulative Counter) * Best for: CN Throughput. Since one query is broken into many fragments, this shows the "workload density" per pod. * Grafana Query:

Copy code

promql
    sum(irate(starrocks_be_fragment_requests_total[5m])) by (instance)

starrocks_fe_unfinished_query

| Gauge Active Queries | CN |

starrocks_be_pip_query_ctx_cnt

| Gauge Query Throughput | FE |

starrocks_fe_query_total

| Counter Workload Throughput | CN |

starrocks_be_fragment_requests_total

| Counter Pro-Tip for Pod Grouping In Grafana, make sure to use

by (instance)

by (kubernetes_pod_name)

depending on how your Prometheus/ServiceMonitor is configured. If you are using the StarRocks Helm chart, the

instance

label will typically show the Pod IP and port (e.g.,

10.x.x.x:8040

). References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/resource_management/query_queues.md References • administration/management/monitoring/metrics.md

Rumit Jain

06/23/2026, 7:58 AM

@Rocky Where to set retry of query in fe config if suddenlyy CN pod goies away due to spot and make fe to retry

Rocky

06/23/2026, 8:02 AM

To enable automatic query retries when a CN (Compute Node) pod disappears (common with Spot instances), you need to configure the max_query_retry_time
parameter in the Frontend (FE). 1. Where to set it? You have two ways to configure this: Option A: Persistent change (
fe.conf
) Add or update this line in your

fe.conf

file on all FE nodes and restart them:

Copy code

properties
max_query_retry_time = 2

* Default:

(StarRocks already tries to retry twice by default). * Meaning: If a query fails due to a connection issue (like a pod being deleted), the FE will automatically restart the query up to this many times. Option B: Runtime change (No restart required) You can apply this change immediately without restarting the FE using the

ADMIN

command:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("max_query_retry_time" = "3");

────────── 2. How it works with Spot Preemption When a CN pod is reclaimed by your cloud provider: 1. Detection: The FE detects an RPC failure or connection loss when trying to talk to that CN. 2. Blacklisting: The FE automatically adds the failed CN to a Blacklist so it doesn't try to use it again immediately. (This is controlled by

black_host_history_sec

, default 120s). 3. Retry: Because

max_query_retry_time

is > 0, the FE re-calculates the query plan, excludes the blacklisted node, and executes the query again on the remaining healthy CN pods. 3. Recommended "Graceful" Setup For Spot instances, relying solely on retries can cause latency spikes. If your cloud provider gives a termination notice (e.g., 2 minutes for AWS Spot), you should also use a Graceful Exit: 1. PreStop Hook: Configure your Kubernetes Pod spec to run

stop_cn.sh -g

when the pod is terminating. 2. Effect: The CN tells the FE it is going offline. The FE stops sending new queries to that node but lets existing queries finish. This prevents the query from failing in the first place if it can finish before the pod is deleted. References • administration/management/FE_configuration.md • administration/management/BE_blacklist.md • administration/management/graceful_exit.md References • administration/management/FE_configuration.md • administration/management/BE_blacklist.md • administration/management/graceful_exit.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md -CONFIG) • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rocky

06/23/2026, 8:02 AM

Rumit Jain

06/23/2026, 8:07 AM

@Rocky In what interval fe going to retry in this case? Is that time configurable?

Rocky

06/23/2026, 8:07 AM

In StarRocks, the query retry after a node failure (like a CN pod being reclaimed) happens immediately. There is no configurable "retry interval" or backoff delay for user queries. Here is how the timing works and what you can control: 1. No Configurable Interval StarRocks does not wait between retries. For analytical queries, the goal is to return a result as fast as possible. Once the Frontend (FE) detects that a query fragment has failed (e.g., due to a connection reset or timeout), it instantly checks if it has retry attempts remaining (based on

max_query_retry_time

). If attempts are left, it triggers the retry logic immediately. 2. The "Detection" Delay The only real "interval" the user sees is the time it takes for the FE to realize the CN is gone. * RPC Timeout: If the pod disappears abruptly, the FE may wait for the RPC timeout (configured by

thrift_rpc_timeout_ms

brpc_send_plan_fragment_timeout_ms

) before it considers the fragment failed. * Re-planning: After failure, the FE must re-calculate which nodes to use. Since you have 2 pods and one is down, it will re-plan to send all fragments to the remaining pod. 3. Avoiding the "Down" Node (Blacklisting) To ensure the retry doesn't attempt to use the same dead pod again, StarRocks uses an automatic Blacklist. This is where the configuration comes in: * black_host_history_sec
: (Default: 120s) This is the duration a failed node stays on the blacklist. * black_host_connect_failures_within_time
: (Default: 5) If a node fails this many times, it is blacklisted. When the query retries, the FE looks at the blacklist and excludes the dead CN pod from the new execution plan. This prevents the query from failing a second time on the same missing node. Summary of Configs for Spot Nodes Parameter | Default | Description max_query_retry_time
|

| Total number of times to try the query. black_host_history_sec
|

| How long to ignore the "dead" spot pod after it fails. Retry Interval |

N/A

| **Not configurable**; retries are triggered immediately after failure detection. Pro-Tip: If your queries are failing because the "Detection" is taking too long (e.g., the query hangs for 60s before retrying), you can lower

brpc_send_plan_fragment_timeout_ms

Rocky

06/23/2026, 8:07 AM

• sql-reference/sql-statements/data-manipulation/RETRY_FILE.md References • administration/management/FE_configuration.md

Rumit Jain

07/23/2026, 3:51 AM

@Rocky Why I am seeing these errors in prometheus?

Copy code

time=2026-07-23T03:48:13.548Z level=WARN source=scrape.go:1906 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/prod/kube-starrocks-fe/0 target=<http://10.X.X.X:8030/metrics> num_dropped=44

Rocky

07/23/2026, 3:52 AM

The error "Error on ingesting samples with different value but same timestamp" in Prometheus occurs when a single scrape (or multiple overlapping scrapes) attempts to save two different data values for the exact same metric and label set at the same moment. In the context of your StarRocks FE, this is usually caused by one of the following three scenarios: 1. Duplicate Metrics in the
/metrics
Output StarRocks sometimes has bugs (like Issue #42753) where the FE or BE accidentally exports the same metric twice in a single HTTP response. If the metric name and all labels (like

table

mv_name

, etc.) are identical but the values differ, Prometheus will reject them. * How to check: Run a curl command against the FE pod IP mentioned in your error:

Copy code

bash
    curl -s http://<FE_POD_IP>:8030/metrics | grep -v "^#" | sort | uniq -d

If this returns any lines, those metrics are being duplicated by StarRocks itself. 2. Label Collisions in
ServiceMonitor
Your error source is

serviceMonitor/prod/kube-starrocks-fe/0

. This suggests that your Prometheus

ServiceMonitor

might be configured in a way that causes multiple FE pods to appear as the same "instance" to Prometheus. * The Cause: If your relabeling rules or the

ServiceMonitor

setup strips away the unique

pod

instance

label, Prometheus will think metrics from FE-0, FE-1, and FE-2 are all the same series. Since they are being scraped at the same time but have different values (e.g., different query counts), Prometheus throws this error. * Fix: Ensure your

ServiceMonitor

configuration includes a label that uniquely identifies the pod (usually

instance

kubernetes_pod_name

). 3. Overlapping Scrape Jobs You might have two different Prometheus jobs scraping the same StarRocks FE. For example: * One job from a

ServiceMonitor

. * Another job from a static config or a different

PodMonitor

. If both jobs assign the same

job

and

instance

labels to the target, the scrapes will collide. Troubleshooting Steps 1. Identify the specific metric: Look further down in the Prometheus logs or check the Targets page in the Prometheus UI. It often lists which metric is "dropped." 2. Verify unique labeling: Go to the Prometheus UI, search for any FE metric (e.g.,

starrocks_fe_query_total

), and see if you see multiple pods with the exact same labels. If you do, you have a label collision. 3. Check StarRocks Version: If you are on an older version of StarRocks, certain Materialized View (MV) metrics were known to duplicate. Upgrading or disabling MV metrics (if not needed) may resolve it. Quick fix for Kubernetes: Check your

ServiceMonitor

and make sure

honorLabels: true

is set, or ensure that your

relabelConfigs

are not overwriting unique pod identifiers.

Rumit Jain

07/23/2026, 3:56 AM

@Rocky Seeing these but only in 1 fe out of 3

Copy code

kubectl exec -it -n prod kube-starrocks-fe-2 -c fe -- /bin/sh
$ curl -s <http://localhost:8030/metrics> | grep -v "^#" | sort | uniq -d
starrocks_fe_thread_pool{name="batch-write-load", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="batch-write-load", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="batch-write-load", type="pool_size"} 0
starrocks_fe_thread_pool{name="batch-write-load", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="cluster_pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="cluster_pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="cluster_pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="cluster_pool", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="connector-trigger-analyze-pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="connector-trigger-analyze-pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="connector-trigger-analyze-pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="connector-trigger-analyze-pool", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="coordinator-be-assigner", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="coordinator-be-assigner", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="hms-event-processor-executor", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="hms-event-processor-executor", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="hms-event-processor-executor", type="pool_size"} 0
starrocks_fe_thread_pool{name="hms-event-processor-executor", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="materialized view_pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="materialized view_pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="materialized view_pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="materialized view_pool", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="port-connectivity-checker", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="port-connectivity-checker", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="query-deploy", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="query-deploy", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="refresh-dictionary-cache-pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="refresh-dictionary-cache-pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="refresh-dictionary-cache-pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="refresh-dictionary-cache-pool", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="schema change_pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="schema change_pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="schema change_pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="schema change_pool", type="task_in_queue"} 0
starrocks_fe_thread_pool{name="starrocks-taskrun-pool", type="active_thread_num"} 0
starrocks_fe_thread_pool{name="starrocks-taskrun-pool", type="completed_task_count"} 0
starrocks_fe_thread_pool{name="starrocks-taskrun-pool", type="pool_size"} 0
starrocks_fe_thread_pool{name="starrocks-taskrun-pool", type="task_in_queue"} 0

Rocky

07/23/2026, 4:00 AM

The error you are seeing in Prometheus is caused by a known bug in the StarRocks FE metrics exporter, where the same metric line is printed twice (or more) in a single

/metrics

response. Prometheus rejects these samples because its time-series database requires every unique combination of a metric name and labels to have exactly one value per timestamp. When StarRocks sends the same line twice, Prometheus sees two values for the same series at the same moment and drops them. Why is this happening on FE-2? In StarRocks, some background services or thread pools are only active on the Leader FE or are initialized differently based on the FE role. 1. Double Registration: It is likely that on FE-2, a component was re-initialized (possibly due to a transient network issue or role change), and it registered its thread pools into the global

MetricRepo

a second time without clearing the first registration. 2. Leader Role: Check if FE-2 is currently the Leader. Run

SHOW FRONTENDS;

in your SQL client. The Leader FE handles more background tasks (like

hms-event-processor

materialized view_pool

), which matches the specific thread pools you see duplicated in your

curl

output. How to fix it 1. Immediate Workaround: Restart FE-2 Since these metrics are stored in the FE's memory registry, restarting the

kube-starrocks-fe-2

pod will clear the duplicate entries and resolve the Prometheus ingestion error immediately.

Copy code

bash
kubectl delete pod kube-starrocks-fe-2 -n prod

2. Short-term Workaround: Filter in Prometheus If you cannot restart the FE immediately, you can tell Prometheus to ignore the

starrocks_fe_thread_pool

metric for now to stop the logs from filling up: In your

ServiceMonitor

or Prometheus config, add a

metric_relabel_configs

to drop that specific metric name. 3. Long-term Fix: Upgrade This issue is related to logic bugs in the FE's metric collection (similar to GitHub Issue #42753 where Materialized View metrics were duplicated). These are typically fixed in later patch releases. * Check your current StarRocks version. * If you are on an older 3.x or 2.x version, consider moving to the latest maintenance release of your branch (e.g., 3.1.x -> 3.1.17+). Summary of your
curl
results: The fact that your

curl

shows duplicates with the value

for pools like

batch-write-load

and

hms-event-processor-executor

Rocky

07/23/2026, 4:00 AM

• introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • introduction/Architecture.md

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rocky

07/23/2026, 4:00 AM

Rumit Jain

07/23/2026, 5:33 AM

@Rocky If I delete the fe leader pod, will it be a downtime?

Rocky

07/23/2026, 5:33 AM

Deleting the FE Leader pod will typically not cause a total cluster downtime, provided you have a High Availability (HA) setup with multiple Follower FE nodes. However, there will be a brief interruption for specific types of operations. Here is what happens when you delete the Leader pod: 1. The Interruption (Failover Window) * Write Operations: There will be a brief downtime for write operations (like

INSERT

LOAD

, or DDL statements). These require a Leader to coordinate. When the Leader pod is deleted, the cluster must elect a new Leader from the remaining Follower nodes. This election usually completes within a few seconds. * Read Operations: If your application connects through a Load Balancer (LB) or Service that points to all FE pods, read queries can continue to be served by the remaining Follower or Observer nodes. * Query Planning: Any query currently being planned or executed specifically on the Leader node at the moment of deletion will likely fail and need to be retried by the client. 2. The Requirement for Quorum To avoid a complete service outage, your FE cluster must maintain a Quorum (more than half of the Follower nodes must be alive). * 3 FE Followers: If you have 3 Followers and delete the Leader, 2 are left alive. This is a quorum (2 > 1.5), so a new leader will be elected successfully. * 2 FE Followers: If you only have 2 Followers and delete the Leader, only 1 is left. This is not a quorum (1 is not > 1), and the remaining FE will stop providing services until the second pod is back and healthy. 3. Metadata Lag (Read Impact) By default, non-leader FEs can serve reads as long as their metadata is not too far behind the Leader (controlled by

meta_delay_toleration_second

, default 300s). If the election takes longer than this (highly unlikely) or if the remaining nodes were already lagging, they might stop serving reads until the cluster stabilizes. Summary Checklist for "No Downtime" To ensure this is as seamless as possible: 1. Check FE Status: Run

SHOW FRONTENDS;

and ensure at least 2 other nodes have the

Role

FOLLOWER

and

Alive

true

. 2. Use a Load Balancer: Ensure your application connects to a VIP or Kubernetes Service that handles the health checks of all FEs. 3. Client Retries: Ensure your client application/driver has basic retry logic to handle the 2-10 second window during the leader election. Verdict: It is a "partial downtime" (briefly for writes, transient errors for active queries) but not a "cluster downtime" if your quorum is healthy. References • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • administration/management/FE_configuration.md References • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • administration/management/FE_configuration.md

Rumit Jain

07/23/2026, 5:51 AM

@Rocky Why I am seeing CN1 when I have only one instance of CN i.e cn0 running?

Copy code

show compute nodes\G;
*************************** 1. row ***************************
        ComputeNodeId: 12050
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.devtest04.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-07-23 02:08:13
        LastHeartbeat: 2026-07-23 05:49:23
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 2.43 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: Status: Normal, DiskUsage: 240.5MB/159.8GB, MemUsage: 2.9MB/2.5GB
       HasStoragePath: true
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 320
*************************** 2. row ***************************
        ComputeNodeId: 12051
                   IP: kube-starrocks-cn-1.kube-starrocks-cn-search.devtest04.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-06-11 06:30:26
        LastHeartbeat: 2026-06-11 06:30:26
                Alive: false
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg: java.net.UnknownHostException: kube-starrocks-cn-1.kube-starrocks-cn-search.devtest04.svc.cluster.local
              Version: 4.0.1-cd9df36
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 0.00 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: N/A
       HasStoragePath: false
           StatusCode: DISCONNECTED
          StarletPort: 9070
             WorkerId: 2
        WarehouseName: default_warehouse
            TabletNum: 0
2 rows in set (0.01 sec)

ERROR:
No query specified

Starrocks >

Rocky

07/23/2026, 5:51 AM

You see cn-1 because StarRocks persists all registered nodes in its internal metadata (on the FE). Even if you scale down your Kubernetes Deployment or StatefulSet, the StarRocks FE does not automatically "forget" the node. It keeps it in the list with an

Alive: false

status so that if the node returns, it can be re-recognized. In your specific case, cn-1 has been disconnected since June 11th (over a month ago), and the

UnknownHostException

confirms that the Kubernetes DNS entry for that pod no longer exists. How to remove it Since CN nodes do not store permanent data (they only host temporary data cache), it is perfectly safe to remove this defunct entry from the metadata using the following SQL command:

Copy code

sql
ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.devtest04.svc.cluster.local:9050";

(Note: 9050 is the
HeartbeatPort
shown in your
SHOW COMPUTE NODES
output.) Why did this happen? 1. Manual Scaling: If the cluster was ever scaled up to 2 CNs and then scaled back down to 1 via

kubectl scale

, the FE was never told to drop the second node. 2. Operator Sync: If you are using the StarRocks Operator, it usually handles this automatically. However, if the pod was deleted manually or if there was a sync issue when the IP/Host became unreachable, the metadata entry can persist. After running the

DROP

command,

SHOW COMPUTE NODES

will only show the single active cn-0 node. References • administration/management/Scale_up_down.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/management/Scale_up_down.md • administration/stargo.md

Rumit Jain

07/23/2026, 8:07 AM

@Rocky What is the negative effect if we keep getting this WARN in prometheus and we do not restart the FE Leader?

Rocky

07/23/2026, 8:07 AM

If you choose not to restart the FE Leader and continue to ignore the Prometheus warnings, the negative effects are primarily related to monitoring blindness rather than cluster stability. The core SQL engine will continue to function, but your ability to manage and debug the cluster will be compromised. The specific negative effects are: 1. Total Data Loss for Duplicated Metrics Prometheus rejects samples when it encounters duplicate labels in a single scrape. This means for all the thread pools listed in your

curl

output (like

batch-write-load

materialized view_pool

, and

starrocks-taskrun-pool

), you will have zero data in Prometheus/Grafana. * Charts for these metrics will appear empty or show "No Data." * You won't be able to see historical trends for these specific pools. 2. Alerting Failure If you have configured alerts based on these metrics (e.g., "High number of tasks in

batch-write-load

queue" or "Materialized View refresh pool is full"), these alerts will never fire. * If your loads start backing up or Materialized Views stop refreshing, you won't receive a notification from Prometheus because the data stream is broken at the ingestion point. 3. "Blindness" During Performance Issues The thread pools currently affected are critical for debugging performance: * batch-write-load: Needed to debug slow Stream Loads or Routine Loads. * materialized view_pool: Needed to see if MV refreshes are stuck. * starrocks-taskrun-pool: Needed to monitor scheduled Tasks. If the cluster starts behaving slowly, you will be "blind" to whether the bottleneck is in these FE thread pools because the metrics are missing. 4. Prometheus Log Bloat Your Prometheus server logs will continue to fill up with

err="duplicate label set"

out-of-order sample

warnings. Depending on your logging configuration, this can consume unnecessary disk space on your monitoring stack and make it harder to find actual errors in the Prometheus logs. 5. Potential (though unlikely) Memory Creep The fact that metrics are being registered twice suggests a logic leak in the FE's internal

MetricRepo

. While the memory footprint of a few duplicate string objects in the metric registry is tiny, it indicates that a background service was re-initialized without properly cleaning up its previous state. Summary Is it critical? No. Your queries and loads will still run. Is it recommended to leave it? No. You are losing visibility into critical background processes. Since you have 3 FEs, restarting the Leader is the best path forward. It will trigger a leader election (taking ~5-10 seconds), clear the duplicate registry, and restore your monitoring visibility. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • faq/operation_maintenance_faq.md

Rumit Jain

07/23/2026, 9:40 AM

@Rocky is there any document which says this?

Copy code

Since you have 3 FEs, restarting the Leader is the best path forward. It will trigger a leader election (taking ~5-10 seconds), clear the duplicate registry, and restore your monitoring visibility.

Rocky

07/23/2026, 9:40 AM

There is no single "official" StarRocks document that uses that exact wording word-for-word, but the advice is a synthesis of several documented behaviors in the StarRocks architecture. The technical facts supporting that statement are found across these sections of the documentation: 1. Leader Election is Automatic The FAQ on Operation and Maintenance explicitly states:

How to manually switch FE leader?

Stop the current Leader, and a new leader will be elected automatically.

(Source) 2. Election Timing (~5–10 Seconds) While the documentation doesn't guarantee a "5–10 second" window in every scenario, it is the standard operational expectation for the underlying BDBJE (Berkeley DB Java Edition) protocol used by StarRocks. * The

bdbje_heartbeat_timeout_second

defaults to 30 seconds, but an active leader shutdown (restart) triggers an immediate resignation, typically allowing a new leader to be elected in under 10 seconds. * The system allows a metadata lag of up to 300 seconds (

meta_delay_toleration_second

) before followers stop serving, meaning a 5–10 second election window is well within the "safe" zone for cluster health. 3. Clearing the "Duplicate Registry" The "duplicate label set" warning in Prometheus occurs because the FE's internal MetricRepo (a Java object in memory) has registered the same metric name twice. * StarRocks Documentation on Metrics explains that metrics are collected and exported via the

/metrics

endpoint. * Because this registry is volatile (in-memory), restarting the FE process is the standard way to wipe the Java heap and re-initialize the metric registry from a clean state. There is no SQL command to "reset" the Prometheus exporter state without a process restart. 4. High Availability (3 FEs) The Deployment/HA documentation confirms that having 3 Followers is the requirement for a cluster to remain functional during a leader restart: * "In production, we recommend you deploy at least THREE Follower FE nodes... Each time the Leader FE node fails, StarRocks re-elects a new Leader FE node as long as more than half of the Follower FE nodes are alive." (Source) Summary: The advice you received is an expert recommendation based on how StarRocks handles high availability and in-memory state. You can verify the individual components in the FE Configuration and Operation FAQ pages. References • faq/operation_maintenance_faq.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/Meta_recovery.md

Rumit Jain

07/24/2026, 1:43 AM

@Rocky Will is disconnect all the existing connections to Starrocks for all users created in Starrocks?

Rocky

07/24/2026, 1:43 AM

Restarting the FE Leader will not disconnect every user in the entire cluster, but it will disconnect any user currently connected to that specific Leader node. Here is exactly what happens to connections during a Leader restart: 1. Connections to the Leader Node (Disconnected) Because you are stopping the FE process on the Leader, the TCP connections between the clients and that specific node will be severed. * Active queries: Any query currently running on the Leader will fail. * Idle sessions: Any open connection held in a pool (like HikariCP or a BI tool) that points directly to the Leader's IP will be closed. 2. Connections to Follower/Observer Nodes (Remain Active) If you have a 3-FE setup, users connected to the other two FEs (the Followers) will not be disconnected. They can continue to perform read operations. * Write operations: If a user on a Follower node tries to run a write command (like

INSERT

CREATE TABLE

) during the few seconds while the cluster is electing a new Leader, that specific command might time out or return an error until the new Leader is active. 3. How to avoid impact (Load Balancers) If your users connect through a Load Balancer (like F5, HAProxy, or a Kubernetes Service), the impact is minimized: _ The Load Balancer will detect the Leader is down and stop sending _new* connections to it. _ The _existing* connections to that Leader will still be dropped, but most modern applications/drivers (like the MySQL JDBC driver with

autoReconnect=true

) will automatically retry and be routed by the Load Balancer to a healthy Follower. 4. User Accounts (Persistent) The "users created in StarRocks" (your user accounts and permissions) are stored in the global metadata. Restarting the Leader does not delete or affect these accounts. Once the new Leader is elected or the node comes back up, everyone can log back in as usual. Summary * Cluster-wide disconnect? No. * Leader-specific disconnect? Yes. * Recommendation: If you have high-priority production traffic, perform the restart during a maintenance window or ensure your application has retry logic. References • faq/operation_maintenance_faq.md • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md References • administration/management/resource_management/Load_balance.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • quick_start/helm.md

Rumit Jain

07/24/2026, 6:16 AM

@Rocky Can I put some rate limiting as well on users like they cannot query more than X number of records in a single query etc. I am suing FE+CN architecure.

Rocky

07/24/2026, 6:16 AM

Yes, you can enforce limits on users to prevent them from running expensive queries that scan too many records or consume excessive resources. In StarRocks, this is handled through Resource Groups and User Properties. 1. Limit Records/Rows Scanned (Resource Groups) You can create a Resource Group that automatically terminates queries if they exceed a certain number of scanned rows, memory usage, or CPU time. These limits apply per BE node. * big_query_scan_rows_limit
: Limits the number of rows a query can scan. * big_query_mem_limit
: Limits the memory a query can consume. * big_query_cpu_second_limit
: Limits the total CPU time. Example: Limit a specific user to 100,000 rows First, create a resource group with the limits and a Classifier that binds the group to a specific user (e.g.,

'jack'

Copy code

sql
CREATE RESOURCE GROUP user_limit_group
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "20%",
    "big_query_scan_rows_limit" = "100000", -- Limit to 100k rows scanned per BE
    "big_query_mem_limit" = "1073741824"    -- Limit to 1GB memory per BE
)
FOR (user='jack');

If the user

jack

tries to run a query that scans more than 100,000 rows on any BE, StarRocks will kill the query and return:

ERROR 1064 (HY000): exceed big query scan_rows limit: current is ... but limit is 100000

2. Limit Connections and Session Variables You can also set limits directly on the user account for connection concurrency and query timeouts. * max_user_connections
: Limits how many simultaneous connections a user can open. * query_timeout
: Automatically kills queries that run longer than X seconds. Example: Set properties for a user

Copy code

sql
-- Limit user to 10 concurrent connections
ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "10");

-- Force a 60-second timeout for all of this user's queries (v3.3.3+)
ALTER USER 'jack' SET PROPERTIES ('session.query_timeout' = '60');

3. Query Queues (Concurrency Limiting) If you want to prevent users from overwhelming the system with too many simultaneous queries (rather than just killing them), you can use Query Queues. This puts new queries into a "waiting" state if the cluster is under heavy load.

Copy code

sql
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL query_queue_concurrency_limit = 100;

Summary of Recommendations • To limit rows/data volume: Use Resource Groups with

big_query_scan_rows_limit

. • To limit query duration: Use

ALTER USER ... SET PROPERTIES ('session.query_timeout' = 'X')

. • To limit "Rate" (Queries per second/Connections): Use

max_user_connections

and

concurrency_limit

in Resource Groups. References • administration/management/resource_management/resource_group.md • administration/management/monitor_manage_big_queries.md • administration/management/resource_management/Query_management.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md • administration/management/monitor_manage_big_queries.md

Rocky

07/24/2026, 6:17 AM

Yes, you can achieve this in an FE+CN (Compute Node) architecture. In StarRocks, CN nodes function as the compute layer (similar to stateless BEs), and they support the same resource management features. You can enforce these limits using Resource Groups and User Properties. 1. Limit Records Scanned per Query To prevent a user from scanning more than $X$ number of records, you should use a Resource Group with the

big_query_scan_rows_limit

parameter. In an FE+CN architecture, this limit applies to each CN node. If a query attempts to scan more than the limit on any single node, it will be terminated immediately. Example: Limit user
analyst
to 1,000,000 scanned rows

Copy code

sql
-- 1. Create a Resource Group with the row limit
CREATE RESOURCE GROUP limit_records_group
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "20%",
    "big_query_scan_rows_limit" = "1000000" -- Limit to 1M rows scanned per CN
)
FOR (user='analyst');

* Result: If the user runs

SELECT COUNT(*)

on a massive table, and the scan on a CN exceeds 1,000,000 rows, the query fails with:

ERROR 1064 (HY000): exceed big query scan_rows limit

. 2. Rate Limiting (Concurrency & Connections) StarRocks doesn't have a literal "Queries Per Second (QPS)" setting, but you can effectively rate-limit users by controlling their concurrency and connections. A. Concurrency Limiting (Resource Groups) You can limit how many queries a user can run simultaneously. If they exceed this, the queries will either fail or wait in a queue (if Query Queues are enabled).

Copy code

sql
ALTER RESOURCE GROUP limit_records_group WITH ("concurrency_limit" = "5");

B. Connection Limiting (User Properties) You can limit the number of active TCP connections a specific user can have with the FE.

Copy code

sql
ALTER USER 'analyst' SET PROPERTIES ("max_user_connections" = "10");

3. Global Rate Limiting (Query Queues) If your goal is to protect the CN nodes from being overwhelmed by too many users at once, you can enable Query Queues. This "cushions" the cluster by queuing queries instead of failing them when the CNs are at high CPU or memory utilization.

Copy code

sql
-- Enable the queue for SELECT queries
SET GLOBAL enable_query_queue_select = true;

-- Set global concurrency limit across the cluster
SET GLOBAL query_queue_concurrency_limit = 100;

Summary for FE+CN Architecture: * Rows Scanned: Use

big_query_scan_rows_limit

in a Resource Group. * Concurrent Queries: Use

concurrency_limit

in a Resource Group. * Total Connections: Use

max_user_connections

via

ALTER USER

. * Architecture Note: Even though the documentation often refers to "BE nodes," in your architecture, these limits apply directly to your CN nodes. References • administration/management/resource_management/resource_group.md • administration/management/resource_management/Query_management.md • administration/management/monitor_manage_big_queries.md • faq/resource_isolation_faq.md References • administration/management/resource_management/Query_management.md • faq/Others.md • administration/management/resource_management/query_queues.md • administration/management/monitor_manage_big_queries.md

Rumit Jain

07/24/2026, 8:00 AM

@Rocky Whta re the steps to onboard a user in starrocks. Wite from creations of group, giving access to specific database and setting passwords and number of connections

Rocky

07/24/2026, 8:00 AM

Onboarding a user in StarRocks involves setting up their resource limits, identity, and access permissions. Here is the step-by-step guide to onboarding a user from scratch. Step 1: Create a Resource Group (Optional but Recommended) A Resource Group allows you to limit the CPU, memory, and query complexity (like scanned rows) for the user.

Copy code

sql
-- Create a resource group that limits a user to scanning 1 million rows per node
CREATE RESOURCE GROUP analyst_group
WITH (
    "cpu_weight" = "10",
    "mem_limit" = "20%",
    "big_query_scan_rows_limit" = "1000000",
    "concurrency_limit" = "10"
);

Step 2: Create the User and Set Password When creating the user, you define their password and their identity (username and the host they can connect from). Use

'%'

to allow connections from any IP.

Copy code

sql
-- Create the user 'jack' with a password
CREATE USER 'jack'@'%' IDENTIFIED BY 'MySecurePassword123';

Step 3: Map the User to the Resource Group To ensure the user is governed by the resource group created in Step 1, add a Classifier. This tells StarRocks to apply the

analyst_group

limits whenever 'jack' runs a query.

Copy code

sql
ALTER RESOURCE GROUP analyst_group ADD (user='jack');

Step 4: Grant Access to a Specific Database By default, a new user has no permissions. You must grant them access to the specific database they need to work in.

Copy code

sql
-- Grant all privileges (Create, Drop, Alter, Select) on a specific database
GRANT ALL ON DATABASE my_business_db TO USER 'jack'@'%';

-- OR, Grant read-only access (Recommended for analysts)
GRANT SELECT ON ALL TABLES IN DATABASE my_business_db TO USER 'jack'@'%';

Step 5: Set Connection Limits To prevent a single user from exhausting all connection slots on the FE, set the

max_user_connections

property.

Copy code

sql
-- Limit 'jack' to 20 concurrent connections
-- Note: Specify only the username here, not the full 'user@host' identity
ALTER USER 'jack' SET PROPERTIES ("max_user_connections" = "20");

Summary of Commands Action | SQL Statement Limit Rows/CPU |

CREATE RESOURCE GROUP ... WITH ("big_query_scan_rows_limit" = "X")

Create Identity |

CREATE USER 'name'@'%' IDENTIFIED BY 'pass'

Set Concurrency |

ALTER USER 'name' SET PROPERTIES ("max_user_connections" = "X")

Give DB Access |

GRANT SELECT ON DATABASE db_name TO USER 'name'@'%'

Verification You can verify the user's setup with these commands:

Copy code

sql
SHOW USERS;                  -- Check if user exists
SHOW PROPERTY FOR 'jack';    -- Check max_user_connections
SHOW GRANTS FOR 'jack'@'%';  -- Check database permissions
SHOW RESOURCE GROUPS ALL;    -- Check if the user is correctly mapped to the group

References • sql-reference/sql-statements/account-management/CREATE_USER.md • administration/management/resource_management/resource_group.md • administration/user_privs/authorization/User_privilege.md • administration/user_privs/authentication/native_authentication.md References • administration/user_privs/authentication/native_authentication.md • sql-reference/sql-statements/account-management/ALTER_USER.md • sql-reference/sql-statements/account-management/CREATE_USER.md

Rumit Jain

07/28/2026, 6:13 AM

@Rocky What will happen if Starrocks Operator scaled down due to some issue in AWS EKS. Will FE+CN pods becomes unhealthy?

Rocky

07/28/2026, 6:13 AM

In an FE+CN architecture on AWS EKS, if the StarRocks Operator pod scales down (becomes unavailable) or crashes, your FE and CN pods will continue to run and remain healthy. The Operator acts as the "manager" (control plane), while the FE and CN pods are the "workers" (data plane). They are not tightly coupled for second-to-second operations. 1. Why FE and CN pods stay healthy When the StarRocks Operator creates a cluster, it generates standard Kubernetes objects like StatefulSets (for FE) and Deployments (for CN). * Kubernetes Governance: Once these objects are created, the standard Kubernetes controllers (not the StarRocks Operator) are responsible for ensuring the desired number of pods are running. * No Runtime Dependency: FE and CN pods do not "heartbeat" to the Operator to stay alive. They only need to communicate with each other. 2. What happens while the Operator is down? While the Operator is unavailable due to an EKS issue (e.g., node pressure or pod eviction), the following management functions will pause: * No Configuration Updates: If you change your

StarRocksCluster

YAML (e.g., changing memory limits), the changes won't take effect until the Operator pod is back online. * Automatic Scaling (CN): While the Operator creates the HPA (Horizontal Pod Autoscaler) for CN nodes, some advanced scaling logic or custom rebalancing handled by the Operator may stop. * Self-Healing of the Cluster Logic: If an FE node fails and requires specific metadata reconciliation that only the Operator performs (like leader election assistance in some edge cases), that logic will be delayed. However, for most common pod restarts, the StatefulSet will handle it automatically. 3. Impact of the "AWS EKS Issue" The impact depends on what the "issue" is: * Operator Pod Crash/Eviction: No impact on query performance or cluster health. * Node Failure (where the Operator lived): No impact on other nodes. * Total Cluster Issue (API Server down): The pods will still run and process queries, but you won't be able to run

kubectl

commands, and any automated scaling will freeze. Summary * Queries: Will continue to work. * Data Health: Remains intact. * FE/CN Pods: Remain

Running

and

Ready

. * Management: You cannot scale, upgrade, or change the cluster configuration until the Operator is restored. Recommendation: In production EKS environments, ensure the StarRocks Operator has appropriate Resource Requests/Limits and is protected by Pod Disruption Budgets (PDBs) to prevent it from being easily evicted during EKS maintenance. References • deployment/sr_operator.md • quick_start/helm.md • administration/management/resource_management/query_queues.md • administration/management/monitor_manage_big_queries.md • deployment/sr_operator.md • quick_start/helm.md • administration/management/resource_management/query_queues.md • administration/management/monitor_manage_big_queries.md References • quick_start/helm.md • deployment/sr_operator.md • deployment/deploy_manually.md

Rumit Jain

07/28/2026, 6:20 AM

@Rocky I am going to upgrade starrocks FE+CN version from 4.0.1 to 4.1.3. What are the recommended steps to perform the rollout. Also, will there be any downtime?

Rocky

07/28/2026, 6:20 AM

Upgrading from StarRocks 4.0.1 to 4.1.3 is a minor version upgrade. Because you are using the FE+CN architecture on Kubernetes, you can perform a rolling upgrade with no cluster-wide downtime. StarRocks is designed so that newer versions of CNs/BEs are backward compatible with older FEs, allowing the cluster to continue serving queries throughout the process. Pre-Upgrade Recommendations 1. Read Release Notes: Check the 4.1.3 release notes for any specific "Behavior Changes" or "Manual Migration" steps (e.g., specific SQL variables that must be set). 2. Create Metadata Image: On your current cluster, force a metadata checkpoint to ensure your backup is fresh.

Copy code

sql
    ALTER SYSTEM CREATE IMAGE;

Wait until the FE logs (

fe.log

) show that the image has been successfully synchronized to all Follower FEs. 3. Universal Compatibility Config: It is best practice to disable tablet balancing and cloning during the upgrade to prevent the background processes from trying to move data while nodes are restarting.

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0");
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0");
    ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
    ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "true");

────────── Rollout Steps (Using StarRocks Operator) In an FE+CN architecture, the Operator manages the rollout. You must update the CN nodes first, then the FE nodes. Step 1: Upgrade CN Nodes Patch the

StarRocksCluster

resource with the new 4.1.3 image. The Operator will perform a rolling restart of the CN pods one by one.

Copy code

bash
kubectl patch starrockscluster <your-cluster-name> --type='merge' -p '{"spec":{"starRocksCnSpec":{"image":"starrocks/cn-ubuntu:4.1.3"}}}'

Monitor progress:

kubectl get pods -w

. Ensure all CN pods return to

Running

and

Ready

status. Step 2: Upgrade FE Nodes Once all CNs are healthy, patch the FE spec. The Operator is intelligent enough to upgrade Follower FEs first and the Leader FE last to maintain metadata stability.

Copy code

bash
kubectl patch starrockscluster <your-cluster-name> --type='merge' -p '{"spec":{"starRocksFeSpec":{"image":"starrocks/fe-ubuntu:4.1.3"}}}'

Verification: After the FE rollout, log in to StarRocks and run

SHOW FRONTENDS;

and

SHOW COMPUTE NODES;

to verify the

Version

column shows

4.1.3

. Step 3: Post-Upgrade Tasks Re-enable balancing and scheduling once the cluster is stable.

Copy code

sql
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "10000");
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "500");
ADMIN SET FRONTEND CONFIG ("disable_balance" = "false");
ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "false");

────────── Will there be downtime? * Service Availability: No. Since this is a rolling upgrade, the FE Load Balancer (or Service) will route traffic to the healthy pods while others are restarting. * Query Impact: There is no "cluster downtime," but individual queries running on a specific CN or FE pod at the exact moment it restarts will fail. * Recommendation: Use a retry mechanism in your application. StarRocks v3.4+ has improved "Graceful Exit" logic, which allows CNs to finish existing queries before shutting down, significantly reducing the chance of failed queries. Important Warning (Downgrades) Note that once you upgrade to v4.1, you cannot downgrade back to any version below v4.0.6. Version 4.1 introduces internal data layout changes (tablet splitting/distribution) that are not backward compatible with early 4.0.x versions. References • deployment/upgrade.md • deployment/sr_operator.md

Rocky

07/28/2026, 6:20 AM

• administration/management/graceful_exit.md • release_notes/release-4.1.md References • administration/management/graceful_exit.md • deployment/sr_operator.md • deployment/upgrade.md

Rumit Jain

07/28/2026, 6:43 AM

@Rocky After scaling down CN pods form 2 to 1, I strted seeing these erros in fe pods: 2026-07-28 064211.707Z WARN (heartbeat-mgr-pool-7|381) [HeartbeatMgr$BackendHeartbeatHandler.call():354] backend heartbeat got exception, addr: kube-starrocks-cn-1.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local:9050 org.apache.thrift.TException: java.net.SocketTimeoutException: Connect timed out

Rocky

07/28/2026, 6:44 AM

The errors you are seeing occur because StarRocks FE still has the old CN pod registered in its metadata. When you scale down a CN (Compute Node) Deployment or StatefulSet in Kubernetes, the pod is removed from the cluster, but the StarRocks FE Leader still expects that node to be "Alive" and continues to send heartbeat signals to its last known address. Since the pod is gone, the connection times out, resulting in the

SocketTimeoutException

you see in the logs. How to Resolve This To stop the errors and clean up the cluster state, you must manually remove the decommissioned CN node from the StarRocks metadata. 1. Identify the inactive node Run the following SQL command in your StarRocks cluster:

Copy code

sql
SHOW PROC '/compute_nodes';

Look for the node where the

Alive

column is

false

. The

IP

Host

should match the one mentioned in your error log (e.g.,

kube-starrocks-cn-1...

). 2. Drop the Compute Node Use the

ALTER SYSTEM

command to remove the node from the cluster metadata. Replace the address with the one from the

Host

column in the previous step:

Copy code

sql
-- Example command
ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local:9050";

Why does this happen? In StarRocks, scaling in (reducing the number of nodes) is a two-step process: 1. Infrastructure Scale-in: The Kubernetes Operator or

kubectl

reduces the pod count. 2. System Metadata Cleanup: The FE must be told to "Drop" the node so it stops trying to schedule queries or send heartbeats to it. Currently, the StarRocks Operator does not always automatically execute the

DROP

command for CN nodes to prevent accidental data loss or metadata inconsistency. Manual intervention is the standard procedure to ensure the cluster state is intentional. Summary * Is it harmful? Not directly. The FE will simply stop trying to use that node for queries. However, it clutters your logs and makes the

Alive

status of your cluster look unhealthy. * Recommendation: Always run

ALTER SYSTEM DROP COMPUTE NODE

after scaling down your CN replicas in Kubernetes. References • administration/management/Scale_up_down.md • deployment/sr_operator.md • administration/management/BE_blacklist.md References • deployment/downgrade.md • deployment/upgrade.md • administration/stargo.md • deployment/sr_operator.md • administration/management/Scale_up_down.md

Rumit Jain

07/28/2026, 6:45 AM

@Rocky I am getting this:

Copy code

SHOW PROC '/compute_nodes'\G;
*************************** 1. row ***************************
        ComputeNodeId: 11019
                   IP: kube-starrocks-cn-0.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local
        HeartbeatPort: 9050
               BePort: 9060
             HttpPort: 8040
             BrpcPort: 8060
        LastStartTime: 2026-07-28 14:31:56
        LastHeartbeat: 2026-07-28 14:31:56
                Alive: true
 SystemDecommissioned: false
ClusterDecommissioned: false
               ErrMsg:
              Version: 4.1.1-14b7e3f
             CpuCores: 8
             MemLimit: 12.960GB
    NumRunningQueries: 0
           MemUsedPct: 0.00 %
           CpuUsedPct: 0.0 %
     DataCacheMetrics: N/A
       HasStoragePath: false
           StatusCode: OK
          StarletPort: 9070
             WorkerId: 1
        WarehouseName: default_warehouse
            TabletNum: 0
1 row in set (0.01 sec)

ERROR:
No query specified

Starrocks > ALTER SYSTEM DROP COMPUTE NODE "kube-starrocks-cn-1.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local:9050";
ERROR 1064 (HY000): compute node does not exists[kube-starrocks-cn-1.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local:9050]
Starrocks >

Rocky

07/28/2026, 6:49 AM

The error "compute node does not exists" combined with the fact that

SHOW PROC '/compute_nodes'

only shows

cn-0

suggests that the node
cn-1
has already been removed from the Compute Nodes list. However, the FE is still trying to heartbeat it. This typically happens for one of two reasons: 1. It is registered as a "Backend" (BE) instead of a "Compute Node" (CN) In StarRocks, CN and BE are managed in different internal lists. If

cn-1

was accidentally added as a Backend (which can happen if the

starRocksBeSpec

was used or if it was added manually as a backend),

DROP COMPUTE NODE

will fail because the system is looking in the wrong list. Action: Run this command to check if it's there:

Copy code

sql
SHOW BACKENDS;
-- or
SHOW PROC '/backends';

If you see

kube-starrocks-cn-1...

in this list, you must drop it using:

Copy code

sql
ALTER SYSTEM DROP BACKEND "kube-starrocks-cn-1.kube-starrocks-cn-search.devtestfkt04.svc.cluster.local:9050";

2. You are connected to a Follower FE with stale metadata If you are connected to a Follower FE, it might be showing you a different view of the metadata if it's out of sync, or the logs you are seeing are coming from a Follower that hasn't processed the "Drop" command yet. Action: 1. Check who the leader is:

SHOW FRONTENDS;

2. Connect directly to the Leader FE and run

SHOW PROC '/compute_nodes';

again. 3. If the node is truly gone from the Leader's metadata, the heartbeats in the log should eventually stop (they might persist for a few minutes as threads finish their cycles). 3. Check for specific warehouses (StarRocks 3.x/4.x) Since you are on v4.1.1, StarRocks supports Multi-Warehouse. While the Operator usually puts everything in

default_warehouse

, it's possible the node is registered in a different one. Action:

Copy code

sql
-- Check if there are other warehouses
SHOW WAREHOUSES;
-- Check all nodes across all warehouses
SHOW PROC '/compute_nodes';

(Note:
SHOW PROC
usually shows all, but verify if there are any other warehouses listed.) Summary Checklist 1. Try

SHOW BACKENDS;

. If it's there,

DROP BACKEND

. 2. Confirm you are on the Leader FE. 3. If

SHOW PROC '/compute_nodes'

shows only one node and it's the

cn-0

, and the log continues for more than 5 minutes on the Leader FE, try restarting the Leader FE to clear any orphaned heartbeat threads. References • administration/management/Scale_up_down.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/resource_management/Multi_warehouse.md • administration/management/Scale_up_down.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/resource_management/Multi_warehouse.md References • administration/management/Scale_up_down.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/resource_management/Multi_warehouse.md • administration/management/FE_configuration.md

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rocky

07/28/2026, 6:49 AM

Rumit Jain

07/28/2026, 7:01 AM

@Rocky If version upgrade is no downtime, then why existing connections to the fe cluster drops to 0.

Rocky

07/28/2026, 7:01 AM

When we say a StarRocks upgrade has "no downtime," it means the cluster remains available to process new requests. It does not mean that existing TCP connections are kept alive. Here is why your connections drop to 0 during a rolling upgrade: 1. TCP Connections are Tied to Processes StarRocks FEs are Java processes. To upgrade the software, the existing process must be stopped and replaced by a new one running the 4.1.3 image. * When a pod restarts, the operating system closes all active TCP/IP sockets associated with that process. * Since your SQL client or application is connected directly to a specific FE pod (either via a Load Balancer or a Kubernetes Service), that specific connection is physically severed. 2. The "Rolling" Nature of the Upgrade In a rolling upgrade, Kubernetes restarts the FE pods one by one: 1. FE-Follower-1 restarts: All connections to this pod drop. 2. FE-Follower-2 restarts: All connections to this pod drop. 3. FE-Leader restarts: All remaining connections drop. By the time the upgrade finishes, every single pod has been restarted, meaning 100% of the original connections will have been disconnected at some point during the process. 3. Leader Handover Gap When the Leader FE restarts, the cluster must elect a new Leader. * Even if you have other FEs running, there is a brief window (usually a few seconds) where no FE is the "Leader." * During this window, the cluster cannot accept new "write" metadata (like DDLs), and many drivers/proxies will see the cluster as "temporarily unavailable" until the new Leader is established. 4. Kubernetes Service/Load Balancer Behavior If you are using a Kubernetes

Service

(the default for the StarRocks Operator): * The Service does not "migrate" a TCP session from an old pod to a new pod. It only manages the routing of new connection requests. * When a pod is marked as "Not Ready" (terminating), the Service removes it from its endpoint list. Existing connections to that pod are simply killed. How to handle this in production To achieve true "seamlessness" from the application's perspective, you should implement the following: 1. Connection Pooling with Validation: Use a pool like HikariCP or Druid. Configure it to "test" connections (e.g.,

SELECT 1

) when they are borrowed from the pool. If a connection was dropped by an FE restart, the pool will silently discard it and create a new one to a healthy FE. 2. Application Retries: Ensure your application logic has a retry mechanism. If a query fails with a "Socket closed" or "Communication link failure" during the upgrade, the app should simply try again. 3. Graceful Shutdown: StarRocks (v3.4+) supports "Graceful Exit." This allows the node to stop accepting new queries and wait for existing queries to finish before the process actually terminates, which minimizes (but does not eliminate) connection drops. Summary: "No downtime" means the cluster's service capacity is never 0, but your individual connections are 100% guaranteed to be reset because the underlying servers are being replaced.

Rumit Jain

07/28/2026, 4:25 PM

@Rocky I have a task where I need to run one query on one table daily and fetch the records. Is their something like View in starrocks which I can use rather than calling the main table. or what is the recommended way for such use case?

Rocky

07/28/2026, 4:25 PM

For a daily query where you need to fetch specific records from a main table, the recommended approach is to use an Asynchronous Materialized View (Async MV). While StarRocks supports standard (logical) Views, they do not store data physically and will re-run the full query against your main table every time you call them. An Async MV, however, pre-computes the results and stores them as a physical table, which makes fetching records significantly faster and less resource-intensive. Why use an Async MV? * Performance: It stores the "snapshot" of your query results. When you fetch the records daily, you are reading from a small, optimized table rather than scanning the large main table. * Automation: You can set a refresh schedule (e.g., once a day) so the data is always ready when you need to fetch it. * Cost-Effective: It prevents repetitive, expensive calculations on your main cluster by doing the work once per day. Example: Daily Refreshing MV If you want to filter records from a table named

user_activity

for the previous day, you can create the MV like this:

Copy code

sql
CREATE MATERIALIZED VIEW daily_fetch_mv
-- Refresh this MV once every day
REFRESH ASYNC EVERY (INTERVAL 1 DAY)
DISTRIBUTED BY HASH(user_id)
AS
SELECT 
    user_id, 
    action, 
    request_time 
FROM user_activity
WHERE request_time >= date_add(current_date(), INTERVAL -1 DAY);

When to use a standard View instead? Use a Logical View (

CREATE VIEW ...

) only if: 1. The query on the main table is very fast and simple. 2. You need real-time data (Async MVs have a delay until the next refresh). 3. You want to avoid the extra storage cost of keeping a second copy of the data. Summary of Recommendations * For heavy/complex daily reports: Use Asynchronous Materialized View. * For simple shortcuts/alias: Use Logical View. * For moving data out of StarRocks: If your task involves moving this data to S3 or an external system daily, consider using the EXPORT command. References • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md • using_starrocks/async_mv/Materialized_view.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md ) command. /sql-reference/sql-statements/data-manipulation/EXPORT) command. References • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • using_starrocks/async_mv/Materialized_view.md • using_starrocks/async_mv/use_cases/data_modeling_with_materialized_views.md

Rumit Jain

07/28/2026, 4:31 PM

@Rocky How to export a table from starrocks FE+CN in EKS to S3?

Rocky

07/28/2026, 4:31 PM

To export data from StarRocks in EKS to S3, you have two primary methods: the classic EXPORT
command or the modern INSERT INTO FILES
(available from v3.1+). Since you are running on EKS, it is highly recommended to use IAM Roles for Service Accounts (IRSA) for authentication instead of hardcoding Access Keys. Method 1: INSERT INTO FILES (Recommended) This method is the most flexible because it allows you to use a standard

SELECT

query to filter or transform data before exporting.

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://your-bucket-name/export/path/file_prefix_>",
    "format" = "parquet", -- or "csv"
    "compression" = "snappy",
    "aws.s3.use_instance_profile" = "true", -- Required for IRSA/EKS IAM roles
    "aws.s3.region" = "us-east-1"
)
SELECT * FROM your_table_name;

* Pros: Supports complex queries, multiple file formats (Parquet/CSV), and single/multiple file output. * Authentication: Set

"aws.s3.use_instance_profile" = "true"

to let the CN pods use the IAM role attached to their Kubernetes Service Account. ────────── Method 2: EXPORT TABLE This is an asynchronous background task. It is better for very large tables where you want to fire-and-forget.

Copy code

sql
EXPORT TABLE your_table_name 
TO "<s3a://your-bucket-name/export/path/>"
PROPERTIES (
    "column_separator" = ",",
    "timeout" = "3600"
)
WITH BROKER (
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-east-1"
);

* Status Check: You must use

SHOW EXPORT;

to track the progress of this job. * Path: Note the use of

s3a://

prefix for the classic

EXPORT

command. ────────── Key Considerations for EKS 1. IAM Permissions: The IAM role assigned to your CN pods must have

s3:PutObject

and

s3:AbortMultipartUpload

permissions on the target bucket. 2. CN Role: In a StarRocks cluster, the CN (Compute Nodes) are the ones that actually write the data to S3. Ensure your EKS nodes (or the specific pods via IRSA) have the correct permissions. 3. Brokerless: From v2.5 onwards, you do not need to deploy a separate "Broker" process. The

WITH BROKER

keyword is still required for syntax, but the CN handles the logic internally. 4. Network: Ensure your EKS cluster has network access to the S3 endpoint (either via an Internet Gateway or an S3 VPC Endpoint). How to check if it's working If using

EXPORT

, run:

Copy code

sql
SHOW EXPORT\G;

Look for

State: FINISHED

. If it shows

CANCELLED

, the

ErrorMsg

field will usually tell you if it was a "Permission Denied" (IAM issue) or a "Connection Timeout" (Network issue). References • unloading/unload_using_insert_into_files.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md • unloading/Export.md • loading/s3.md • deployment/shared_data/s3.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md References • unloading/Export.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md

Rumit Jain

07/28/2026, 4:43 PM

@Rocky Why this export failed?

Copy code

EXPORT TABLE TABLENAME
    -> TO "<s3a://bucket-name/export_data/>"
    -> PROPERTIES (
    ->     "column_separator" = ",",
    ->     "timeout" = "3600"
    -> )
    -> WITH BROKER (
    ->     "aws.s3.use_instance_profile" = "true",
    ->     "aws.s3.region" = "us-east-1"
    -> );

show export\G;
*************************** 1. row ***************************
     JobId: 592819
   QueryId: 019fa999-9e57-73ec-a377-506f788a7d1d
     State: CANCELLED
  Progress: 100%
  TaskInfo: {"partitions":["*"],"column separator":",","columns":["*"],"tablet num":1,"broker":"","coord num":1,"db":"DBNAME","tbl":"TABLENAME","row delimiter":"\n","mem limit":2147483648}
      Path: <s3a://bucket-name/export_data/>
CreateTime: 2026-07-28 16:40:44
 StartTime: 2026-07-28 16:40:46
FinishTime: 2026-07-28 16:40:52
   Timeout: 3600
  ErrorMsg: type:RUN_FAIL; msg:move tmp file to final destination fail, errors while check if path <s3a://bucket-name/export_data/data_019fa999-9e57-73ec-a377-506f788a7d1d_0_0.csv> exist
1 row in set (0.01 sec)

ERROR:
No query specified

Rocky

07/28/2026, 4:43 PM

The error

move tmp file to final destination fail

with the specific message

errors while check if path ... exist

indicates that StarRocks successfully wrote the data to a temporary S3 location but failed during the final "Rename/Move" operation. In S3, a "Move" is not a native operation; StarRocks must copy the file to the final destination and then delete the temporary file. This process requires a specific set of IAM permissions that go beyond just

PutObject

. 1. Missing IAM Permissions For an

EXPORT

to succeed using an Instance Profile (IRSA) on EKS, your IAM role must have all four of these permissions. The error you see (failure to "check if path exist") is almost always caused by a missing

s3:ListBucket

s3:DeleteObject

permission. Ensure your IAM policy looks exactly like this:

Copy code

json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3ObjectActions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": "arn:aws:s3:::bucket-name/*"
        },
        {
            "Sid": "S3BucketActions",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::bucket-name"
        }
    ]
}

* s3:ListBucket
: Required to check if the destination path exists before moving. * s3:DeleteObject
: Required to clean up the temporary folder (

__starrocks_export_tmp_...

) after the move. * s3:GetBucketLocation
: Often required by the S3 SDK to verify the region. 2. S3 Path Scheme While

s3a://

is supported for backward compatibility, modern StarRocks (v2.5+) running on EKS with IRSA works best with the native s3://
prefix. Try changing your path: * Change:

TO "<s3a://bucket-name/export_data/>"

* To:

TO "<s3://bucket-name/export_data/>"

3. Check for Existing Files The error can also trigger if a file with the exact same name already exists in the destination and the IAM role does not have permission to overwrite/delete it. * Try exporting to a new, empty sub-folder to see if it resolves the conflict. 4. Kubernetes Service Account Since you are on EKS, verify that the CN pods (Compute Nodes) are actually using the IAM role. 1. Check that the CN Pod has the environment variables

AWS_ROLE_ARN

and

AWS_WEB_IDENTITY_TOKEN_FILE

. 2. If these are missing, your IRSA (IAM Roles for Service Accounts) is not correctly associated with the ServiceAccount used by the StarRocks operator. Summary Checklist 1. Does the IAM role have

s3:ListBucket

on the Bucket (not the prefix)? 2. Does the IAM role have

s3:DeleteObject

on the Objects? 3. Have you tried using the

s3://

prefix instead of

s3a://

? References • sql-reference/aws_iam_policies.md • unloading/Export.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md • integrations/authenticate_to_aws_resources.md References • sql-reference/aws_iam_policies.md • unloading/Export.md • data_source/catalog/iceberg/iceberg_rest_s3.md • administration/management/FE_configuration.md • integrations/authenticate_to_aws_resources.md

Rumit Jain

07/28/2026, 4:50 PM

@Rocky Permission is correct,

Copy code

{
  "Action": [
    "s3:PutObject",
    "s3:GetObject",
    "s3:DeleteObject",
    "s3:ListBucket",
    "s3:ListBucketMultipartUploads",
    "s3:AbortMultipartUpload",
    "s3:PutObjectAcl",
    "s3:GetBucketLocation"
  ],
  "Effect": "Allow",
  "Resource": [
    "arn:aws:s3:::BUCKET-NAME",
    "arn:aws:s3:::BUCKET-NAME/*"
  ]
}

CHANGED s3a to s3 also, but still same error:

Copy code

ErrorMsg: type:RUN_FAIL; msg:move tmp file to final destination fail, errors while check if path <s3://BUCKET-NAME/export_data/data_019fa99f-6647-760f-91b9-e2a9613821a6_0_0.csv> exist

Rocky

07/28/2026, 4:50 PM

Based on the error message and your provided SQL, there are two likely causes for this failure. 1. Remove the brackets
< >
from the path (Most Likely) In your SQL command, you wrapped the S3 URI with

and

. These are placeholder symbols used in documentation to show where your values go—they should not be included in the actual SQL string. If you include them inside the quotes, StarRocks treats them as literal characters in the bucket name or path, which S3 will reject or fail to locate during the "move" (rename) phase. Change this:

TO "<s3://bucket-name/export_data/>"

To this:

TO "<s3://bucket-name/export_data/>"

────────── 2. The FE (Front-End) needs IAM permissions In a StarRocks

EXPORT

job, the workflow is: 1. CNs (Compute Nodes) scan the data and write it to a temporary folder on S3. 2. The Leader FE then performs a "Rename" (which is a

Copy

Delete

operation in S3) to move those files to your final destination. In EKS, if you are using IRSA (IAM Roles for Service Accounts), you must ensure that the IAM role is attached to both the CN and the FE pods. If only the CNs have the role, the data will be written successfully (Step 1), but the FE will fail when it tries to check the path or move the files (Step 2), resulting in the

move tmp file to final destination fail

error. Verify: * Check your FE pod description:

kubectl describe pod <fe-pod-name>

. * Ensure it has the same

AWS_ROLE_ARN

and

AWS_WEB_IDENTITY_TOKEN_FILE

environment variables as your CN pods. ────────── Recommended Alternative:
INSERT INTO FILES
If you continue to have trouble with the

EXPORT

command's multi-step "Rename" logic, use

INSERT INTO FILES

. This is the more modern way to export data; it is performed entirely by the CN nodes and does not require the FE to coordinate a file rename on S3.

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://bucket-name/export_data/data_>",
    "format" = "csv",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-east-1"
)
SELECT * FROM TABLENAME;

Why this is better for your case: * Direct Write: No "temporary folder" move/rename step. * Simplified Permissions: If the CN can write to S3, the job succeeds. * Synchronous: You get the result immediately instead of having to run

SHOW EXPORT

. References • unloading/Export.md • unloading/unload_using_insert_into_files.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md • integrations/authenticate_to_aws_resources.md References • administration/management/FE_configuration.md • unloading/Export.md

Rumit Jain

07/28/2026, 4:54 PM

@Rocky I am using pod IAM role, do i need to make instacne profile as false and sdk as true?

Rocky

07/28/2026, 4:54 PM

Yes, for EKS with IRSA (Pod-level IAM Roles), you should switch your authentication properties. While

use_instance_profile

works for EC2 nodes, EKS Pod Identity (IRSA) relies on a Web Identity Token. The most reliable way for StarRocks to pick this up is to tell it to use the AWS SDK's default behavior. The Correct Configuration for EKS IRSA Modify your

EXPORT

command properties like this:

Copy code

sql
EXPORT TABLE TABLENAME
TO "<s3://BUCKET-NAME/export_data/>"
PROPERTIES (
    "column_separator" = ",",
    "timeout" = "3600"
)
WITH BROKER (
    "aws.s3.use_aws_sdk_default_behavior" = "true", -- Enables IRSA token detection
    "aws.s3.use_instance_profile" = "false",         -- Disables EC2-level check
    "aws.s3.region" = "us-east-1"
);

Why this matters 1. IRSA Detection: When

aws.s3.use_aws_sdk_default_behavior

true

, the internal AWS SDK looks for the environment variables

AWS_ROLE_ARN

and

AWS_WEB_IDENTITY_TOKEN_FILE

. These are the variables injected into your Pods by the EKS OIDC provider. 2. Instance Profile vs. Pod Role:

use_instance_profile

is designed for the EC2 metadata service. In some EKS environments, Pods are blocked from accessing the EC2 metadata service for security, causing that check to fail or time out. Final Troubleshooting Check If it still fails with the same

move tmp file

error after this change: * FE Pod Identity: Ensure your FE pods have the IAM role. In the

EXPORT

workflow, the FE is the one that executes the "move/rename" at the end. If the FE pod doesn't have the ServiceAccount with the IAM role annotation, it cannot finalize the job. * Trust Relationship: Verify the IAM Role's "Trust Relationship" allows the OIDC provider of your EKS cluster to assume the role for the specific Namespace and ServiceAccount name used by your FE and CN pods. Recommendation: If you continue to face "move" errors, try

INSERT INTO FILES

. It bypasses the FE's "move" step entirely, writing directly from the CN nodes to the final destination, which is much simpler to debug in EKS. References • deployment/shared_data/s3.md • unloading/Export.md • administration/management/FE_configuration.md • sql-reference/aws_iam_policies.md • integrations/authenticate_to_aws_resources.md References • administration/management/FE_configuration.md • deployment/shared_data/s3.md

Rumit Jain

07/28/2026, 4:55 PM

@Rocky how to check status of INSERT INTO?

Rocky

07/28/2026, 4:55 PM

INSERT INTO FILES

is a synchronous command. This means your SQL client (like MySQL CLI or DBeaver) will wait and stay "busy" until the export is finished. Here is how you can check the status depending on whether it is currently running or already finished: 1. If it is currently running (from another session) Since the command blocks your current terminal, you must open a new connection to check its progress: * View active queries:

Copy code

sql
    SHOW PROCESSLIST;

Look for the query that starts with

INSERT INTO FILES

. The

Time

column shows how long it has been running. * View detailed progress (v3.1 and later): StarRocks tracks

INSERT

jobs in the

information_schema

. You can see the rows processed in real-time:

Copy code

sql
    SELECT * FROM information_schema.loads
    WHERE TYPE = 'INSERT'
    ORDER BY CREATE_TIME DESC LIMIT 1\G

* PROGRESS: Shows ETL and Load progress percentages. * SCAN_ROWS: Shows how many rows have been read from the source table so far. 2. If the command has finished * Success: The terminal will return a summary once complete:

Copy code

text
    Query OK, 1000000 rows affected (15.23 sec)

* Failure: It will return an error message immediately in the terminal (e.g.,

Access Denied

Timeout

). 3. Using a Label for easier tracking To make it easier to find your specific job in

information_schema.loads

, you can assign a label to the

INSERT

statement:

Copy code

sql
-- Set a label for the session
SET insert_label = 'export_users_batch_1';

INSERT INTO FILES (
    "path" = "<s3://bucket/path/>",
    "format" = "csv",
    ...
)
SELECT * FROM my_table;

Then you can find it specifically with:

Copy code

sql
SELECT * FROM information_schema.loads WHERE LABEL = 'export_users_batch_1';

SHOW EXPORT

SHOW PROCESSLIST

information_schema.loads

Failure Handling | Fails at the "move" step often | Fails immediately while writing Troubleshooting Timeouts If the job takes a long time and your connection drops, the export might be cancelled. You can increase the session timeout before running the command:

Copy code

sql
SET query_timeout = 7200; -- Set to 2 hours

References • loading/minio.md • loading/s3.md • sql-reference/information_schema/loads.md • loading/loading_introduction/feature-support-loading-and-unloading.md • unloading/unload_using_insert_into_files.md References • loading/minio.md • loading/s3.md • sql-reference/information_schema/loads.md • loading/loading_introduction/feature-support-loading-and-unloading.md • unloading/unload_using_insert_into_files.md • sql-reference/sql-statements/loading_unloading/INSERT.md loading/INSERT) /INSERT) • sql-reference/sql-functions/table-functions/files.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md

Rocky

07/28/2026, 4:55 PM

• administration/management/FE_configuration.md • unloading/Export.md • deployment/shared_data/s3.md • sql-reference/aws_iam_policies.md • integrations/authenticate_to_aws_resources.md References • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/information_schema/loads.md • unloading/unload_using_insert_into_files.md • loading/minio.md

Rumit Jain

07/28/2026, 4:58 PM

@Rocky Insert query wworks but in place of column name, record is coming

Rocky

07/28/2026, 4:58 PM

By default,

INSERT INTO FILES

and the

EXPORT

command do not include a header row (column names) in the output CSV files. This is why you are seeing data records in the first row where you expect column names. To include column names as the first row of your CSV output, you must use the following property (note that this requires StarRocks v3.5.13+, v4.0.6+, or v4.1.0+): For
INSERT INTO FILES
Add

"csv.include_header" = "true"

to your properties list:

Copy code

sql
INSERT INTO FILES (
    "path" = "<s3://bucket-name/export_data/data_>",
    "format" = "csv",
    "csv.include_header" = "true",  -- Add this line
    "aws.s3.use_aws_sdk_default_behavior" = "true",
    "aws.s3.region" = "us-east-1"
)
SELECT * FROM TABLENAME;

For the
EXPORT
command If you are using the standard

EXPORT

command instead, the property name is slightly different:

Copy code

sql
EXPORT TABLE TABLENAME
TO "<s3://bucket-name/export_data/>"
PROPERTIES (
    "with_header" = "true"  -- Add this line
)
WITH BROKER (
    "aws.s3.use_aws_sdk_default_behavior" = "true",
    "aws.s3.region" = "us-east-1"
);

Important Considerations: 1. Version Requirement: If your StarRocks version is older than the ones listed above, these properties will not be recognized, and the export will continue to exclude headers. 2. Viewing the file: If you are opening the CSV in a tool like Excel or DBeaver and it is missing a header, those tools often "guess" that the first row of data is the header row, which can be confusing. 3. Loading back: If you later try to load this file back into StarRocks, remember to set

"csv.skip_header" = "1"

so that StarRocks doesn't try to import your column names as a row of data. References • sql-reference/sql-functions/table-functions/files.md • unloading/unload_using_insert_into_files.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md References • sql-reference/sql-functions/table-functions/files.md • faq/loading/Stream_load_faq.md • loading/Etl_in_loading.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md

Rumit Jain

07/28/2026, 5:03 PM

@Rocky What is the difference between export and insert into files and which is beter and recommended

Rocky

07/28/2026, 5:04 PM

Both

INSERT INTO FILES

and

EXPORT

achieve the same goal—moving data from StarRocks to external storage—but they differ significantly in how they operate and what features they support. Key Differences Feature | INSERT INTO FILES
| EXPORT
Execution | Synchronous (Foreground). Your client waits for completion. | Asynchronous (Background). Job runs after the command returns. Status Check |

SHOW PROCESSLIST

information_schema.loads

. |

SHOW EXPORT

. Ease of Use | High. Uses standard SQL

INSERT

syntax. | Medium. Requires a specific

EXPORT

syntax. Partitioning | Supported. Can use

partition_by

to create sub-folders. | Not Supported. Writes all files to one flat path. Single File | Supported via

"single" = "true"

. | Supported via

"single" = "true"

(v3.1+). File Formats | Parquet, CSV, ORC (v3.3+). | Parquet, CSV, ORC. Timeout Handling | Subject to

query_timeout

session variable. | Subject to

timeout

property in the command. ────────── Which one is better? Use
INSERT INTO FILES
(Recommended for most cases) This is the modern, preferred way to unload data in StarRocks (v3.2+). * Best for: Small to medium datasets, or when you need a specific folder structure (partitioning). * Why: It is more flexible and integrates perfectly with the AWS SDK behavior you are already using. It also bypasses the "move" step that often causes permission issues in

EXPORT

. * Pro Tip: If you are worried about the connection dropping for a long export, just increase your timeout:

SET query_timeout = 7200;

. Use
EXPORT
* Best for: Very large datasets (billions of rows) where the export might take hours. * Why: Since it runs in the background, you don't need to keep your SQL terminal open. You can start the job and walk away, then check

SHOW EXPORT

later to see if it finished. Recommendation For your EKS IRSA setup, I recommend INSERT INTO FILES
. It is generally more "cloud-native," easier to debug because errors appear immediately in your terminal, and it handles the IAM role permissions more reliably than the multi-step

EXPORT

process. References • unloading/unload_using_insert_into_files.md • loading/loading_introduction/feature-support-loading-and-unloading.md • sql-reference/sql-statements/loading_unloading/unloading/EXPORT.md • sql-reference/sql-functions/table-functions/files.md References • unloading/unload_using_insert_into_files.md • sql-reference/sql-functions/table-functions/files.md • loading/loading_introduction/feature-support-loading-and-unloading.md

Rumit Jain

07/29/2026, 2:03 AM

@Rocky Is it mandatory to have operator up and running before starting the version upgrade of FE+CN?

Rocky

07/29/2026, 2:03 AM

If your StarRocks cluster is managed by the StarRocks Operator (i.e., you deployed it using a

StarRocksCluster

resource), then yes, the Operator must be up and running to perform the upgrade. Here is why and how the process works: 1. The Operator is the Orchestrator When you upgrade an Operator-managed cluster, you do not manually restart pods or replace binaries. Instead, you update the

image

tag in your

StarRocksCluster

YAML (or via

kubectl patch

). * The Operator detects this change in the desired state. * The Operator then executes the rolling upgrade logic: it ensures nodes are taken down gracefully, upgraded in the correct order (CN/BE first, then FE), and that the cluster remains healthy throughout the process. * If the Operator is down: You can change the YAML, but nothing will happen to your cluster because there is no "brain" to execute the update. 2. Recommended Upgrade Sequence If you are also planning to upgrade the Operator itself, you should follow this order: 1. Upgrade the Operator first: Apply the new Operator manifests or update the Helm chart. This replaces the Operator pod. 2. Verify Operator health: Ensure the new Operator pod is

Running

and healthy. 3. Upgrade the StarRocks Cluster: Patch the

StarRocksCluster

CRD with the new StarRocks version images. The newly upgraded Operator will then handle the FE and CN rolling updates. 3. When is it NOT mandatory? It is only "not mandatory" if you are performing a manual upgrade of a cluster that was never managed by the Operator (e.g., you deployed using raw StatefulSets or binary files). In that case, you are the orchestrator, and you must manually stop/start nodes in the correct order: * Order: Upgrade all CNs/BEs first, then Follower FEs, and finally the Leader FE. Summary: For a Kubernetes-native deployment using the StarRocks Operator, the Operator is the component that actually performs the upgrade. Ensure it is running and healthy before you trigger the version change. References • deployment/sr_operator.md • deployment/upgrade.md References • deployment/sr_operator.md • deployment/upgrade.md • deployment/deploy_manually.md • faq/operation_maintenance_faq.md References • deployment/upgrade.md • deployment/sr_operator.md • deployment/deploy_manually.md • faq/operation_maintenance_faq.md

Rumit Jain

07/29/2026, 6:56 AM

@Rocky Is their any prometheus metrics for Fe-Proxy health check?

Rocky

07/29/2026, 6:56 AM

The FE-Proxy (specifically

starrocks-fe-proxy

) is an optional component in the StarRocks Kubernetes Operator environment, based on Nginx. Its primary purpose is to act as a load balancer/proxy for multiple FE nodes, specifically to handle Stream Load requests correctly across multiple FEs. Because it is an Nginx-based component, its metrics and health checks follow Nginx standards rather than internal StarRocks logic. 1. Health Check Endpoint The

fe-proxy

typically exposes a health check endpoint at: * Path:

/health

* Behavior: It is configured to return a

200 OK

status to indicate the proxy service is up. * Usage: This is used by Kubernetes for Liveness and Readiness probes to ensure the proxy pod is healthy. 2. Prometheus Metrics The StarRocks Operator does not currently include a built-in, dedicated StarRocks-logic exporter for the

fe-proxy

. However, you can monitor it using standard Nginx Prometheus patterns: * Standard Nginx Metrics: If you need detailed metrics (like request counts, connection rates, or error rates from the proxy level), you would typically need to enable the

ngx_http_stub_status_module

in the Nginx config and use a sidecar like the

nginx-prometheus-exporter

. * ServiceMonitor: While the StarRocks Helm chart supports

ServiceMonitor

for FE, BE, and CN nodes to scrape their

/metrics

endpoints, the

fe-proxy

does not have a default

/metrics

endpoint unless you customize the deployment to include an exporter. Summary of Health Monitoring Component | Health Check Endpoint | Prometheus Metrics Endpoint FE Node |

http://<fe_ip>:8030/api/health

http://<fe_ip>:8030/metrics

BE Node |

http://<be_ip>:8040/api/health

http://<be_ip>:8040/metrics

FE-Proxy |

http://<proxy_ip>:<port>/health

| Not provided by default (requires Nginx exporter) Recommendation: If you want to verify if the FE-Proxy is working, the best way is to check the Kubernetes pod status or curl the

/health

endpoint. For deep traffic analysis at the proxy level, I recommend integrating the Nginx Prometheus Exporter as a sidecar in your

starrocksFeProxySpec

. References • sql-reference/sql-functions/table-functions/files.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • deployment/sr_operator.md References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • deployment/sr_operator.md • sql-reference/sql-functions/table-functions/files.md References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md

Rumit Jain

08/03/2026, 11:46 AM

@Rocky I want to use plugin auditLoader. But I dont see data populating in the loader tbale. This is my plugins.conf file:

Copy code

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   <http://www.apache.org/licenses/LICENSE-2.0>
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

### plugin configuration

# The max size of a batch, default is 50MB
max_batch_size=52428800

# The max interval of batch loaded, default is 60 seconds
max_batch_interval_sec=60

# the max stmt length to be loaded in audit table, default is 1048576
max_stmt_length=1048576

# StarRocks FE host for loading the audit, default is 127.0.0.1:8030
# this should be the host port for stream load
frontend_host_port=127.0.0.1:8030

# If the response time of a query exceed this threshold, it will be recored in audit table as slow_query
qe_slow_log_ms=5000

# the capacity of audit queue, default is 1000
max_queue_size=1000

# Database of the audit table
database=starrocks_audit

# Audit table name, to save the audit data
table=audit_log

# StarRocks user. This user must have import permissions for the audit table
user=auditloader

# StarRocks user's password
password=Audit@123

# StarRocks password encryption key
secret_key=

# Filter conditions when importing audit information
filter=

# Timeout for uninstalling the plugin, default is 5000 ms
uninstall_timeout=5000

and this is my fe.conf:

Copy code

run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = starrocks-data
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.us-east-1.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=false
    aws_s3_use_aws_sdk_default_behavior=true
    enable_trace_historical_node = true
    audit_log_modules = slow_query, query, connection
    audit_log_json_format = true
    qe_slow_log_ms = 1000
    aws_s3_enable_partitioned_prefix=true
    aws_s3_num_partitioned_prefix=64
    automated_cluster_snapshot_interval_seconds=3600
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx11g -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    mysql_service_nio_enabled = true
    sys_log_level = INFO
    min_graceful_exit_time_second = 25
    label_keep_max_num=50000

Rocky

08/03/2026, 11:46 AM

Based on your configuration, the most likely reason your

auditLoader

table is empty is a format mismatch between your FE logs and the plugin's parser. 1. Disable JSON Audit Logs (Most Likely Cause) In your

fe.conf

, you have set:

audit_log_json_format = true

The

AuditLoader

plugin is designed to parse the default pipe-separated/text format of the

fe.audit.log

. When you enable JSON format, the structure of the log file changes completely, and the plugin's parser cannot find the fields it expects. Action: Change this to

false

in your

fe.conf

and restart the FE:

Copy code

properties
audit_log_json_format = false

2. Check Plugin Status Ensure the plugin is actually active in the cluster. Run:

Copy code

sql
SHOW PLUGINS;

The status should be

INSTALLED

. If it is not listed, you need to run the

INSTALL PLUGIN

command. 3. Verify Table and Privileges The

AuditLoader

does not create the table automatically. You must manually create the database and table as specified in the documentation. * Database:

starrocks_audit

* Table:

audit_log

* User: The user

auditloader

must have

INSERT

and

LOAD

privileges on that table. Check if partitions exist (if you used dynamic partitioning):

Copy code

sql
SHOW PARTITIONS FROM starrocks_audit.audit_log;

If no partitions exist, the Stream Load performed by the plugin will fail. 4. Search for Errors in
fe.log
Since the

AuditLoader

runs as a background thread within the FE, it prints its error messages to the standard

fe.log

. Look for the "audit" keyword to see the specific failure (e.g., connection refused, parsing error, or access denied).

Copy code

bash
grep -i "audit" fe.log

5. Check
frontend_host_port
You are using

127.0.0.1:8030

. * Ensure your FE's

http_port

is indeed

(this is the default). * If you are running in a containerized environment (like Kubernetes) and the FE is not listening on the loopback interface, try using the actual Pod IP or a Service DNS name, though

127.0.0.1

usually works as the plugin runs inside the same process. Summary Checklist 1. Set

audit_log_json_format = false

fe.conf

. 2. Verify the table

starrocks_audit.audit_log

exists. 3. Grant

LOAD_PRIV

to user

auditloader

. 4. Check

fe.log

for "AuditLoader" related exceptions. References • administration/management/audit_loader.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/audit_loader.md • knowledge/usage/best-practice.md

Rumit Jain

08/03/2026, 11:48 AM

@Rocky My fe.log is empty

Rocky

08/03/2026, 11:48 AM

If your

fe.log

is completely empty, it usually indicates that the logging subsystem (Log4j) is either failing to initialize or is being redirected elsewhere. Since you are troubleshooting the AuditLoader, having no logs is a major blocker. Please check the following in order: 1. Check
fe.out
When

fe.log

is empty, the most important file to check is fe.out
in the same log directory. * Startup Errors: If the FE failed to start (e.g., due to a port conflict or Java error), the error will be in

fe.out

, not

fe.log

. * Log4j Failures: If there is a syntax error in your configuration or a permissions issue that prevents Log4j from starting, the error message will be printed here. 2. Is
sys_log_to_console
enabled? In many containerized environments (like Kubernetes), StarRocks is configured to send logs to the console (

stdout/stderr

) instead of a file. * Check your

fe.conf

for

sys_log_to_console = true

. * Check if the environment variable

SYS_LOG_TO_CONSOLE=1

is set. * If enabled: You will not find logs in

fe.log

. Instead, you must use

kubectl logs <fe-pod-name>

docker logs

to see them. 3. Verify
sys_log_dir
and Permissions Ensure the directory where you expect the logs to be is actually the one StarRocks is using. * The default is

${STARROCKS_HOME}/log

. * Verify the FE process user has write permissions to that folder. * Check if the disk is full (

df -h

). If the disk is at 100%, the FE cannot write new logs. 4. Search for "AuditLoader" in other logs If

fe.log

is empty but the FE is running, check for these alternative files in the log directory: * fe.warn.log
: Significant errors are mirrored here. * fe.internal.log
: If the plugin is considered an internal module in your version, logs might go here. 5. Validate your
fe.conf
syntax Ensure there are no hidden characters or syntax errors in your

fe.conf

. In the snippet you provided earlier, the

run_mode = shared_data

block looked indented or had leading spaces. While StarRocks is usually flexible, ensure the parameters are at the start of the line. Summary of Action: Run

ls -al

in your log directory. If

fe.out

has content,

cat fe.out

. If you are on Kubernetes, run

kubectl logs <pod_name>

to find the "AuditLoader" error messages. References • administration/management/logs.md • administration/management/FE_configuration.md • faq/Deploy_faq.md • administration/management/audit_loader.md References • administration/management/logs.md • administration/management/FE_configuration.md

Rumit Jain

08/03/2026, 11:49 AM

@Rocky How to check Is
sys_log_to_console
enabled?

Rocky

08/03/2026, 11:50 AM

You can check if

sys_log_to_console

is enabled using the following three methods: 1. Using SQL (Recommended if FE is running) Connect to your StarRocks cluster via a MySQL client and run:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE 'sys_log_to_console';

* If the

Value

column is

true

, logs are being sent to the console (stdout/stderr) and will not appear in the

fe.log

file. 2. Check the Configuration File Open your

fe.conf

file (usually located in the

conf/

directory) and look for the parameter:

Copy code

properties
sys_log_to_console = true

If this line exists and is set to

true

, file logging is disabled. 3. Check Environment Variables In containerized environments (Docker/Kubernetes), this setting is often controlled by an environment variable. Run the following command in your terminal/pod:

Copy code

bash
env | grep SYS_LOG_TO_CONSOLE

If the output is

SYS_LOG_TO_CONSOLE=1

, then console logging is enabled. ────────── What to do if it IS enabled: If

sys_log_to_console

is true, you will find your logs here instead of the file: * Kubernetes: Run

kubectl logs <fe-pod-name>

. * Docker: Run

docker logs <container-id>

. * Linux/Bare Metal: Look at the screen where you started the process, or check the

fe.out

file which often captures the console output. Note: If you want to use the

AuditLoader

plugin, remember that it specifically looks for the physical

fe.audit.log

file on the disk. If you have moved to JSON logging or redirected everything to the console, the plugin may fail to find the source data it needs to load. References • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/audit_loader.md References • administration/management/FE_configuration.md

Rumit Jain

08/03/2026, 11:52 AM

@Rocky Is it mandaptory for logs to come in fe.log for auditLoader to load the table?

Rumit Jain

08/03/2026, 11:54 AM

@Rocky This is my fe.conf:

Copy code

run_mode = shared_data
    cloud_native_storage_type = S3
    aws_s3_path = starrocks-data
    aws_s3_region = us-east-1
    aws_s3_endpoint = <https://s3.us-east-1.amazonaws.com>
    enable_load_volume_from_conf=true
    aws_s3_use_instance_profile=false
    aws_s3_use_aws_sdk_default_behavior=true
    enable_trace_historical_node = true
    audit_log_modules = slow_query, query, connection
    audit_log_json_format = true
    qe_slow_log_ms = 1000
    aws_s3_enable_partitioned_prefix=true
    aws_s3_num_partitioned_prefix=64
    automated_cluster_snapshot_interval_seconds=3600
    JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx11g -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
    mysql_service_nio_enabled = true
    sys_log_level = INFO
    min_graceful_exit_time_second = 25
    label_keep_max_num=50000

and this :

Copy code

ADMIN SHOW FRONTEND CONFIG LIKE 'sys_log_to_console';
+--------------------+------------+-------+---------+-----------+---------+
| Key                | AliasNames | Value | Type    | IsMutable | Comment |
+--------------------+------------+-------+---------+-----------+---------+
| sys_log_to_console | []         | true  | boolean | false     |         |
+--------------------+------------+-------+---------+-----------+---------+
1 row in set (0.01 sec)

Starrocks >

Rocky

08/03/2026, 11:54 AM

Because

sys_log_to_console

is set to true
, all system logs are being redirected to the standard output (stdout) instead of the

fe.log

file. This is why your log file is empty. To fix your

auditLoader

and find the errors, follow these steps: 1. View the Logs in the Console Since you are likely running in a containerized environment (Kubernetes or Docker), you need to use the container's logging command to see why the plugin is failing: * Kubernetes:

kubectl logs <fe-pod-name>

* Docker:

docker logs <container-id>

Look for lines containing "AuditLoader" or "Stream load". These will tell you if the plugin is failing because of a password error, a missing table, or a parsing issue. 2. Mandatory Change: Disable JSON Audit Logs In your

fe.conf

, you have:

audit_log_json_format = true

The
auditLoader
plugin cannot parse JSON logs. It expects the standard pipe-separated text format. 1. Change this to

false

in your

fe.conf

. 2. Restart the FE. 3. Ensure the Audit Log File Exists The

auditLoader

plugin works by reading the physical

fe.audit.log

file from the disk. When

sys_log_to_console

true

, sometimes the audit logs are also only sent to the console. Check your StarRocks

log/

directory. If you do not see a file named

fe.audit.log

, the plugin will have nothing to read. If the file is missing, you may need to: 1. Set

sys_log_to_console = false

to force logs back to files. 2. OR check your

log4j2.properties

to ensure

fe.audit.log

is still being written to disk even if system logs go to the console. 4. Verify Database and User Ensure the following was done manually (the plugin won't do it for you): * Create the DB:

CREATE DATABASE starrocks_audit;

* Create the Table: Use the specific

CREATE TABLE

DDL for

audit_log

found in the AuditLoader documentation. * Permissions: Run

GRANT INSERT ON starrocks_audit.audit_log TO 'auditloader'@'%';

Summary of required
fe.conf
changes:

Copy code

properties
# Change this to false
audit_log_json_format = false

# (Optional) If fe.audit.log file is still not appearing in the log folder:
sys_log_to_console = false

References • administration/management/audit_loader.md • administration/management/FE_configuration.md • administration/management/logs.md

Rocky

08/03/2026, 11:54 AM

The reason your

fe.log

and

fe.audit.log

files are empty is confirmed: sys_log_to_console
is set to
true
. When this is enabled, StarRocks redirects all logs to the standard output (stdout) instead of writing them to files on the disk. Why this breaks AuditLoader The

auditLoader

plugin works by monitoring and reading the physical fe.audit.log
file on the local disk. Since

sys_log_to_console = true

, that file is empty, so the plugin has nothing to load into your table. How to fix it To get the AuditLoader working, you must make two changes to your

fe.conf

and restart the FE: 1. Disable console logging so that logs are written to files:

Copy code

properties
    sys_log_to_console = false

2. Disable JSON format because the AuditLoader plugin cannot parse JSON logs; it requires the default pipe-separated text format:

Copy code

properties
    audit_log_json_format = false

Verification Steps After restarting the FE with these changes: 1. Check the file system: Run

ls -l

in your log directory. You should see

fe.audit.log

growing in size as you run queries. 2. Check the plugin logs: Since logs are now going to files, you can check

fe.log

for any errors from the "AuditLoader" thread:

Copy code

bash
    grep "audit" fe.log

3. Check the table: After about 60 seconds (your

max_batch_interval_sec

), data should begin appearing in

starrocks_audit.audit_log

. Note: If you are running in Kubernetes, disabling

sys_log_to_console

will mean you can no longer see logs using

kubectl logs

. You will have to

kubectl exec

into the pod to read the log files or ensure your log collection system (like Fluentd or Filebeat) is configured to watch the log files on the files in the log directory. References • administration/management/audit_loader.md • administration/management/FE_configuration.md • administration/management/logs.md References • loading/s3.md • data_source/catalog/iceberg/iceberg_rest_s3.md • loading/loading_introduction/feature-support-loading-and-unloading.md

Rumit Jain

08/03/2026, 11:55 AM

@Rocky Seeing these in logs:

Copy code

2026-08-03 11:55:10.629Z WARN (starrocks-http-pool-0|144) [RestBaseAction.handleRequest():140] failed to process url: /api/starrocks_audit_db__/starrocks_audit_tbl__/_stream_load?=null
com.starrocks.authorization.AccessDeniedException: Access denied for root@127.0.0.1
	at com.starrocks.http.rest.RestBaseAction.execute(RestBaseAction.java:205) ~[fe-core-4.1.3.jar:?]
	at com.starrocks.http.rest.RestBaseAction.handleRequest(RestBaseAction.java:138) ~[fe-core-4.1.3.jar:?]
	at com.starrocks.http.HttpServerHandler.lambda$handleActionAsync$0(HttpServerHandler.java:140) ~[fe-core-4.1.3.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
java.lang.Exception: status is not TEMPORARY_REDIRECT 307 or HTTP_OK 200, status: 401, response: {"status":"FAILED","code":"1","msg":"Access denied for root@127.0.0.1","message":"Access denied for root@127.0.0.1"}, request is: curl -v -X PUT \
  -H "Authorization":"Basic cm9vdDo=" \
  -H "Expect":"100-continue" \
  -H "Content-Type":"text/plain; charset=UTF-8" \
  -H "max_filter_ratio":"1.0" \
  -H "column_separator":|*$&$*| \
  -H "row_delimiter":~*$&$*~ \
  -H "columns":"queryId, timestamp, queryType, clientIp, user, authorizedUser, resourceGroup, catalog, db, state, errorCode,queryTime, scanBytes, scanRows, returnRows, cpuCostNs, memCostBytes, stmtId, isQuery, feIp, stmt, digest, planCpuCosts, planMemCosts, pendingTimeMs, candidateMVs, hitMvs, warehouse" \
  "<http://127.0.0.1:8030/api/starrocks_audit_db__/starrocks_audit_tbl__/_stream_load>?"
	at com.starrocks.plugin.audit.StarrocksStreamLoader.loadBatch(StarrocksStreamLoader.java

Rocky

08/03/2026, 11:56 AM

The error

Access denied for root@127.0.0.1

with a

status confirms that the AuditLoader plugin cannot authenticate with the StarRocks FE to perform the Stream Load. The log shows the plugin is sending the header

Authorization: Basic cm9vdDo=

, which translates to user

root

with no password. 1. Verify your
root
password If you have set a password for the

root

user in StarRocks, the AuditLoader will fail until you provide that password in its configuration. You can verify if a password is required by trying to connect via the MySQL client from the same machine without a password:

Copy code

bash
mysql -h 127.0.0.1 -P 9030 -u root

If this prompts for a password or denies access, then the AuditLoader's current configuration is incorrect. 2. Update the AuditLoader configuration To fix this, you must update the

plugin.conf

file inside the AuditLoader package and reinstall it: 1. Uninstall the current plugin:

Copy code

sql
    UNINSTALL PLUGIN AuditLoader;

2. Locate and unzip your
auditloader.zip
(the one you used for installation). 3. Edit
plugin.conf
: * Find the

user

and

password

lines. * Set

user=root

. * Set

password=your_actual_root_password

. * Ensure

frontend_host_port=127.0.0.1:8030

. 4. Re-zip the package:

Copy code

bash
    zip -q -m -r auditloader.zip auditloader.jar plugin.conf plugin.properties

5. Re-install the plugin:

Copy code

sql
    INSTALL PLUGIN FROM "/path/to/auditloader.zip";

3. Permissions Check (If not using
root
) If you prefer not to use the

root

user, you can create a dedicated user, but it must have the

LOAD

privilege on the audit table:

Copy code

sql
CREATE USER 'audit_admin'@'%';
SET PASSWORD FOR 'audit_admin'@'%' = PASSWORD('your_password');
GRANT INSERT ON starrocks_audit_db__.starrocks_audit_tbl__ TO USER 'audit_admin'@'%';

Then use

audit_admin

in your

plugin.conf

. 4. Important: Disable JSON logging As mentioned previously, your

fe.conf

showed

audit_log_json_format = true

. The AuditLoader plugin cannot parse JSON logs. * Ensure

audit_log_json_format = false

is set in

fe.conf

. * Restart the FE if you changed this. Once these steps are complete, the

401 Access denied

errors should disappear from your

fe.log

, and data will begin to flow into the table. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md

Rocky

08/03/2026, 11:56 AM

No, it is not mandatory for logs to be in

fe.log

for the plugin to work, but it is mandatory for the plugin to be able to write to its own status logs there so you can see why it's failing. Here is the distinction you need to understand: 1.
fe.log
vs
fe.audit.log
* fe.log
: This is the system log where StarRocks (and the AuditLoader plugin) writes its status, heartbeat, and errors. If this file is empty, you cannot see the error messages the plugin is throwing when it tries to load data. * fe.audit.log
: This is the file that contains the actual query records. The AuditLoader plugin typically tails this file or hooks into the events that generate it to collect data for the

audit_log

table. 2. The
audit_log_json_format
Conflict The most likely reason your table is empty is your setting:

audit_log_json_format = true

. * The

AuditLoader

plugin is specifically written to parse the default pipe-separated text format of StarRocks audit logs. * When you switch to JSON format, the plugin's internal parser fails to recognize the log entries, and as a result, it has nothing to "load." You must set
audit_log_json_format = false
in your
fe.conf
for the AuditLoader plugin to work. 3. Why is your
fe.log
empty? If

fe.log

is empty, you are "flying blind." It is likely empty because: * Console Logging: You might have

sys_log_to_console = true

fe.conf

, which redirects all system logs (including AuditLoader errors) to the standard output (check

fe.out

or your container logs). * Permissions: The FE process may not have write permissions to the log directory. Recommended Steps to Fix: 1. Revert to Text Logs: Set

audit_log_json_format = false

fe.conf

and restart the FE. 2. Find the Errors: If data still doesn't appear, you must find where your system logs are going. Check

fe.out

or run

kubectl logs <pod-name>

(if on K8s). Look for lines containing

AuditLoader

. 3. Check the Audit File: Verify that

fe.audit.log

actually has content. If that file is empty, the plugin has nothing to process. 4. Confirm Plugin Load: Run

SHOW PLUGINS;

in MySQL to ensure

AuditLoader

is listed and the status is

INSTALLED

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rocky

08/03/2026, 11:56 AM

Rumit Jain

08/03/2026, 12:12 PM

@Rocky These are current permissins on auditLoader user:

Copy code

SHOW GRANTS FOR 'auditloader';
+-------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------+
| UserIdentity      | Catalog         | Grants                                                                                                                                 |
+-------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------+
| 'auditloader'@'%' | default_catalog | GRANT DELETE, DROP, INSERT, SELECT, ALTER, EXPORT, UPDATE, REFRESH ON ALL TABLES IN DATABASE starrocks_audit TO USER 'auditloader'@'%' |
+-------------------+-----------------+----------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

Starrocks >

Rocky

08/03/2026, 12:12 PM

The permissions you have granted to the

auditloader

user are sufficient for the plugin to function in StarRocks 3.x. Specifically, the

INSERT

privilege you've granted on all tables in the

starrocks_audit

database is the primary requirement for the plugin to perform its internal Stream Load tasks. Key points to verify based on your configuration: 1. Database and Table Names: Your grants are for a database named

starrocks_audit

. Ensure that in your

plugin.conf

(inside the

auditloader.zip

), the

database

and

table

parameters match this exactly. If your

plugin.conf

still uses the example name

starrocks_audit_db__

, the load will fail even with the right permissions. 2. User Identity: The user in

plugin.conf

must be exactly

auditloader

. StarRocks permissions are case-sensitive and identity-specific. 3. Catalog Context: Your grant is on

default_catalog

. This is correct, as

AuditLoader

typically works within the internal StarRocks storage. 4. Dynamic Partitions: While the user has permissions to load data, the load will fail if the target table does not have a valid partition for the current time. * Verify partitions exist:

SHOW PARTITIONS FROM starrocks_audit.<your_table_name>;

* If no partitions exist, check if

dynamic_partition.enable

true

in your table properties. Troubleshooting "Empty Logs" with these Permissions: Since your

fe.log

was empty earlier, even if these permissions are correct, the plugin might be failing silently or logging elsewhere. * If you are on Kubernetes: The

auditloader

user's connection attempts and any "Access Denied" errors will appear in the FE pod logs (

kubectl logs <fe-pod>

). * Audit File Check: Confirm that the FE is actually generating the source file. Run

ls -lh

in your log directory to see if

fe.audit.log

is growing. If the source file is empty, there is nothing for the user to load. Summary: Your

GRANT

list is excellent and covers more than the minimum required. Focus now on ensuring the

plugin.conf

matches these names and that your

fe.audit.log

file is actually being populated. References • administration/management/audit_loader.md • loading/loading_introduction/loading_concepts.md • sql-reference/sql-statements/account-management/GRANT.md References • loading/loading_introduction/loading_concepts.md • administration/user_privs/authorization/privilege_item.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/InsertInto.md • loading/Etl_in_loading.md

Rumit Jain

08/03/2026, 12:24 PM

@Rocky How to uninstall and reinstall the plugin?

Rocky

08/03/2026, 12:25 PM

To uninstall and then reinstall the AuditLoader plugin (usually done to update configuration settings), follow these steps: 1. Uninstall the current plugin Run the following SQL command in your MySQL client:

Copy code

sql
UNINSTALL PLUGIN AuditLoader;

* Verify: Run

SHOW PLUGINS;

to confirm that

AuditLoader

is no longer in the list. 2. Update the configuration (Optional) If you are reinstalling to fix a configuration issue (like the

user

password

, or

database

settings): 1. Unzip your existing

auditloader.zip

. 2. Edit the

plugin.conf

file with the correct details. 3. Re-zip the files. Ensure the structure inside the zip is flat (no sub-folders):

Copy code

bash
    zip -q -m -r auditloader.zip auditloader.jar plugin.conf plugin.properties

3. Reinstall the plugin Use the

INSTALL PLUGIN

command, providing the absolute path to the zip file on your FE node(s). From a local file path:

Copy code

sql
INSTALL PLUGIN FROM "/home/starrocks/auditloader.zip";

From an HTTP URL (Requires md5sum):

Copy code

sql
INSTALL PLUGIN FROM "<http://your-server/auditloader.zip>" 
PROPERTIES("md5sum" = "your_file_md5_here");

Important Notes: * Identical Paths: If you have multiple FEs, the

auditloader.zip

file must exist at the exact same absolute path on every FE machine. * Persistence: Do not delete the

.zip

file after installation. StarRocks needs it to stay in that location to reload the plugin if the FE restarts. * Verify: After installation, run

SHOW PLUGINS;

again. The

Status

should be

INSTALLED

. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md • knowledge/usage/best-practice.md

Rumit Jain

08/03/2026, 12:55 PM

@Rocky I am seeing these items as well in the audit table. How to stop them :

Copy code

0 |            0 | SHOW COMPUTE NODES              |
| 019fc798-e6f5-7d51-b86b-93c749f1c16c            | 2026-08-03 12:28:34 | root | 10.214.78.253:43552 |                 | query      | EOF   |         2 |         0 |        0 |         0 |            0 | SHOW COMPUTE NODES              |
| 019fc79b-f222-70bf-b642-01732a4b9ee7            | 2026-08-03 12:31:53 | root | 10.214.78.253:33410 |                 | query      | EOF   |         2 |         0 |        0 |         0 |            0 | SHOW COMPUTE NODES              |
| 019fc79d-e9f6-7d2c-8136-fca6f6b0526a            | 2026-08-03 12:34:02 |

Rocky

08/03/2026, 12:56 PM

To stop these specific system queries from filling up your audit table, you can use the filter
parameter in the AuditLoader configuration. This allows you to exclude queries based on their SQL text (

stmt

), user, or IP address. Method 1: Use the AuditLoader Filter (Recommended) The

AuditLoader

plugin supports a

filter

configuration that acts like a

WHERE

clause during the loading process. 1. Uninstall the current plugin:

Copy code

sql
    UNINSTALL PLUGIN AuditLoader;

2. Unzip your

auditloader.zip

and open

plugin.conf

. 3. Modify the
filter
line to exclude the specific statement. You can use standard SQL syntax:

Copy code

ini
    # Exclude queries starting with SHOW COMPUTE NODES
    filter=stmt not like 'SHOW COMPUTE NODES%'

If you want to exclude multiple patterns or users:

Copy code

ini
    filter=stmt not like 'SHOW COMPUTE NODES%' and user != 'root'

4. Re-zip the files (

auditloader.jar

plugin.conf

plugin.properties

) and reinstall:

Copy code

sql
    INSTALL PLUGIN FROM "/absolute/path/to/auditloader.zip";

Method 2: Suppress Internal SQL in FE Config If these queries are being generated internally by StarRocks itself (and not an external monitoring script), you can reduce the noise by changing a global FE configuration in `fe.conf`: * enable_internal_sql = false
: When set to

false

, internal system SQL statements are suppressed or replaced with a

in the audit logs. Note that this requires an FE restart to take effect. Why are you seeing these? Queries like

SHOW COMPUTE NODES

are typically health checks. They often come from: * StarRocks Operator: If you are on Kubernetes, the operator runs these to check node status. * Load Balancers: High-availability setups often run simple queries to ensure the FE is responsive. * Monitoring Tools: Prometheus exporters or custom scripts frequently poll these commands. Recommendation: Check the

clientIp

in your audit table for those rows. If the IP belongs to your K8s operator or a monitoring node, you can also filter by that IP in the

plugin.conf

(e.g.,

filter=clientIp not like '10.214.78.253%'

). References • administration/management/audit_loader.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md References • administration/management/FE_configuration.md • administration/management/audit_loader.md

Rumit Jain

08/03/2026, 1:01 PM

@Rocky What are these rows :

Copy code

connection-af534bc0-9ab6-4dd8-b5e2-bfd259ebf1df | 2026-08-03 12:42:38 | root | 10.214.78.253:34718 | null            | connection | OK    |        -1 |        -1 |       -1 |        -1 |           -1 |                                 |

Rocky

08/03/2026, 1:02 PM

These rows represent connection events (such as user logins and logouts). StarRocks version 3.0 and later includes a

connection

module in the audit logs. Since these are not SQL queries, fields like

queryTime

scanRows

, and

cpuCostNs

are set to

-1

null

, and the "Statement" (SQL text) is empty. To stop these rows from appearing in your audit table, you have two options: Method 1: Disable Connection Logging in FE (Recommended) You can tell the Frontends (FEs) to stop recording connection events in the

fe.audit.log

entirely. This reduces log noise at the source. 1. Run this command in your SQL client to check the current setting:

Copy code

sql
   ADMIN SHOW FRONTEND CONFIG LIKE 'audit_log_modules';

2. By default, it usually includes

query, slow_query, connection

. To remove

connection

, run:

Copy code

sql
   ADMIN SET FRONTEND CONFIG ("audit_log_modules" = "query, slow_query");

Note: This change is runtime-persistent on the leader FE, but for permanent safety, also update
fe.conf
on all FE nodes. Method 2: Filter them out in AuditLoader If you want the logs to exist on the disk but not in your database table, you can use the

filter

property in the AuditLoader plugin. 1. Uninstall the plugin:

UNINSTALL PLUGIN AuditLoader;

2. Edit
plugin.conf
inside your zip file and add/update the filter line:

Copy code

ini
   filter=queryType != 'connection'

3. Re-zip and Reinstall the plugin:

Copy code

sql
   INSTALL PLUGIN FROM "/path/to/auditloader.zip";

Which should you choose? * Choose Method 1 if you don't care about auditing login/logout times at all. * Choose Method 2 if you want to keep the record in the raw

fe/log/fe.audit.log

file for security forensics but want your StarRocks table to stay clean for performance analysis. References • administration/management/FE_configuration.md • administration/management/audit_loader.md • administration/management/logs.md References • administration/management/audit_loader.md • administration/management/logs.md • knowledge/usage/best-practice.md

Rumit Jain

08/03/2026, 1:03 PM

@Rocky How to add this filter in existing filter?

Copy code

# Filter conditions when importing audit information
filter=stmt not like 'SHOW COMPUTE NODES%'

Rocky

08/03/2026, 1:04 PM

To add an additional filter, you can combine conditions using the and
operator, just like a standard SQL

WHERE

clause. To exclude both the

SHOW COMPUTE NODES

statements and the

connection

events, update your

plugin.conf

to look like this:

Copy code

ini
# Filter conditions: exclude both health check queries and login/logout events
filter=stmt not like 'SHOW COMPUTE NODES%' and queryType != 'connection'

Steps to apply the change: 1. Uninstall the current plugin:

Copy code

sql
    UNINSTALL PLUGIN AuditLoader;

2. Edit the

plugin.conf

file with the combined filter line above. 3. Re-zip the files (

auditloader.jar

plugin.conf

plugin.properties

Copy code

bash
    zip -q -m -r auditloader.zip auditloader.jar plugin.conf plugin.properties

4. Reinstall the plugin from the absolute path:

Copy code

sql
    INSTALL PLUGIN FROM "/home/starrocks/auditloader.zip";

Filter Tips: * Case Sensitivity: The

stmt

and

queryType

values in the audit log are generally lowercase or exactly as they appear in the log file. * Multiple Conditions: You can add as many conditions as you need (e.g.,

and user != 'monitoring_user'

). * Syntax: The

filter

parameter uses the same syntax as the

WHERE

clause in a StarRocks Stream Load. References • administration/management/audit_loader.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md • knowledge/usage/best-practice.md

Rumit Jain

08/03/2026, 1:18 PM

@Rocky The StarRocks secret must be passed as a Kubernetes secret reference rather than being mounted or passed via an AWS secret. But I see in starrocks fe and cn, root user password is in environment variables. How we can avoid this as i am also using initPasswrord flow for root user. in fe + cn helm chart.

Rocky

08/03/2026, 1:18 PM

To avoid having the

root

password appear as a plain text environment variable in your FE and CN pods, you should use the secretKeyRef
mechanism within the

envVars

configuration. Even when using the

initPassword

flow, the StarRocks Operator or Helm chart might inject the password into the pods (e.g., as

MYSQL_PWD

STARROCKS_ROOT_PASSWORD

) to support health checks or internal management scripts. 1. Use
secretKeyRef
in Helm Values Instead of letting the chart/operator manage the environment variable injection automatically (which might result in plain text), explicitly define the environment variable using a reference to your Kubernetes secret. Update your

values.yaml

for both FE and CN:

Copy code

yaml
starrocks:
  initPassword:
    enabled: true
    passwordSecret: "starrocks-root-pass"  # Your existing secret name

  starrocksFESpec:
    feEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password

  starrocksCnSpec:
    cnEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: starrocks-root-pass
            key: password

2. Why is it there? (Probes and Health Checks) The FE and CN pods often require the root password for their

readinessProbe

and

livenessProbe

. These probes frequently execute commands like

mysqladmin ping

or internal health scripts that need credentials to verify the database is responsive. If you find that the Operator is still injecting a plain-text version of the password alongside your secret-ref version, you can try to suppress the unsupported environment variables using a specific Operator configuration (if supported by your version): * Check if your chart supports

KUBE_STARROCKS_UNSUPPORTED_ENVS

. This can be used to prevent specific keys from being passed to the container. 3. Check for Plain Text in
values.yaml
Ensure that you are not setting the password in plain text anywhere in your Helm values: * Do not use

starrocks.initPassword.password: "yourpassword"

. * Only use

starrocks.initPassword.passwordSecret: "your-secret-name"

. 4. Verify Root User Registration Note that the FE process itself does not need the

root

password to start (it reads the password from its internal metadata). The CN/BE registration is handled by the StarRocks Operator, which executes the

ALTER SYSTEM ADD...

commands. The Operator uses the

passwordSecret

you provided to perform these actions from outside the FE/CN pods, so the pods themselves should only have the password if needed for the health probes mentioned above. Summary of Recommendations: 1. Verify that

starrocks.initPassword.password

is empty/null. 2. Explicitly map the secret to the

MYSQL_PWD

environment variable using

valueFrom.secretKeyRef

in the

feEnvVars

and

cnEnvVars

sections of your Helm values. This satisfies the requirement of using a Kubernetes secret reference. References • quick_start/helm.md • deployment/helm.md • administration/management/FE_configuration.md References • quick_start/helm.md • deployment/helm.md • administration/management/FE_configuration.md References • quick_start/helm.md • administration/management/FE_configuration.md

Rumit Jain

08/03/2026, 1:39 PM

@Rocky Can we keep the password in a configmap?

Rocky

08/03/2026, 1:40 PM

Technically, you cannot use a

ConfigMap

for the

initPassword

automated flow, and from a security perspective, you should not use it for passwords. 1. The
initPassword
logic requires a Secret The StarRocks Operator and Helm chart are hardcoded to look for a

Secret

when the

initPassword

feature is enabled. The parameter

passwordSecret

specifically expects the name of a Kubernetes Secret object. There is no equivalent

passwordConfigMap

parameter in the chart. 2. Security Risks Kubernetes

ConfigMaps

are designed for non-sensitive configuration data (like

fe.conf

or environment variables like

LOG_LEVEL

). * Plain Text: ConfigMaps are stored in plain text in the etcd database. * Visibility: They are often visible in logs, describing the pod, or to anyone with basic read access to the namespace. * Compliance: Using a ConfigMap for a password will likely fail security audits (e.g., SOC2, HIPAA, PCI). 3. How to meet your "Secret Reference" requirement If your goal is to ensure the password is "passed as a Kubernetes secret reference" (and not plain text in the YAML or mounted as a file), the

secretKeyRef

approach mentioned previously is exactly what you need. In your

values.yaml

, you map the environment variable to the secret without exposing the actual password value:

Copy code

yaml
starrocks:
  initPassword:
    enabled: true
    passwordSecret: "my-starrocks-root-secret" # Refers to a Secret object

  starrocksFESpec:
    feEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: my-starrocks-root-secret # The reference
            key: password                 # The key inside the secret

If you still want to use a ConfigMap (Not Recommended) If you decide to proceed with a ConfigMap despite the risks, you would have to: 1. Disable the

initPassword

automation (

enabled: false

). 2. Manually initialize the password via a MySQL client. 3. Manually map the environment variable in your Helm values:

Copy code

yaml
    feEnvVars:
      • name: MYSQL_PWD
        valueFrom:
          configMapKeyRef:
            name: my-configmap
            key: root-password

Recommendation: Stick with Kubernetes Secrets. They fulfill your requirement of being a "secret reference" and are the standard, secure way to handle credentials in Kubernetes. References • quick_start/helm.md • deployment/helm.md • administration/management/FE_configuration.md References • quick_start/helm.md • administration/management/FE_configuration.md

Rumit Jain

08/03/2026, 1:42 PM

@Rocky Can we do like this for root user?

Copy code

secrets:
    - name: sr-admin-user-password
      mountPath: /etc/starrocks/secrets/sr-admin-user-password

Rocky

08/03/2026, 1:43 PM

The snippet you provided is for mounting a secret as a file. However, your initial requirement stated that the secret must be passed as a secret reference rather than being mounted. In Kubernetes terminology: * Secret Reference (
valueFrom.secretKeyRef
): The password is and remains in the Secret object, but the container sees it as an environment variable. * Secret Mount (
mountPath
): The password is projected into the container as a file. Why the
mountPath
approach is usually not recommended for the root user: 1. StarRocks Logic: StarRocks (FE and CN) does not natively look for a "password file" on disk to authenticate the

root

user for internal cluster management or health checks. It typically expects the password via the

MYSQL_PWD

environment variable or the standard metadata. 2. initPassword
Flow: The automated

initPassword

feature in the StarRocks Operator is specifically designed to work with a Secret Reference (

passwordSecret

). It does not currently support reading the initialization password from a mounted file path. 3. Probes and Health Checks: The StarRocks pods use health check scripts (like

health_check.sh

) that look for

MYSQL_PWD

. If you mount the secret as a file, these scripts will fail unless you also modify the probe commands to read from that file. How to achieve your goal (Secret Reference +
initPassword
): To satisfy the requirement of using a Kubernetes secret reference while keeping the password out of your YAML/plain text, use the following configuration in your

values.yaml

. This approach ensures that no plain-text password exists in your configuration files, fulfilling the "secret reference" requirement.

Copy code

yaml
starrocks:
  initPassword:
    enabled: true
    # This is the "Secret Reference". 
    # The Operator uses this to set the root password initially.
    passwordSecret: "sr-admin-user-password" 

  starrocksFESpec:
    feEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: sr-admin-user-password
            key: password  # The key inside your secret

  starrocksCnSpec:
    cnEnvVars:
      - name: MYSQL_PWD
        valueFrom:
          secretKeyRef:
            name: sr-admin-user-password
            key: password

If you MUST avoid environment variables entirely: If your security policy forbids both mounting and environment variables, you cannot use the

initPassword

automated flow. You would have to: 1. Initialize the password manually via a MySQL client. 2. Disable the default probes (liveness/readiness) that require

MYSQL_PWD

. 3. Accept that the StarRocks Operator may have limited ability to manage/auto-register nodes if it cannot authenticate as

root

. Recommendation: Stick with the

secretKeyRef

(Secret Reference) approach. It is the Kubernetes-native way to handle secrets securely without hardcoding values or relying on file-based mounts that the application isn't built to read. References • quick_start/helm.md • deployment/helm.md • administration/management/FE_configuration.md References • quick_start/helm.md • administration/management/FE_configuration.md

Rumit Jain

08/04/2026, 3:29 AM

@Rocky Is htere a way to automate the installation of auditloader plugin so that it gets deployed along with FE pods itself during the helm install of starrocks FE_CN cluster?

Rocky

08/04/2026, 3:29 AM

There is currently no native "one-line" property in the StarRocks Helm chart or Operator (like

auditLoader.enabled: true

) to automate the installation of the AuditLoader plugin. To automate this during your Helm deployment, the recommended approach is to use a Kubernetes Job with a Helm post-install hook. This job will wait for the FE to be ready and then execute the necessary SQL commands. Prerequisites The

INSTALL PLUGIN

command requires the

.zip

package to be accessible to all FE pods. You should either: 1. Host the zip file on an internal HTTP server (e.g., Nginx, S3, or Artifactory). 2. Mount a Shared Volume to the FE pods containing the zip file. Automation Step: Post-Install Job You can add a Job template to your Helm chart (or as a separate manifest) that runs the following logic:

Copy code

yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: starrocks-auditloader-init
  annotations:
    "<http://helm.sh/hook|helm.sh/hook>": post-install,post-upgrade
    "<http://helm.sh/hook-delete-policy|helm.sh/hook-delete-policy>": hook-succeeded
spec:
  template:
    spec:
      containers:
      - name: auditloader-installer
        image: mariadb:latest # Or any image with a mysql client
        env:
        - name: FE_HOST
          value: "kube-starrocks-fe-service" # Your FE service name
        - name: ROOT_PWD
          valueFrom:
            secretKeyRef:
              name: starrocks-root-pass
              key: password
        command: ["/bin/sh", "-c"]
        args:
        - |
          # 1. Wait for FE to be ready
          echo "Waiting for FE..."
          until mysql -h$FE_HOST -P9030 -uroot -p$ROOT_PWD -e "SELECT 1"; do sleep 5; done

          # 2. Create Database and Table
          mysql -h$FE_HOST -P9030 -uroot -p$ROOT_PWD <<EOF
          CREATE DATABASE IF NOT EXISTS starrocks_audit_db__;
          CREATE TABLE IF NOT EXISTS starrocks_audit_db__.starrocks_audit_tbl__ (
            queryId VARCHAR(64), 
            timestamp DATETIME,
            queryType VARCHAR(12),
            ... (add full schema here) ...
          ) ENGINE=OLAP 
          DISTRIBUTED BY HASH(queryId) BUCKETS 3;
          
          # 3. Install Plugin (using a remote URL is easiest for automation)
          INSTALL PLUGIN FROM "<http://your-internal-repo/auditloader.zip>" 
          PROPERTIES("md5sum" = "your_zip_md5");
          EOF
      restartPolicy: OnFailure

Key Considerations for Automation 1. Schema Versioning: The AuditLoader table schema changes slightly between StarRocks versions. Ensure the SQL in your automation matches the version you are deploying. 2. plugin.conf
: The

auditloader.zip

contains a

plugin.conf

. For automation to be fully "hands-off," you must pre-configure the

user

password

, and

frontend_host_port

inside that zip file before uploading it to your HTTP server. 3. Idempotency: Use

CREATE TABLE IF NOT EXISTS

and check

SHOW PLUGINS

before running

INSTALL PLUGIN

to prevent the Job from failing on upgrades or re-runs. Alternative: Sidecar / InitContainer While you can use an

initContainer

to download the zip file to a shared volume, the

INSTALL PLUGIN

command must be run against a running FE. Therefore, a Job or a Sidecar script is more reliable than an

initContainer

for the SQL execution part. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • quick_start/helm.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md

Rocky

08/04/2026, 3:29 AM

• quick_start/helm.md • sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md References • quick_start/helm.md

Rumit Jain

08/04/2026, 3:33 AM

@Rocky If a new FE pod starts, does it automatically recognize the installed plugin from the FE metadata, or does the installation need to be repeated?

Rocky

08/04/2026, 3:34 AM

Yes, once you install a plugin like

AuditLoader

on the Leader FE, the installation is recorded in the StarRocks metadata (Edit Logs). However, there are specific behaviors regarding how new or restarted FE pods handle the actual plugin files: 1. Metadata Persistence The cluster's metadata stores the plugin's status, name, and the source path/URL used during the initial

INSTALL PLUGIN

command. When a new FE pod joins the cluster (as a Follower or Observer), it replays the Edit Logs and sees the installation record. 2. Automatic Re-download (The "URL" Advantage) If you installed the plugin using a URL (e.g.,

INSTALL PLUGIN FROM "<http://internal-repo/auditloader.zip>"

), new FE pods will automatically attempt to download and load the plugin from that URL when they start up and process the metadata. * Requirement: The new pod must have network access to that URL. * Advantage: This is the best way to automate the process for Kubernetes/Helm, as it ensures all pods (even those starting months later) can retrieve the binaries. 3. Local Path Issues (Avoid for Automation) If you installed the plugin using a local file path (e.g.,

/home/starrocks/auditloader.zip

), a new FE pod will look for that exact path on its own local disk. * In Kubernetes, new pods have ephemeral filesystems. Unless you manually put the zip file into the new pod at that exact path before it starts, the plugin loading will fail on that specific node. 4. Configuration Persistence (
plugin.conf
) The

AuditLoader

configuration (like the target database, user, and password) is stored inside the

plugin.conf

file within the zip package. * When a new node re-downloads the zip from a URL, it uses the configuration inside that zip. * Important: If you need to change the AuditLoader password later, you must either update the zip at the URL and re-install, or use

SET GLOBAL

(if the plugin supports dynamic updates) or manually update the file on all pods. Summary for your Automation: To ensure the plugin "sticks" across pod restarts and scaling: 1. Use a URL: Always install using

INSTALL PLUGIN FROM "http://..."

. 2. Persistent Storage: While not strictly required for the plugin binaries if you use a URL, ensure your FE pods use Persistent Volumes (PVCs) for their

meta

directory so they don't lose the record of the installation itself. 3. Check Status: You can verify if the plugin is active on all nodes by running:

Copy code

sql
    SHOW PLUGINS;

Check the

Status

column to ensure it says

INSTALLED

ACTIVE

on every FE host listed. References • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md References • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/SHOW_PLUGINS.md References • deployment/upgrade.md • administration/management/FE_configuration.md • deployment/deploy_manually.md

Rocky

08/04/2026, 3:34 AM

• administration/management/BE_configuration.md • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md

Rumit Jain

08/04/2026, 3:36 AM

@Rocky Does AuditLoader support reading credentials from environment variables or an external configuration file instead of embedding them in

plugin.conf

Rocky

08/04/2026, 3:37 AM

No, AuditLoader does not support reading credentials directly from environment variables or external configuration files. The plugin's configuration is strictly tied to the

plugin.conf

file contained within the

auditloader.zip

package. To manage credentials securely or avoid hardcoding plain-text passwords, you have two primary options: 1. Password Encryption (Recommended) While you cannot use environment variables, you can avoid plain-text passwords in

plugin.conf

by using the built-in encryption feature: 1. Generate an encrypted password in StarRocks:

Copy code

sql
    -- Replace 'your_password' and 'your_secret_key' (max 16 bytes)
    SELECT TO_BASE64(AES_ENCRYPT('your_password','your_secret_key'));

2. Update
plugin.conf
with the resulting string and the secret key:

Copy code

properties
    password = <your_encrypted_base64_string>
    secret_key = your_secret_key

secret_key

is provided, AuditLoader automatically decrypts the

password

field at runtime. 2. Manual Automation via Helm/Kubernetes Since you are using Helm, you can automate the "injection" of these credentials during your deployment process, even though the plugin itself doesn't read env vars: * Step A: Create a Kubernetes Secret containing your

plugin.conf

content. * Step B: Use a Helm post-install Job or a Sidecar to: 1. Download the standard

auditloader.zip

. 2. Replace the

plugin.conf

inside the zip with the one from your Kubernetes Secret (using

zip -u

). 3. Upload the modified zip to an internal HTTP server or mount it to a shared volume. 4. Run the

INSTALL PLUGIN FROM ...

command. Why this limitation exists AuditLoader runs as a background thread within the FE process. It is designed to be self-contained within its plugin package. Because it uses Stream Load (HTTP PUT) to send logs back into StarRocks, it requires its own set of credentials and an endpoint (

frontend_host_port

), which it pulls exclusively from its internal configuration file. Summary: You must put the credentials in

plugin.conf

, but you should use

secret_key

to encrypt them so they are not stored in plain text. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • administration/management/FE_configuration.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md References • administration/management/BE_configuration.md • sql-reference/System_variable.md

Rumit Jain

08/04/2026, 3:39 AM

@Rocky If we update

plugin.conf

(new encrypted password) and replace the ZIP at the same URL, will existing FE nodes automatically reload the new configuration, or is an

UNINSTALL PLUGIN

INSTALL PLUGIN

(or FE restart) required?

Rocky

08/04/2026, 3:40 AM

No, existing FE nodes will not automatically reload the new configuration if you simply replace the ZIP file at the URL. StarRocks downloads and unzips the plugin to a local directory (usually

fe/plugins/

) only during the initial

INSTALL PLUGIN

execution or when a new/restarted FE pod first initializes. Once the plugin is loaded into memory, it does not poll the source URL or the local file for changes. To update the configuration, you must follow these steps: 1. The Standard Way: Uninstall/Install The official way to modify

plugin.conf

is to cycle the plugin. This ensures the metadata is updated and all FE nodes re-download the package.

Copy code

sql
-- 1. Remove the existing plugin
UNINSTALL PLUGIN AuditLoader;

-- 2. Install again with the new ZIP URL and the NEW MD5 sum
INSTALL PLUGIN FROM "<http://your-internal-repo/auditloader.zip>" 
PROPERTIES("md5sum" = "NEW_MD5_OF_YOUR_UPDATED_ZIP");

*Note: If you use the same URL, you must update the

md5sum

property. If the MD5 in the command doesn't match the ZIP at the URL, the installation will fail.* 2. The Kubernetes/Restart Way (Not Recommended) If an FE pod restarts, it will re-download the plugin based on the metadata. However, this is problematic because: * MD5 Mismatch: If the metadata contains the old MD5 but the URL provides a new ZIP, the FE will likely fail to start or fail to load the plugin because the checksum validation fails. * Consistency: Different FE pods might end up with different versions of the configuration if some restart and others don't. 3. Can I use
SET GLOBAL
? Unlike standard StarRocks variables, AuditLoader configurations (like

user

password

frontend_host_port

) are not mutable via

SET GLOBAL

. They are read once from

plugin.conf

when the plugin thread starts. Summary of Best Practice for Updates: If you are automating this in Kubernetes: 1. Update your

plugin.conf

and create a new ZIP. 2. Upload it to your server. 3. Run a small script or Kubernetes Job that executes the

UNINSTALL

and

INSTALL

SQL commands. This is the only way to ensure the new credentials are propagated to all FE nodes and that the internal background threads are restarted with the new settings. References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • administration/management/audit_loader.md • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • sql-reference/sql-statements/cluster-management/plugin/UNINSTALL_PLUGIN.md References • loading/Flink-connector-starrocks.md • loading/Kafka-connector-starrocks.md

Rocky

08/04/2026, 3:40 AM

• administration/management/FE_configuration.md • loading/Spark-connector-starrocks.md • data_source/catalog/hudi_catalog.md • data_source/catalog/hive_catalog.md

Rumit Jain

08/04/2026, 3:42 AM

@Rocky the Leader propagates the metadata and each FE independently downloads the ZIP, or the plugin must be installed only on the Leader and the rest synchronize automatically.

Rocky

08/04/2026, 3:42 AM

The correct understanding is a combination of both: The Leader propagates the metadata, and then each FE independently handles the plugin files. Here is the breakdown of how the synchronization works: 1. Metadata Propagation (The Leader's Job) When you run

INSTALL PLUGIN

, the Leader FE records the installation command (including the source path or URL and properties like MD5) into the cluster’s Edit Logs. This metadata is synchronized to all Follower and Observer FEs via the BDBJE protocol. 2. Independent Action (The Followers' Job) Each FE node, upon receiving the metadata update, independently attempts to "materialize" the plugin: * If using a URL: Each FE node will independently attempt to download the ZIP file from that URL, verify its MD5, and unzip it into its local

fe/plugins

directory. * If using a Local Path: Each FE node will look for the file at that exact same absolute path on its own local disk. Why this matters for your setup: * The "Installation Fails" Trap: If you use a local path (e.g.,

/tmp/auditloader.zip

) and the file only exists on the Leader's disk, the installation command might return an error or the other FEs will show the plugin status as

ERROR

because they cannot find the file at that path on their own local filesystems. * Network Accessibility: If you use a URL, every FE pod (including future ones) must have network access to that URL. If a new pod starts and cannot reach the URL, it will fail to load the plugin even though it "knows" it should have it from the metadata. * Persistence: You must not delete the source ZIP (if local) or take down the URL after installation. If an FE pod restarts, it may need to refer back to that source to re-verify or re-load the plugin binaries. Summary The Leader does not "push" the plugin binaries to the other FEs. It only pushes the instruction (metadata). It is the responsibility of every FE in the cluster to be able to access the source (URL or local path) specified in that instruction. References • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • administration/management/audit_loader.md References • sql-reference/sql-statements/cluster-management/plugin/INSTALL_PLUGIN.md • loading/Flink_cdc_load.md • administration/management/audit_loader.md

Rumit Jain

08/04/2026, 1:11 PM

@Rocky Is there a way I can deploy Starrocks MCP in AWS as pod and than my agent can connect to it?

Rocky

08/04/2026, 1:11 PM

Yes, you can deploy the StarRocks MCP (Model Context Protocol) Server as a pod in AWS (typically on EKS) to act as a bridge between your AI agents and your StarRocks cluster. What is StarRocks MCP? It is a standalone server that implements Anthropic's Model Context Protocol. It allows LLM agents (like Claude, Cursor, or custom AI agents) to: * Explore your database schema (list tables/columns). * Execute SQL queries safely. * Retrieve metadata to reduce "hallucinations" during data analysis. ────────── Deployment Steps in AWS (EKS) Since the MCP server is a lightweight Python application, you can deploy it as a standard Kubernetes Deployment. 1. Containerize the MCP Server You will need to build a Docker image for the MCP server. You can use a simple

Dockerfile

based on Python:

Copy code

dockerfile
FROM python:3.11-slim
COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /uvbin/uv

WORKDIR /app
# Install the starrocks-mcp-server package
RUN /uvbin/uv tool install starrocks-mcp-server

# The server typically runs via stdio or an HTTP endpoint
ENTRYPOINT ["/root/.local/bin/starrocks-mcp-server"]

2. Create a Kubernetes Deployment Deploy the pod into your EKS cluster. The pod needs environment variables to connect to your StarRocks FE (Frontend).

Copy code

yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: starrocks-mcp
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: mcp-server
        image: your-repo/starrocks-mcp:latest
        env:
        - name: STARROCKS_HOST
          value: "starrocks-fe-service.default.svc.cluster.local"
        - name: STARROCKS_PORT
          value: "9030"
        - name: STARROCKS_USER
          value: "root"
        - name: STARROCKS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sr-pass
              key: password

3. Connect Your Agent How your agent connects depends on where the agent is running: * Internal Agent (Another Pod): If your AI agent is also a pod in the same cluster, it can connect directly to the MCP pod via a Kubernetes Service using the MCP protocol (often over SSE/HTTP). * External Agent (e.g., Claude Desktop or local IDE): * You can expose the MCP server using an AWS LoadBalancer Service. * Alternatively, for local development, use

kubectl port-forward

to map the MCP server to your local machine, allowing your local agent to "see" the StarRocks metadata. Configuration Variables The server relies on these standard environment variables to establish the StarRocks connection: * `STARROCKS_HOST`: The FE address. * `STARROCKS_PORT`: The MySQL query port (default

). * `STARROCKS_USER`: Database username. * `STARROCKS_PASSWORD`: Database password. * `STARROCKS_DATABASE`: (Optional) Default database. Important Note The StarRocks MCP server is an open-source project. You can find the source code and latest installation instructions on the official StarRocks GitHub repository. References • Quick Start - StarRocks • deployment/sr_operator.md • administration/management/FE_configuration.md References • deployment/sr_operator.md • integrations/authenticate_to_aws_resources.md

Rumit Jain

08/04/2026, 1:29 PM

@Rocky Where is the source code or package for

starrocks-mcp-server

? The Dockerfile uses:

Copy code

uv tool install starrocks-mcp-server

but the package isn't available on PyPI. Is it hosted on GitHub, a private registry, or another package index?

Rocky

08/04/2026, 1:29 PM

The official package name for the StarRocks Model Context Protocol server is mcp-server-starrocks
(following the standard

mcp-server-<name>

naming convention). Source Code and Repository The official source code is hosted on GitHub: * GitHub Repository: StarRocks/mcp-server-starrocks Package Index and Installation The package is indeed available on PyPI as

mcp-server-starrocks

. If you were searching for "starrocks-mcp-server," you likely encountered a "not found" error due to the name swap. To install it via

uv

pip

, use the following commands: Option 1: Install from PyPI (Recommended)

Copy code

bash
# Using uv (fastest)
uv tool install mcp-server-starrocks

# Using pip
pip install mcp-server-starrocks

Option 2: Install directly from GitHub If you need the absolute latest version from the main branch or are having trouble with the index:

Copy code

bash
uv tool install git+<https://github.com/StarRocks/mcp-server-starrocks.git>

Updated Dockerfile Snippet If you are building an image for AWS/EKS, use the corrected package name:

Copy code

dockerfile
FROM python:3.11-slim
COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /uvbin/uv

WORKDIR /app
# Corrected package name
RUN /uvbin/uv tool install mcp-server-starrocks

# Ensure the bin path is in PATH
ENV PATH="/root/.local/bin:${PATH}"

# Launch the server
ENTRYPOINT ["mcp-server-starrocks"]

Quick Verification You can verify the package is live by checking the mcp-server-starrocks PyPI page. References • Quick Start - StarRocks • administration/management/FE_configuration.md References • Quick Start - StarRocks • administration/management/FE_configuration.md • deployment/sr_operator.md • integrations/authenticate_to_aws_resources.md

Rumit Jain

08/04/2026, 1:43 PM

@Rocky I am seeing this in the pod:

Copy code

Installed 1 executable: mcp-server-starrocks
warning: `/root/.local/bin` is not on your PATH. To use installed tools, add the directory to your PATH.
+ echo 'Starting StarRocks MCP Server...'
+ exec /root/.local/bin/mcp-server-starrocks --mode streamable-http --host 0.0.0.0 --port 3000
Starting StarRocks MCP Server...
Traceback (most recent call last):
  File "/root/.local/bin/mcp-server-starrocks", line 4, in <module>
    from mcp_server_starrocks import main
  File "/root/.local/share/uv/tools/mcp-server-starrocks/lib/python3.11/site-packages/mcp_server_starrocks/__init__.py", line 14, in <module>
    from . import server
  File "/root/.local/share/uv/tools/mcp-server-starrocks/lib/python3.11/site-packages/mcp_server_starrocks/server.py", line 27, in <module>
    server = Server("mcp-server-starrocks", SERVER_VERSION)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: Server.__init__() takes 2 positional arguments but 3 were given

Rocky

08/04/2026, 1:44 PM

This error is caused by a breaking change in the Model Context Protocol (MCP) Python SDK v2.0.0, which was released very recently (late July 2026). The

mcp-server-starrocks

package (v0.4.0) was written for the v1.x SDK, where the

Server

constructor required two positional arguments:

name

and

version

. In the new v2.0 SDK, the

Server

class has been rebuilt, and it no longer accepts the version as a second positional argument. The Fix: Pin the MCP SDK Version Since you are using

uv tool install

, it is fetching the latest (incompatible) version of the

mcp

library by default. You need to force it to use a version earlier than

2.0.0

. Option 1: Update your Dockerfile (Recommended) Change your installation line to include a version constraint for the dependency:

Copy code

dockerfile
# Force uv to use the compatible v1.x SDK
RUN /uvbin/uv tool install mcp-server-starrocks --with "mcp<2.0.0"

Option 2: Run via
uv run
If you are running this in a container or local shell directly:

Copy code

bash
uv run --with "mcp-server-starrocks" --with "mcp<2.0.0" mcp-server-starrocks --mode streamable-http ...

Why this happened * Today's Context: You are likely installing the server in an environment that pulls the latest PyPI packages. * The Break: The

mcp

package jumped from

1.x

2.x

. In

1.x

, the code was

Server("name", "version")

. In

2.x

, this now triggers a

TypeError

because the signature changed. * Upstream Status: The StarRocks team will likely release an update (e.g., v0.5.0+) soon to support the v2 SDK, but until then, pinning to
<2.0.0
is the standard workaround. Corrected Dockerfile Example

Copy code

dockerfile
FROM python:3.11-slim
COPY --from=<http://ghcr.io/astral-sh/uv:latest|ghcr.io/astral-sh/uv:latest> /uv /uvbin/uv

WORKDIR /app

# Pin mcp to v1.x to avoid the Server.__init__ TypeError
RUN /uvbin/uv tool install mcp-server-starrocks --with "mcp<2.0.0"

ENV PATH="/root/.local/bin:${PATH}"

# Example entrypoint using standard environment variables
ENTRYPOINT ["mcp-server-starrocks", "--mode", "streamable-http", "--host", "0.0.0.0", "--port", "3000"]

References • Quick Start - StarRocks • deployment/sr_operator.md References • Quick Start - StarRocks • deployment/sr_operator.md • administration/management/FE_configuration.md • integrations/authenticate_to_aws_resources.md

Rumit Jain

08/04/2026, 3:12 PM

@Rocky Why I am seeing these errors in starrocks mcp pod?

Copy code

2026-08-04 15:11:17 | INFO | mcp_server_starrocks.connection_health_checker:start:96 | Connection health checker thread started
2026-08-04 15:11:17 | INFO | mcp_server_starrocks.server:main:588 | CORS enabled for streamable-http transport - allowing all origins
2026-08-04 15:11:17 | WARNING | mcp_server_starrocks.connection_health_checker:_connection_health_checker_loop:66 | Database connection is unhealthy: Error executing statement 'show databases': 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0


╭──────────────────────────────────────────────────────────────────────────────╮
│                                                                              │
│                                                                              │
│                         ▄▀▀ ▄▀█ █▀▀ ▀█▀ █▀▄▀█ █▀▀ █▀█                        │
│                         █▀  █▀█ ▄▄█  █  █ ▀ █ █▄▄ █▀▀                        │
│                                                                              │
│                                                                              │
│                                                                              │
│                                FastMCP 3.4.5                                 │
│                            <https://gofastmcp.com>                             │
│                                                                              │
│                 🖥  Server:      mcp-server-starrocks, 3.4.5                  │
│                 🚀 Deploy free: <https://horizon.prefect.io>                   │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯


[08/04/26 15:11:17] INFO     Starting MCP server                transport.py:361
                             'mcp-server-starrocks' with
                             transport 'streamable-http' on
                             <http://0.0.0.0:3000/mcp>
INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on <http://0.0.0.0:3000> (Press CTRL+C to quit)
2026-08-04 15:11:47 | WARNING | mcp_server_starrocks.connection_health_checker:_connection_health_checker_loop:77 | Database connection remains unhealthy: Error executing statement 'show databases': 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system erro

Rocky

08/04/2026, 3:12 PM

The error

2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

indicates that the MCP server successfully opened a TCP connection to the StarRocks FE, but the FE closed the connection immediately before the MySQL handshake could finish. In an AWS EKS environment, this usually points to one of the following issues: 1. FE "Readiness" & Binding Issues If the StarRocks FE is still starting up or hasn't fully joined the cluster, its MySQL port (9030) might be open at the OS level but the service isn't ready to handle queries. * Check FE Status: Run

kubectl get pods

and ensure the FE pods show

1/1 READY

. * Check FE Logs: Look for lines like

FE type: FOLLOWER

FE type: LEADER

. If you see errors about "metadata not synced" or "replaying edit log," the FE will drop connections. * Priority Networks: If your FE is binding to the wrong internal IP (e.g., a Docker-internal 172.x.x.x instead of the VPC/EKS node IP), it won't respond to the MCP pod correctly. Ensure

priority_networks

in your

fe.conf

matches your EKS pod CIDR. 2. Network Security (Security Groups/NetworkPolicies) Even if the TCP connection "starts," some network filters (like AWS Security Groups or K8s NetworkPolicies) allow the SYN/ACK but drop the payload if they are misconfigured for stateful inspection. * Security Groups: If your StarRocks cluster and MCP pod are in different Security Groups, ensure the StarRocks SG allows Inbound TCP 9030 from the MCP pod's SG. * NetworkPolicy: If you have a

NetworkPolicy

in the namespace, ensure it explicitly allows egress from the MCP pod to the FE and ingress to the FE pod on port 9030. 3. StarRocks Internal Whitelisting Check if your StarRocks cluster has global connection limits or IP restrictions. * Check if

max_connection

has been reached (though this usually gives a "Too many connections" error). * Ensure the user (e.g.,

root

) is allowed to connect from the MCP pod's IP range (StarRocks uses MySQL-style

user@'host'

permissions). 4. FE Configuration (
mysql_service_nio_enabled
) In some StarRocks versions, the NIO (Non-blocking I/O) for the MySQL service can cause issues in specific containerized environments. * Try setting

mysql_service_nio_enabled = false

fe.conf

and restarting the FE to see if it stabilizes the initial packet exchange. Recommended Debugging Steps Run these commands from inside your MCP pod to narrow it down: 1. Test the port:

nc -zv <FE_HOST> 9030

(If this fails, it's a basic networking/firewall issue). 2. Manual Login: If the pod has a mysql client:

mysql -h <FE_HOST> -P 9030 -u root

. If this also fails with "reading initial communication packet," the issue is definitely on the FE/Network side, not the MCP code. Note on FastMCP: Your logs show

FastMCP 3.4.5

. This is a newer framework used by recent versions of the StarRocks MCP server. The error you're seeing is coming from the database driver level, not the framework itself.

Rumit Jain

08/04/2026, 4:05 PM

@Rocky How to check this value?

mysql_service_nio_enabled

Rocky

08/04/2026, 4:05 PM

To check the value of

mysql_service_nio_enabled

, you can use the ADMIN SHOW FRONTEND CONFIG
command through any MySQL-compatible client connected to your StarRocks cluster. Option 1: Using SQL (Recommended) Connect to your StarRocks FE (port 9030) and run:

Copy code

sql
ADMIN SHOW FRONTEND CONFIG LIKE 'mysql_service_nio_enabled';

* If the result is
true
: The FE is using asynchronous Non-blocking I/O (NIO) for MySQL connections. * If the result is
false
: It is using standard Blocking I/O (BIO). * If no row is returned: The parameter is not explicitly set in your

fe.conf

and is using the system default (which is

true

in recent versions). Option 2: Checking the Configuration File If you cannot connect via MySQL due to the "Lost connection" error, you must check the configuration file directly on the FE pod/server: 1. Locate the file: Usually found at

conf/fe.conf

within the StarRocks installation directory. 2. Search for the key:

Copy code

bash
    grep "mysql_service_nio_enabled" fe.conf

3. Default Value: If the line is missing, the default is true
. Why change this? In some containerized or heavily firewalled environments (like specific EKS CNI configurations), the Java NIO implementation for the MySQL protocol can occasionally struggle with the initial handshake "communication packet," leading to the

2013 (HY000)

error you are seeing. To test if this is the cause, try disabling it: 1. Add

mysql_service_nio_enabled = false

to your

fe.conf

. 2. Restart the FE. 3. Test the connection from the MCP pod again. Related Parameters You may also want to check these related settings using the same

ADMIN SHOW FRONTEND CONFIG

command: * `mysql_service_io_threads_num`: Number of threads handling MySQL I/O (Default: 4). * `qe_max_connection`: Total allowed connections to the FE (Default: 1024). References • administration/management/FE_configuration.md • deployment/deploy_manually.md References • administration/management/FE_configuration.md • deployment/deploy_manually.md • deployment/environment_configurations.md • faq/Deploy_faq.md References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • unloading/Spark_connector.md

Rumit Jain

08/04/2026, 4:06 PM

@Rocky I am not getting anything:

Copy code

Starrocks > ADMIN SHOW FRONTEND CONFIG LIKE 'mysql_service_nio_enabled';
Empty set (0.00 sec)

Starrocks >

How to set it false?

Rocky

08/04/2026, 4:07 PM

If the command

ADMIN SHOW FRONTEND CONFIG LIKE 'mysql_service_nio_enabled'

returns an Empty set, it means the parameter is using its default value and is not currently overridden in your FE's active configuration registry. Because this is a static configuration item, it cannot be modified using SQL (

ADMIN SET CONFIG

). You must modify the configuration file and restart the FE. How to set it to
false
1. If you are using the StarRocks Operator (Kubernetes) Modify your

StarRocksCluster

YAML file. Find the

starrocksFESpec

section and add the parameter to the

config

block:

Copy code

yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks
spec:
  starrocksFESpec:
    config: |
      # ... other configs ...
      mysql_service_nio_enabled = false

Apply the change:

kubectl apply -f your_cluster.yaml

. The operator will perform a rolling restart of the FE pods. 2. If you are using Helm Update your

values.yaml

file:

Copy code

yaml
starrocksFESpec:
  config: |
    mysql_service_nio_enabled = false

Then run

helm upgrade <release_name> starrocks/kube-starrocks -f values.yaml

. 3. If you are managing
fe.conf
manually (Bare Metal or Custom Docker) 1. Locate your

fe.conf

(usually in the

conf/

directory of your StarRocks installation). 2. Add the following line to the file:

Copy code

properties
    mysql_service_nio_enabled = false

3. Restart the FE process:

Copy code

bash
    ./bin/stop_fe.sh
    ./bin/start_fe.sh --daemon

Why this is necessary The "Lost connection to MySQL server at 'reading initial communication packet'" error often occurs when the Java NIO (Non-blocking I/O) stack in the FE fails to complete the handshake with certain network proxies, load balancers, or specific client drivers (like the one used in the MCP server). By setting

mysql_service_nio_enabled = false

, you force the FE to use Standard Blocking I/O (BIO) for MySQL connections, which is significantly more compatible with complex network environments and is a standard troubleshooting step for this specific error. References • administration/management/FE_configuration.md • deployment/deploy_manually.md References • administration/management/FE_configuration.md • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md • faq/Deploy_faq.md References • administration/management/FE_configuration.md • unloading/Spark_connector.md • administration/management/BE_configuration.md

Rumit Jain

08/04/2026, 4:49 PM

@Rocky Still getting same error in mcp pod:

Copy code

2026-08-04 16:43:01 | WARNING | mcp_server_starrocks.connection_health_checker:_connection_health_checker_loop:77 | Database connection remains unhealthy: Error executing statement 'show databases': 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0

Rocky

08/04/2026, 4:50 PM

Since disabling NIO didn't resolve the issue, the "reading initial communication packet" error means the FE is still closing the connection before the handshake starts. This usually points to a readiness or network security issue. Please check the following three areas: 1. Check FE Readiness and Metadata Status If the FE is still replaying logs or hasn't successfully elected a Leader, it will accept the TCP connection but drop it immediately. * Run this SQL command (from any working client):

Copy code

sql
    SHOW FRONTENDS;

Ensure your target FE has

IsMaster

true

(or is a healthy Follower) and

Join

true

. * Check FE Logs: Look at

fe.log

on the StarRocks FE pod at the exact time the MCP pod logs the error (

16:43:01

). * Do you see messages like

wait catalog to be ready

? * Do you see

replaying edit log

? * Do you see any

Rejected connection from /<MCP_POD_IP>

? 2. Disable DNS Lookups (
skip_name_resolve
) StarRocks (like MySQL) often tries to perform a reverse DNS lookup on the incoming connection's IP address to check permissions. In Kubernetes/EKS, DNS lookups can time out, causing the FE to close the connection before the "initial communication packet" is sent. * Action: Add the following to your

fe.conf

and restart the FE:

Copy code

properties
    # This prevents the FE from trying to resolve the MCP pod's IP to a hostname
    skip_name_resolve = true

3. Verify Connection from the MCP Pod To rule out the MCP server code, test the connection manually from inside the MCP pod: 1. Enter the pod:

kubectl exec -it <mcp-pod-name> -- /bin/bash

2. Try to reach the port:

nc -zv <FE_HOST> 9030

3. If you have a mysql client installed:

mysql -h <FE_HOST> -P 9030 -u <user> -p

If
nc
works but
mysql
fails with the same error, it confirms the FE is actively rejecting the handshake. This is almost always due to: *

skip_name_resolve

being missing (as noted above). * The FE binding to a

priority_network

that doesn't include the MCP pod's IP range. * An intermediate Load Balancer (like an AWS NLB) terminating the connection prematurely. Summary of next steps: 1. Check

fe.log

for any "Rejected connection" or "Catalog not ready" errors. 2. Add

skip_name_resolve = true

fe.conf

and restart. 3. Ensure the user you are using in the MCP config has permission to connect from the MCP pod's IP (e.g.,

GRANT ALL ON _._ TO 'user'@'%'

). References • faq/operation_maintenance_faq.md • faq/Others.md • faq/Sql_faq.md • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/Meta_recovery.md

Open in Slack

Previous Next