@Rocky what does this metric

starrocks_be_resource_group_inuse_cpu_cores

return ? Is it the count of physical or virtual cores?

The metric

starrocks_be_resource_group_inuse_cpu_cores

returns the estimated number of logical CPU cores (vCPUs) currently in use by a specific resource group on a Backend (BE) node. Key details about this metric: * Logical Cores (vCPUs): StarRocks identifies CPU resources based on the logical processors visible to the operating system. In environments like AWS, GCP, or containerized setups, this corresponds to vCPUs (which include hyper-threaded cores) rather than physical CPU sockets or physical cores. * Estimation Method: This value is an approximation. It is calculated as an average over the time interval between two metric retrievals (typically based on the CPU time consumed by the threads associated with that resource group). * Usage: It is often used alongside the SQL command

SHOW USAGE RESOURCE GROUPS

(which shows

BEInUseCpuCores

) to monitor how much of the allocated CPU capacity a resource group is actually consuming. If you are running in a containerized environment (like Kubernetes), StarRocks typically respects the CPU limits/requests defined for the pod, provided the

num_cores

is correctly detected or configured in

be.conf

. References • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • faq/resource_isolation_faq.md • deployment/plan_cluster.md

@Rocky how to find this virtual cpu core count?

You can find the logical (virtual) CPU core count that StarRocks is using by using any of the following methods: 1. SQL Commands (Easiest) Run the following command in your MySQL client. The

CpuCores

column shows the total number of logical cores detected or configured for each Backend (BE):

sql
SHOW BACKENDS;
-- OR
SHOW PROC '/backends';

Look for the CpuCores column in the output. 2. BE Configuration File Check the

be.conf

file on your Backend nodes for the parameter `num_cores`: * If

num_cores

is set to a positive integer: This is the fixed number of cores StarRocks will use, overriding any auto-detection. * If

num_cores

is set to

0

(default): StarRocks auto-detects the count. On Linux, it reads

/proc/cpuinfo

and also respects cgroup limits (like those set in Kubernetes or Docker). 3. OS Commands Since StarRocks typically uses all logical cores visible to the operating system (unless restricted by cgroups), you can check the count directly on the BE machine:

bash
# Returns the number of available processing units (logical cores)
nproc

# Detailed CPU architecture info, including "CPU(s)" (total logical cores)
lscpu

4. System View You can also view the CPU cores bound to specific execution threads (which helps verify how StarRocks is distributing tasks across those cores) via:

sql
SELECT BE_ID, NAME, BOUND_CPUS FROM information_schema.be_threads;

Note: If you are running StarRocks in a containerized environment (like Kubernetes), the "virtual cpu core count" StarRocks sees will be the lower of the physical logical cores or the CPU limits/quotas defined for the container. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/management/BE_configuration.md • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky I have configured resource group for a user, I can see some stats under starrocks_be_resource_group_running_queries but i dont see any metrics for this resource group in starrocks_fe_query_resource_group_latency

The reason you see BE metrics but not the FE latency metric for your resource group is likely due to one of the following reasons: 1. Lazy Initialization (Query Completion Required) The FE latency metric (

starrocks_fe_query_resource_group_latency

) is lazy-initialized. Unlike some BE metrics that might show up as 0 immediately, FE resource group metrics usually only appear after at least one query has finished in that specific resource group on that specific FE node. * BE metrics like

running_queries

are instantaneous and update as soon as a query starts executing on the BE. * FE latency metrics are calculated upon query completion. If your queries are still running or if none have been executed since the FE started, the metric will not be exported to the

/metrics

endpoint. 2. FE-Specific Metrics FE metrics are local to each FE node. StarRocks does not aggregate these metrics across the cluster. * If you have multiple FEs (e.g., one Leader and several Followers), and your queries are being sent to

FE-A

, but you are checking the

/metrics

endpoint of

FE-B

, you will not see any data for that resource group on

FE-B

. * Action: Ensure you are monitoring the metrics endpoint of the specific FE node that is receiving and coordinating the queries. 3. Check for the Counter Metric To verify if the FE is correctly identifying the resource group, check for the counter metric:

starrocks_fe_query_resource_group

If this metric is also missing, it means the FE hasn't associated any queries with your new resource group yet. This could be due to: * Classifier mismatch: The user or query characteristics don't match the resource group's classifier. * Default Group: The queries might be falling into the

default_wg

group. Summary Checklist 1. Run a simple query (e.g.,

SELECT 1

) as the user assigned to that resource group and ensure it completes. 2. Check all FEs: If using a load balancer, check the

/metrics

endpoint of every FE. 3. Verify labels: Look for the

resource_group

label in the Prometheus output. The latency metric also uses a

type

label (e.g.,

type="mean"

,

type="99_quantile"

). References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md References • administration/management/monitoring/metrics.md • administration/management/resource_management/resource_group.md

@Rocky from the audit log, I could see that no resource group is attached for this user/query. And the

starrocks_fe_query_resource_group

shows only the default_wg

If the audit log shows an empty string or

default_wg

for the

ResourceGroup

field and the FE metrics only report the default group, it confirms that your classifiers are not matching your queries. Here are the most common reasons why this happens and how to fix it: 1. Verify Classifier Definition Check the exact classifier configuration by running:

sql
SHOW RESOURCE GROUPS ALL;

* Case Sensitivity: Ensure the

user

string in the classifier matches the user name exactly (it is usually case-sensitive). * Attribute Matching: If you have multiple attributes in one classifier (e.g.,

user='A', db='B'

), all conditions must be met for the classifier to match. If you want "User A" OR "Database B," you must create two separate classifiers. 2. Check Global/Session Settings If you are on a version older than v3.1, resource groups are not enabled by default. Ensure the following are set to `true`:

sql
-- Check current status
SELECT @@enable_resource_group, @@enable_pipeline_engine;

-- Enable if necessary (Global)
SET GLOBAL enable_resource_group = true;
SET GLOBAL enable_pipeline_engine = true;

_(Note: In v3.1+,

enable_resource_group

is deprecated and enabled by default, but the Pipeline Engine is still required)._ 3. Test with a Manual Override To determine if the issue is with the Classifier or the Resource Group itself, try to force the resource group in your current session:

sql
SET resource_group = 'your_group_name';
SELECT 1; -- Run a test query

* If this works (appears in audit log/metrics): Your Resource Group is fine, but your Classifier logic is incorrect. * If this still falls back to default: The Resource Group itself might be misconfigured (e.g., check if the

type

was set to

normal

during creation). 4. Debug with

EXPLAIN VERBOSE

You don't need to wait for audit logs. You can see which resource group a query would hit by prefixing it with `EXPLAIN VERBOSE`:

sql
EXPLAIN VERBOSE SELECT * FROM your_table LIMIT 1;

Look for the

RESOURCE GROUP

field in the output. If it says

default_wg

or is empty, the classifier is not matching. 5. Why did you see BE stats earlier? It is likely that the "stats" you saw in

starrocks_be_resource_group_running_queries

were actually for the

default_wg

(check the labels on those Prometheus metrics) or for a different group that was active. If the FE doesn't assign a group, the BE will only ever report activities under

default_wg

. Recommendation: Try

SET resource_group = '...'

first. If that works, simplify your classifier to just

(user='your_user')

and verify with

SHOW RESOURCE GROUPS ALL

. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md • administration/management/monitoring/metrics.md

@Rocky Manual Override works and shows the correct resource group, but otherwise it goes to the default one. The classifier has the user, role, query type declared as well. How to better understand the matching logic?

To understand why your classifiers are not matching, you need to look at how StarRocks evaluates them. The most important rule is that *a single classifier is an "AND" condition*—all attributes defined within that classifier must match the query simultaneously. Here is the breakdown of the matching logic and common pitfalls: 1. The "AND" Logic within a Classifier If your classifier is defined as

(user='alice', role='analyst', query_type in ('select'))

, the query will only match if: * The user is exactly

alice

. * The user has the role

analyst

active in the current session. * The query is a

SELECT

(not a

SHOW

,

DESC

, or

SET

command). If you want the group to apply if it's either the user OR the role, you must create two separate classifiers for the same resource group:

sql
ALTER RESOURCE GROUP rg1 ADD (user='alice');
ALTER RESOURCE GROUP rg1 ADD (role='analyst');

2. Role Matching Requirements The

role

condition matches the active roles in the session. * If a user has multiple roles but only one is active (check via

SELECT current_role()

), the classifier will only match if that specific active role is in the classifier. * If no role is explicitly set, StarRocks uses the user's default roles. 3. Query Type Restrictions The

query_type

filter is very specific: * `'select'`: Matches standard

SELECT

queries. * `'insert'`: Matches

INSERT INTO

,

BROKER LOAD

, and

STREAM LOAD

(from v2.5+). * Pitfall: Queries like

SHOW BACKENDS

,

EXPLAIN

, or metadata queries often do not fall under the

select

type. If your classifier requires

query_type='select'

, these administrative queries will fall into the

default_wg

. 4. Degree of Matching (Tie-Breaking) If a query matches multiple classifiers (e.g., one by

user

and another by

role

), StarRocks calculates a "match weight." The classifier with the highest total weight wins: * Database (

db

): +10 (This usually overrides everything else) * User (

user

): +1 * Role (

role

): +1 * Query Type (

query_type

): +1.x * Source IP: +1.x How to Debug the Match 1. Check current session attributes: Run these inside the session that is failing to match:

sql
    SELECT current_user(), current_role(), current_db();

Ensure these match the strings in

SHOW RESOURCE GROUPS ALL;

exactly (including case). 2. Test with a "Wide" Classifier: Temporarily add a classifier with only the

user

to the group:

sql
    ALTER RESOURCE GROUP your_group ADD (user='your_user');

If this starts matching, then either the

role

wasn't active or the

query_type

was excluding your specific SQL commands. 3. Check `fe.audit.log`: Look at the

User

,

Db

, and

Sql

columns. Compare them against the requirements of your classifier. If

PlanCpuCost

or

PlanMemCost

are involved (v3.1.4+), check those values as well. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md