@Rocky Does resource group only apply to user queries or ingestion as well? I'm thinking whether there is a need to assign a resource group to an application doing stream load. My line of thought is no because there is a lot of things that happens asynchronously that is not linked to ingestion. Compaction which is CPU heavy is not tied to anything, hence it should use default_wg resource groups. What limitations will apply to my application if I assign it a resource group for stream loading data to my shared nothing primary key table?

In StarRocks, resource groups (also known as Workload Groups) have specific support levels for different types of operations. Based on the documentation (up to version 3.3+), here is how they apply to your scenario: 1. Does it apply to Ingestion? Not for Stream Load. Resource group isolation currently only supports specific ingestion methods: * Supported:

INSERT INTO

(since v2.5) and

Broker Load

(since v3.1). * Not Supported:

Stream Load

,

Routine Load

, and

Schema Change

are currently not governed by resource groups. If you assign a resource group to a user or application performing a Stream Load, the ingestion task itself will not be restricted by the group's

cpu_weight

or

mem_limit

. It will typically run using resources in the global pool or

default_wg

, depending on the internal execution path. 2. Compaction and Resource Groups Your line of thought is correct. Compaction is a background process and is not tied to any specific resource group. * Compaction tasks are managed by the BE's internal background threads. * They are governed by system-level BE configurations (e.g.,

compact_threads

,

max_cumulative_compaction_num_threads_per_disk

) rather than workload groups. * Because compaction for Primary Key tables is CPU and memory-intensive (due to the need to merge delete vectors and maintain the PK index), it can impact the entire BE's performance regardless of how you partition user queries into resource groups. 3. Limitations of Assigning a Resource Group If you assign a resource group to your application for Stream Loading: * For Ingestion: No immediate "limitation" will occur because the Stream Load engine doesn't yet honor the resource group's constraints. The load will continue to use the memory defined by

load_mem_limit

(a BE configuration) rather than the resource group's

mem_limit

. * For Queries: If your application performs

SELECT

queries (e.g., checking status or data verification) using the same user/role, those queries will be limited by the resource group's CPU and memory quotas. * Risk of Over-allocation: Since Stream Load runs "outside" the resource group's accounting, if you heavily restrict other workloads via resource groups but leave Stream Load "uncapped," the Stream Load might still starve those restricted groups of CPU/IO during high bursts. Summary Recommendation * No need to assign a resource group for Stream Load specifically, as it won't provide the isolation you are looking for. * For Primary Key tables, focus on tuning BE background settings for compaction and ensuring you have sufficient memory for the PK index. * Use

load_mem_limit

in the BE configuration if you need to cap the memory used by all ingestion tasks on a node. * If you require ingestion to be governed by resource groups, consider switching to Broker Load or

INSERT INTO ... SELECT

if your architecture allows, as these are fully integrated with the workload management system. References • faq/resource_isolation_faq.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md

@Rocky So it is fine to just let my stream loading application authenticate with it's application-specific credentials for the sole purpose of convention (don't use root). Even though its cpu_weight in resource group assigned is 1, stream load ingestion itself will not be affected. It only ever routinely queries FE node every 10 seconds SHOW FRONTENDS to get the leader IP so that stream loads always only route to the leader IP.

Yes, your reasoning is correct and aligned with the current architecture of StarRocks (up to v3.3+). It is perfectly fine to use application-specific credentials and assign them to a dedicated resource group, even with a low

cpu_weight

, for the following reasons: 1. Stream Load Bypass As of the current version, Stream Load is not governed by resource groups. * When your application performs a Stream Load, it bypasses the CPU and memory constraints defined in the workload group. * These loads are typically handled by a dedicated thread pool on the BE (controlled by

enable_streaming_load_thread_pool

) or the general pool, but they do not participate in the pipeline-based resource group scheduling used for queries and

INSERT INTO

. 2. Metadata Queries (

SHOW FRONTENDS

) Queries like

SHOW FRONTENDS

are FE-local metadata operations. * Resource groups are primarily designed to manage and isolate resources (CPU/Memory/IO) on the Backends (BE). * Since

SHOW FRONTENDS

is handled by the Frontend and does not involve the BE's execution engine or scan threads, it will not be throttled or slowed down by a low

cpu_weight

in a resource group. 3. Impact of

cpu_weight = 1

Assigning a

cpu_weight

of 1 is safe for this scenario because: * No Governance: As long as the application only does Stream Loads and basic metadata checks, it won't hit any resource group limits. * Protection: If your application accidentally runs a heavy

SELECT

query, the resource group will then kick in and ensure that query doesn't impact other production workloads, which serves as a good safety rail. Summary of Limitations If you assign a resource group to this application, the only "limitations" you might encounter are: * Internal Table Queries: If the application later needs to perform complex

SELECT

or

INSERT INTO

operations, those will be restricted to the relative share defined by the weight. * Concurrency: If you set a

concurrency_limit

on that resource group, it will only apply to queries, not to the number of concurrent Stream Loads. Recommendation: Go ahead with the application-specific user and resource group. It’s a best practice for security and auditing (via

fe.audit.log

), and it won't negatively impact your ingestion performance or your routine FE leader discovery. References • administration/management/resource_management/resource_group.md • faq/resource_isolation_faq.md • administration/management/BE_configuration.md References • sql-reference/sql-statements/cluster-management/resource_group/SHOW_RESOURCE_GROUP.md • administration/management/resource_management/resource_group.md • sql-reference/sql-statements/cluster-management/resource_group/SHOW_USAGE_RESOURCE_GROUPS.md

@Rocky Does this mean if there is a sudden burst of stream loads, CPU utilization will take precedence over all other queries (even those with resource groups assigned) and reads will be throttled?

Yes, your intuition is correct. Because Stream Load is not governed by the Resource Group (Workload Group) manager, a sudden burst of stream loads can indeed consume CPU resources "outside the law," potentially starving your queries and throttling reads. Here is the technical breakdown of why this happens and what the limitations are: 1. The "Unbounded" Thread Pool Problem By default (since v3.2), StarRocks uses a dedicated thread pool for Stream Load called

stream_load_io

. To prevent deadlocks, this pool is configured to be effectively unbounded (

INT32_MAX

threads). * Contention: When a burst of Stream Loads occurs, StarRocks can spawn a massive number of threads to handle them. These threads compete for CPU cycles at the Operating System level (pthreads). * The Pipeline Bypass: Queries with resource groups assigned are managed by the StarRocks Pipeline Engine. This engine has a fixed number of worker threads and carefully schedules tasks based on

cpu_weight

. However, the Pipeline Engine has no control over the "external" threads used by Stream Load. 2. "Weight" is Relative, Not Absolute If your queries have a

cpu_weight

of 100 and your Stream Load user has a

cpu_weight

of 1, the weight only matters when both are managed by the Pipeline Engine. * Because Stream Load bypasses this system, it essentially functions as a "background process" that takes whatever it needs from the OS. * If Stream Load threads saturate all physical CPU cores, your Pipeline Engine worker threads (handling your reads) will be throttled by the OS scheduler, regardless of their weight within StarRocks. 3. Primary Key Table Complexity Since you are using Primary Key tables, the impact is amplified. During a Stream Load burst: * The BE must perform PK Index lookups and Delete Vector generation. * This is highly CPU-intensive and happens as part of the ingestion task. * Because this work is tied to the Stream Load process, it is not restricted by resource groups. 4. How to protect your queries (Mitigation) If you are on version 3.3.5 or later, you can use CPU Hard Limits to insulate your reads: *

exclusive_cpu_cores

: Instead of

cpu_weight

, you can assign specific CPU cores to a "Read" resource group. * Isolation: When you use

exclusive_cpu_cores

, StarRocks attempts to pin those cores for that group. Other "Shared" tasks (like Stream Load) are expected to run on the remaining "Shared Cores." * Configuration: You should also ensure the BE configuration

enable_resource_group_cpu_borrowing

is set to

false

if you want to prevent shared tasks (like ingestion) from "borrowing" the idle cycles of your query cores. Summary of Limitations If you assign a resource group to your Stream Load application: 1. Ingestion Throughput: It will not be limited. It will continue to use as much CPU as the OS allows. 2. Read Interference: Stream Load bursts can throttle reads by causing OS-level CPU contention. 3. Memory: While CPU isn't restricted, the total memory for all loads is still capped by the BE configuration

load_mem_limit

. This is your only global "emergency brake" for Stream Load. Recommendation: For mission-critical reads, do not rely on

cpu_weight

alone if you have heavy ingestion. Use

exclusive_cpu_cores

(if on v3.3.5+) or strictly manage your application's concurrency at the source to prevent overwhelming the BE nodes. References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md

@Rocky Is there ever a reason to create separate databases in a single PK table? Versus storing everything in the same database but different tables

For Primary Key (PK) tables in StarRocks, there are several practical reasons to organize your data into separate databases rather than just different tables within a single database. While the underlying storage engine treats a table the same regardless of which database it's in, the administrative and operational boundaries change significantly. Here is why you would choose separate databases: 1. Backup and Restore Parallelism This is the most critical operational reason. StarRocks enforces a limit of one active BACKUP or RESTORE job per database. * The Problem: If you have 100 different tables (each belonging to a different customer) in a single database, and you need to back up "Customer A's" table, you cannot start a backup for "Customer B" until A's job finishes. * The Solution: Putting each tenant or major application in its own database allows you to run backup/restore jobs in parallel across those databases. 2. Storage Quotas (

DATA QUOTA

) StarRocks allows you to set a storage limit at the database level, which is not currently available at the individual table level. * Syntax:

ALTER DATABASE <db_name> SET DATA QUOTA 10T;

* Use Case: If you are building a multi-tenant application and want to ensure one tenant's ingestion burst doesn't fill up the entire cluster's disk, using separate databases is the only way to enforce a hard storage "cap" per tenant. 3. Permission Management Managing security is much simpler at the database level. * Database Level: You can

GRANT ALL ON DATABASE <db_name> TO <role>

. Any new tables created in that database automatically inherit those permissions for that role. * Table Level: If everything is in one database, you have to explicitly grant permissions for every new table created, which is error-prone and harder to audit in a dynamic environment. 4. Metadata and Organization * FE Memory: While the primary memory consumer in the Frontend (FE) is the number of Tablets (not the number of databases), having a clear database hierarchy makes the

SHOW

commands and metadata navigation much cleaner. * Namespace Collisions: Separate databases allow you to have tables with the same name (e.g.,

logs

,

users

) for different environments or tenants without conflict. When should you stay in a SINGLE database? * Cross-table Joins: If your tables are frequently joined together, keeping them in the same database is often more intuitive for developers (though StarRocks supports cross-database joins easily via

db_name.table_name

). * Colocate Join: If you plan to use Colocate Groups to ensure data for multiple tables is physically stored on the same nodes for ultra-fast joins, the tables must be in the same database. Summary Feature | Table in Same DB | Separate Databases Backup/Restore | Sequential (Blocked) | Parallel Storage Quota | Global Only | Per-Database Cap Permissions | Granular/Manual | Simplified/Inherited Colocate Join | Supported | Not Supported Recommendation: If you are designing for multi-tenancy or need to manage backups independently, use separate databases. If you are building a single application where tables are tightly coupled and require Colocate Joins, stay within one database. References • table_design/table_design.md • introduction/Architecture.md • faq/Others.md • data_source/catalog/iceberg/iceberg_catalog.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky Is there a specific line in the StarRocks documentation that says Stream Load is not covered by resource groups? I want to look it up myself

Yes, there is a specific support matrix in the official StarRocks documentation that confirms this. The documentation for Resource Group contains a feature support table (found under the "Roadmap of Resource Group" section). It explicitly shows that Stream Load (along with Routine Load and Schema Change) is not supported for resource group management, even in the latest versions. Support Matrix Excerpt Version | Internal Table | INSERT INTO | Broker Load | ***Routine Load, Stream Load, Schema Change*** | CPU Hard Limit 3.1 & 3.2 | √ | √ | √ | × | × 3.3.5+ | √ | √ | √ | × | √ Why it's not covered The reason for this "×" in the documentation is architectural: * Query Engine vs. Load Engine: Resource groups primarily govern the Pipeline Engine, which handles

SELECT

and

INSERT INTO

tasks. * Thread Isolation: Stream Load is handled by a separate set of threads (often referred to as the "streaming load thread pool" on the BE) that do not currently participate in the pipeline scheduling or the resource group's CPU/memory accounting logic. Key Takeaway for Your Setup Since the documentation explicitly marks Stream Load as unsupported (×) for resource isolation: 1. CPU Weight is Ignored: Your application's

cpu_weight = 1

will only affect metadata queries (like

SHOW FRONTENDS

) or any

SELECT

queries it runs. It will have no effect on the actual ingestion process. 2. Resource Contention: If your Stream Load application starts a massive burst of data, it will consume as much CPU as the hardware allows, potentially starving the "Shared" resource groups used by your reads. To look this up yourself, navigate to the Administration > Management > Resource management > Resource group page in the StarRocks documentation and scroll to the version compatibility table. References • faq/resource_isolation_faq.md • administration/management/resource_management/resource_group.md • faq/loading/Stream_load_faq.md • loading/StreamLoad.md

@Rocky If resource groups dont govern stream load, doesn't that mean computationally heavy queries can take CPU away from ingestion and background tasks?

Technically, yes. If your queries are not restricted by hard limits, they can absolutely starve Stream Load and background tasks like compaction. Because Stream Load is not governed by the Resource Group manager, it exists in a "blind spot" for the StarRocks internal scheduler. Here is how that competition works in practice: 1. The "Invisible" Competitor The Resource Group manager (specifically the Pipeline Engine) only schedules and manages the queries it knows about. It calculates how to distribute CPU cycles among queries based on their

cpu_weight

. * The Problem: Since Stream Load and background tasks (like compaction) run in their own dedicated thread pools (

stream_load_io

,

compact_threads

) outside the Pipeline Engine, the Pipeline Engine doesn't "see" them. * The Result: If you have heavy queries and no hard limits, the Pipeline Engine will attempt to use 100% of the available CPU threads to satisfy those queries. It won't "back off" to make room for ingestion because it doesn't know ingestion is happening. 2. OS-Level "Street Fighting" When both a heavy query and a Stream Load are running, they compete at the Operating System (OS) level: * The StarRocks Pipeline threads and the Stream Load threads both ask the Linux kernel for CPU time. * Because StarRocks hasn't told the OS that one is more important than the other (via something like

nice

values), the OS simply multiplexes between them. * If your query engine is trying to use all 64 cores for a massive join, and your Stream Load is trying to use 10 cores for ingestion, the OS will struggle to balance them, often leading to increased ingestion latency and "RPC timeout" errors for loads. 3. Using Resource Groups as a "Shield" The irony is that while Resource Groups don't protect Stream Load directly, they are the only way to save Stream Load from starvation. * Without Resource Groups: Queries will try to take 100% of the CPU, leaving ingestion to fight for the leftovers. * With Hard Limits (

exclusive_cpu_cores

or

max_cpu_cores

): You can force your heavy queries to stay within, for example, 80% of the CPU. This creates a "reserved" 20% "buffer" that the Pipeline Engine is forbidden from touching. Because Stream Load and compaction are not restricted by resource groups, they can freely use that 20% buffer without any competition from queries. 4. Background Tasks (Compaction) Background tasks like Compaction are even more vulnerable. StarRocks typically limits compaction to a small number of threads (controlled by

compact_threads

, often recommended to be 25% of cores). * If heavy queries saturate the CPU, compaction will slow down. * If compaction slows down, your "version count" (Compaction Score) will rise. * Eventually, if the score gets too high, StarRocks will intentionally throttle or reject new Stream Loads to prevent the system from collapsing. Summary * Queries vs. Ingestion: Heavy queries can take CPU away from ingestion because the ingestion threads are not "reserved" or prioritized by StarRocks. * The Fix: Use CPU Hard Limits on your query resource groups. By capping your "Reads," you naturally leave a "hole" in CPU utilization that "Writes" (Stream Load) and "Maintenance" (Compaction) can fill. References • administration/management/resource_management/resource_group.md • administration/management/compaction.md • faq/resource_isolation_faq.md • best_practices/primarykey_table.md References • loading/loading_introduction/troubleshooting_loading.md