http://starrocks.io logo
Join Slack
Powered by
Hi <@U0A71G31CDV>, for some reason in my shared da...
# questions-and-troubleshooting
u
Hi @Rocky, for some reason in my shared data kubernetes deployment even though I specify requests as 20GB ram per CN, after the compute nodes come up, I only see 16.2GB as MemLimit when I run 
SHOW COMPUTE NODES;
any ideas why?
r
This is expected behavior in StarRocks due to a two-tier memory limit calculation. The 
MemLimit
you see in 
SHOW COMPUTE NODES;
(or 
SHOW BACKENDS;
) is typically calculated as 81% of the memory limit detected by the container (20GB _ 0.9 _ 0.9 = 16.2GB). Why this happens StarRocks applies two default 90% coefficients to determine the memory available for query execution: 1. Process Memory Limit (
mem_limit
): By default, StarRocks sets its process-level memory limit to 90% of the total memory detected in the environment (the cgroup limit in Kubernetes). For your 20GB request, this is 18GB. 2. Query Pool Coefficient: Within that process limit, StarRocks reserves a portion for system overhead and non-query tasks. It applies an internal 90% coefficient to the 
mem_limit
to define the "Query Pool" memory. This is the value reported as 
MemLimit
in the SQL output. The Math: 
20GB (K8s Limit) × 0.9 (mem_limit) × 0.9 (Query Pool) = 16.2GB
How to change it If you want StarRocks to use more of the allocated 20GB, you can adjust the 
mem_limit
in your CN configuration (
cn.conf
). * Increase the Process Limit: You can set 
mem_limit
to a higher percentage (e.g., 
95%
) or an absolute value (e.g., 
19G
).
Copy code
properties
    mem_limit = 95%
* Kubernetes Operator: If you are using the StarRocks Operator, you can add this to the 
config
section of your `starRocksCnSpec`:
Copy code
yaml
    starRocksCnSpec:
      config: |
        mem_limit = 95%
      requests:
        memory: 20Gi
      limits:
        memory: 20Gi
Note: It is generally recommended to keep some headroom (the default 81% is a safe "golden rule") to prevent the Kubernetes OOM Killer from terminating the pod if the process's physical memory usage (RSS) spikes slightly above the tracked memory due to allocator overhead or system libraries. References • administration/management/monitoring/alert.md • administration/management/resource_management/Memory_management.md • faq/Sql_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md
u
Okay but can the query pool limit be changed? @Rocky
r
Yes, you can change the Query Pool limit. The 16.2GB you see is determined by a second configuration parameter called 
query_max_memory_limit_percent
. By default, this is set to 90% of the process memory limit (
mem_limit
). How to change it You can adjust this by adding the following parameter to your CN configuration (
cn.conf
or the 
config
section in your Kubernetes YAML):
Copy code
properties
# Sets the % of the process memory (mem_limit) that the Query Pool can use.
# Default is 90. Increase it to allow queries to use more of the 18GB.
query_max_memory_limit_percent = 95
The Calculation Breakdown In your current setup (20GB RAM requested): 1. Process Limit (
mem_limit
): 
20GB * 90% = 18GB
. This is the total memory the StarRocks process is allowed to use. 2. Query Pool Limit: 
18GB * 90% = 16.2GB
. This is the limit for actual query execution (the 
MemLimit
you see in SQL). If you change 
query_max_memory_limit_percent
to 95, your 
MemLimit
would increase to 17.1GB. Important Notes * Version requirement: This parameter was introduced in v3.1.0. * Restart required: This parameter is not mutable at runtime; you must restart the CN pods for the change to take effect. * Safety: We generally recommend keeping the total effective limit (the 16.2GB) slightly below the container request to allow room for the OS, metadata, and memory allocator overhead, which prevents Kubernetes from killing the pod. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md References • sql-reference/System_variable.md • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • administration/management/resource_management/resource_group.md • administration/management/monitoring/metrics.md
u
Even though I changed this 
query_max_memory_limit_percent
to 95 and then to 100, the limit remained same and didn't increase @Rocky.
r
If the 
MemLimit
shown in 
SHOW COMPUTE NODES;
remains stuck at 16.2GB (exactly 81% of your 20GB) even after changing 
query_max_memory_limit_percent
, it usually points to one of three things: the config isn't actually being applied, the version doesn't support that parameter, or you are hitting a display limit in the heartbeat report. Here is how to troubleshoot and fix it: 1. Verify the Configuration is Active The most common issue in Kubernetes is that the configuration change wasn't actually picked up by the process. * Check via SQL: Run the following to see what the CN actually thinks its config is:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG; -- (Usually for FE, but check BE/CN equivalent)
    -- Better yet, check the individual CN's configuration:
    SELECT * FROM information_schema.be_configs WHERE NAME = 'query_max_memory_limit_percent';
* Check via Pod: Exec into the pod and check the 
cn.conf
or the environment variables:
Copy code
bash
    kubectl exec -it <cn-pod-name> -- cat conf/cn.conf | grep query_max_memory_limit_percent
2. Increase the Base 
mem_limit
 The 16.2GB is calculated as: 
20GB (K8s) _ 90% (mem_limit) _ 90% (query pool)
. If you want to see a higher number, try increasing the process-level limit first. This is often more "reliable" for the display in 
SHOW
commands:
Copy code
properties
# Set this in your cn.conf or operator config
mem_limit = 95%
If you set 
mem_limit = 95%
and keep the query pool at 
90%
, the result should be 
20 _ 0.95 _ 0.9 = 17.1GB
. 3. Check your StarRocks Version The 
query_max_memory_limit_percent
parameter was introduced in v3.1.0. * If you are on v3.0.x or older, the CN will ignore that parameter entirely and stay at the hardcoded 90% of 
mem_limit
. * Check your version with 
SELECT starrocks_version();
. 4. Why the UI might not change In some versions of StarRocks, the 
MemLimit
column in 
SHOW COMPUTE NODES
is calculated by the Frontend (FE) during the heartbeat based on the 
process_mem_limit
reported by the CN, applying a fixed 0.9 multiplier for display purposes. * The actual usable memory might have increased even if the 
SHOW
command doesn't reflect it. * To verify the "real" internal limit, you can check the CN's metrics page: 
http://<cn_ip>:<http_port>/metrics
and look for 
starrocks_be_query_pool_mem_limit_bytes
. Recommendation If your goal is to maximize memory for queries, set your config to:
Copy code
properties
mem_limit = 95%
query_max_memory_limit_percent = 95
Warning: Do not set both to 100%. Kubernetes will kill the pod (OOM) because StarRocks' memory tracking doesn't account for the memory used by the OS, thread stacks, or C-library overhead. Keeping the total around 90% (the default 18GB process limit) is the safest practice. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • faq/Sql_faq.md • knowledge/trouble_shooting/fe_memory_problems.md References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • sql-reference/System_variable.md • knowledge/usage/best-practice.md • administration/management/monitoring/metrics.md
u
Hi @Rocky, here are some re-verifications I did. 1. I'm running 4.0.14-7c18dd9 2. MemLimit from 
SHOW COMPUTE NODES
is 3.6GB. I also confirmed from an error message running an expensive query
Copy code
ERROR 1064 (HY000) at line 1: Memory of process exceed limit. try consume:14680064 Backend: decoupled-starrockscluster-cn-1.decoupled-starrockscluster-cn-search.starrocks-dec.svc.cluster.local, Used: 3752361032, Limit: 3865470566. Mem usage has exceed the limit of BE: BE:10001
3. I checked from cn.conf in pod
Copy code
mem_limit = 100%
query_max_memory_limit_percent = 100
4. I also queried 
<http://information_schema.be|information_schema.be>_config
Copy code
mysql> select * from information_schema.be_configs where name like '%query_max_memory_limit_percent%' or name = 'mem_limit';;
+-------+--------------------------------+-------+--------+---------+---------+
| BE_ID | NAME                           | VALUE | TYPE   | DEFAULT | MUTABLE |
+-------+--------------------------------+-------+--------+---------+---------+
| 10002 | mem_limit                      | 100%  | string | 90%     |       0 |
| 10002 | query_max_memory_limit_percent | 100   | int32  | 90      |       0 |
| 10001 | mem_limit                      | 100%  | string | 90%     |       0 |
| 10001 | query_max_memory_limit_percent | 100   | int32  | 90      |       0 |
+-------+--------------------------------+-------+--------+---------+---------+
r
The fact that your 
MemLimit
dropped from 16.2GB to 3.6GB after you changed the configuration suggests that the StarRocks process is now detecting a total system memory of only 4GB (or 3.6GB) instead of the 20GB you requested in Kubernetes. Since $3.6 \text{ GiB} = 4 \text{ GiB} \times 90\%$, it appears that despite your configuration showing 
100%
, there is still a 90% factor being applied somewhere, and the "base" memory is being seen as 4GB. 1. Check for Kubernetes 
limits
vs 
requests
 StarRocks' memory detection logic prioritized Cgroup Limits over K8s Requests. * The Issue: If you specified 
requests: 20Gi
but accidentally set 
limits: 4Gi
(or didn't specify a limit and the namespace has a default limit of 4Gi), StarRocks will see 4GB as the total available memory. * Verification: Run 
kubectl describe pod <cn-pod-name>
and look specifically for the 
Limits
section. 2. Verify Cgroup detection inside the Pod Exec into one of the CN pods and check what the Linux kernel is telling the process about its memory limit:
Copy code
bash
# For Cgroup v1:
cat /sys/fs/cgroup/memory/memory.limit_in_bytes

# For Cgroup v2:
cat /sys/fs/cgroup/memory.max
If these commands return a value close to 4,294,967,296 (4GB), then the issue is in your Kubernetes Pod/Deployment specification, not the StarRocks configuration. 3. The 
100%
Configuration Parsing In some versions of StarRocks, setting 
mem_limit = 100%
can behave unexpectedly if the system cannot reliably determine the "total" memory (it might fall back to a safe default). * Try an Absolute Value: Instead of using percentages, set the limit to a fixed physical value in your `cn.conf`:
Copy code
properties
    mem_limit = 18G
    query_max_memory_limit_percent = 95
Using 
18G
(for a 20GB request) is safer than 
100%
because it bypasses the auto-detection logic that seems to be failing in your environment. 4. Shared Data (CN) Specifics in v4.x You mentioned you are on version 4.0.14. This is a very recent (potentially internal or vendor-specific) version. * In newer versions, StarRocks has introduced more granular resource management. Check if there is a Resource Group limit affecting you. _ Run: 
SELECT _ FROM information_schema.resource_groups;
* If the 
default_wg
has a 
mem_limit
set, it will override the global process settings for your queries. Summary Recommendation 1. Check K8s Limits: Ensure 
limits.memory
is set to 
20Gi
in your YAML. 2. Hardcode the Limit: Change 
mem_limit = 100%
to 
mem_limit = 18G
to force the process to recognize the memory you've allocated. 3. Restart: Ensure the pods are fully restarted after the change. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • administration/management/resource_management/resource_group.md References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • administration/management/resource_management/resource_group.md • sql-reference/System_variable.md • faq/Sql_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md
u
Hi @Rocky, I also tried this configuration and got mem limit as 2.7. It is seeming like there no way to control the internal 90% limit
Copy code
mysql> select * from information_schema.be_configs where name like '%query_max_memory_limit_percent%' or name = 'mem_limit';;
+-------+--------------------------------+-------+--------+---------+---------+
| BE_ID | NAME                           | VALUE | TYPE   | DEFAULT | MUTABLE |
+-------+--------------------------------+-------+--------+---------+---------+
| 10001 | mem_limit                      | 3G    | string | 90%     |       0 |
| 10001 | query_max_memory_limit_percent | 95    | int32  | 90      |       0 |
| 10002 | mem_limit                      | 3G    | string | 90%     |       0 |
| 10002 | query_max_memory_limit_percent | 95    | int32  | 90      |       0 |
+-------+--------------------------------+-------+--------+---------+---------+
r
The behavior you are seeing—where the 
MemLimit
stays at 90% of your configured 
mem_limit
despite changing `query_max_memory_limit_percent`—is likely due to how StarRocks version 4.x (and 3.x) calculates and displays limits when Resource Groups are enabled. Here is why your limit is not changing and how to fix it: 1. The "Hidden" Multiplier: Resource Groups In newer versions of StarRocks, queries do not just use the "Query Pool"; they run within a Resource Group. By default, all queries belong to the 
default_wg
(default resource group). * Even if you set the total Query Pool to 100% of the process memory, the 
default_wg
itself has a 
mem_limit
parameter (default is often 0.9 or 90%). * The 
MemLimit
you see in 
SHOW COMPUTE NODES
or in error messages often reflects the limit of the resource group the query is running in. How to fix it: Check your resource group configuration and increase the limit for the default group:
Copy code
sql
-- Check the current limit
SELECT * FROM information_schema.resource_groups;

-- Increase the memory limit for the default resource group to 100% 
-- (This allows it to use the entire Query Pool)
ALTER RESOURCE GROUP default_wg WITH ('mem_limit' = '100%');
2. Why did your memory drop from 16.2GB to 3.6GB? This is the most critical issue. The math shows that StarRocks now thinks your node only has 4GB of total memory, even though you requested 20GB. * Previous: $20\text{GB} \times 0.9 \text{ (mem\_limit)} \times 0.9 \text{ (query pool)} = \mathbf{16.2\text{GB}}$. * Current: $4\text{GB} \times 0.9 \text{ (mem\_limit)} \times 0.9 \text{ (query pool)} = \mathbf{3.24\text{--}3.6\text{GB}}$. StarRocks determines total memory by looking at the Cgroup limits in Kubernetes. If you have a 
limits: memory: 4Gi
in your Kubernetes YAML (even if 
requests
is 20Gi), StarRocks will cap itself at 4GB. Action: 1. Run 
kubectl describe pod <cn-pod-name>
and verify the 
Limits
section. 2. If the limit is indeed 20Gi, check the Cgroup file inside the pod:
Copy code
bash
    kubectl exec -it <cn-pod-name> -- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
If this number is around 
4,294,967,296
, Kubernetes is restricting the pod to 4GB regardless of your request. 3. The 90% "Safety Margin" StarRocks has a hard-coded safety margin for the Process Limit. When you set 
mem_limit = 100%
, the system often treats the "Hard Limit" (the point where it starts killing queries to save the process) as 90% of that value to prevent a kernel-level OOM (which would crash the whole pod). * If you set 
mem_limit = 3G
, the effective limit is 2.7G. * To get a real 18GB limit, you should set 
mem_limit = 20G
(assuming your pod truly has 20GB+ available). Summary of Recommended Config To get the maximum possible memory for your queries on a 20GB node: 1. K8s YAML: Ensure 
limits.memory
is set to 
20Gi
. 2. cn.conf:
Copy code
properties
    # Use an absolute value to bypass detection issues
    mem_limit = 19G
    query_max_memory_limit_percent = 100
3. SQL:
Copy code
sql
    ALTER RESOURCE GROUP default_wg WITH ('mem_limit' = '100%');
References • administration/management/BE_configuration.md • administration/management/resource_management/resource_group.md • administration/management/resource_management/Memory_management.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md References • faq/resource_isolation_faq.md
u
Hi @Rocky, Does StarRocks natively support any of the below features related to Data Masking and Redaction. o Provides options for masking, tokenization, and encryption of sensitive data. o Supports format-preserving encryption for structured data. o Dynamically redacts sensitive information from logs, reports, and responses. The reports are parametrised and are generated on demand. The responses include any output either through API or frontend.
r
StarRocks provides several native features and built-in integrations to handle data masking, encryption, and redaction. 1. Masking, Tokenization, and Encryption * Encryption (In-Transit & At-Rest): * Transparent Data Encryption (TDE): Supports encrypting data at the storage level (segment files, etc.) for both local and cloud storage (v3.3+). * Cryptographic Functions: Provides built-in SQL functions like 
aes_encrypt()
and 
aes_decrypt()
supporting multiple modes (ECB, CBC, GCM, etc.) to encrypt/decrypt specific columns. * Masking: * Apache Ranger Integration: StarRocks natively integrates with Apache Ranger to provide Dynamic Data Masking. You can define policies in Ranger (e.g., 
Partial Masking
, 
Hash
, 
Nullify
, 
Show Last 4
) which StarRocks enforces automatically when users query the data. * Tokenization: * StarRocks does not have a "Tokenization-as-a-Service" module. Tokenization is typically handled via external integrations or by using Ranger's hashing/masking policies to provide a tokenized representation of the data. 2. Format-Preserving Encryption (FPE) * Native Support: StarRocks does not currently have a native FPE function (one that returns a ciphertext with the same format/length as the plaintext). * Alternative: Standard 
aes_encrypt
returns `VARBINARY`/`VARCHAR` (Base64). If FPE is a hard requirement, users typically implement this via Java UDFs (User Defined Functions) to call external FPE libraries. 3. Dynamic Redaction (Logs, Reports, and Responses) * Log Redaction: * `enable_sql_desensitize_in_log`: When enabled in FE configuration, StarRocks automatically redacts sensitive literals (like passwords or sensitive data in 
WHERE
clauses) from audit logs, internal logs, and query-detail records. * Response Redaction (API/Frontend): * Because StarRocks enforces Ranger masking policies at the execution layer, any response returned via the MySQL protocol, HTTP API, or external catalogs will contain redacted/masked data based on the user's permissions. * Row-Level Security (RLS): Through Ranger or native Row-Level Filter policies, you can dynamically restrict which rows a user can see in their "reports" or "responses." * Parametrized Reports: As a database engine, StarRocks facilitates this by ensuring that regardless of how the query is parametrized by your BI tool or API, the engine applies the masking/redaction logic before the data leaves the cluster. Summary Table Feature | Supported? | Implementation Method Data Masking | Yes | Native integration with Apache Ranger Encryption | Yes | 
aes_encrypt
functions and Storage TDE Tokenization | Partial | Handled via Ranger Hashing or External UDFs Format-Preserving Encryption | No | Requires custom Java UDFs Log Redaction | Yes | 
enable_sql_desensitize_in_log
config Response Redaction | Yes | Dynamic Masking (Ranger) / Row-Level Security For high-security environments, the recommended path is integrating StarRocks with Apache Ranger to manage these policies centrally. References • administration/management/BE_configuration.md • sql-reference/sql-functions/crytographic-functions/aes_encrypt.md • administration/management/FE_configuration.md • table_design/indexes/inverted_index.md • sql-reference/sql-functions/utility-functions/encode_sort_key.md
Open in Slack
PreviousNext
Powered by