http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> what can I adjust with the starrock...
# questions-and-troubleshooting
c
@Rocky what can I adjust with the starrocks cluster if im having this kind of explain output in a query run from locust having concurrent users of 100?
Copy code
[0mSummary[0m
    [0mQueryId: 019e068e-602b-7504-8dd7-a9e1ab4e270d[0m
    [0mVersion: 4.0.9-f647589[0m
    [0mState: Error[0m
    [0mTotalTime: 37s298ms[0m
        [0mExecutionTime: 36s596ms [Scan: 225.440ms (0.62%), Network: 1s14ms (2.77%), ResultDeliverTime: 0ns (0.00%), ScheduleTime: 36s592ms (99.99%)][0m
        [0mCollectProfileTime: 421ms[0m
        [0mFrontendProfileMergeTime: 11.641ms[0m
    [0mQueryPeakMemoryUsage: ?, QueryAllocatedMemoryUsage: 1.170 GB[0m
    [0mTop Most Time-consuming Nodes:[0m
        [1m[38;2;250;128;114m1. EXCHANGE (id=32) [SHUFFLE]: 342.902ms (17.68%)[0m
        [1m[38;2;250;128;114m2. EXCHANGE (id=8) [SHUFFLE]: 297.292ms (15.33%)[0m
        [0m3. HASH_JOIN (id=12) [BUCKET_SHUFFLE, RIGHT OUTER JOIN]: 266.379ms (13.74%)[0m
        [0m4. HASH_JOIN (id=9) [BUCKET_SHUFFLE, RIGHT OUTER JOIN]: 256.054ms (13.21%)[0m
        [0m5. EXCHANGE (id=14) [SHUFFLE]: 212.701ms (10.97%)[0m
        [0m6. EXCHANGE (id=22) [SHUFFLE]: 110.227ms (5.68%)[0m
        [0m7. OLAP_SCAN (id=2) : 108.385ms (5.59%)[0m
        [0m8. EXCHANGE (id=11) [SHUFFLE]: 101.917ms (5.26%)[0m
        [0m9. OLAP_SCAN (id=0) : 98.742ms (5.09%)[0m
        [0m10. OLAP_SCAN (id=1) : 59.208ms (3.05%)[0m
    [0mTop Most Memory-consuming Nodes:[0m
    [0mNonDefaultVariables:[0m
        [0mautocommit: true -> false[0m
        [0menable_adaptive_sink_dop: false -> true[0m
        [0menable_profile: false -> true[0m
r
Based on the profile you provided, the most critical metric is 
ScheduleTime: 36s592ms
, which accounts for 99.99% of the total execution time. This indicates that your StarRocks cluster is suffering from a concurrency bottleneck. The query itself is actually very fast (only a few hundred milliseconds for scan and join), but it is spending nearly 37 seconds waiting in the pipeline execution queue for a worker thread to become available. To improve performance for 100 concurrent users, you should focus on reducing the resource footprint of individual queries to allow more of them to run simultaneously. 1. Adjust 
pipeline_dop
(Primary Fix) By default, StarRocks sets 
pipeline_dop
to half the number of CPU cores. For a high-concurrency workload (100 users), having each query claim multiple threads causes massive CPU context switching and scheduling delays. * Adjustment: Lower the parallelism per query to free up threads for more concurrent users. * Action: Run the following command (can be set globally or per session):
Copy code
sql
    SET GLOBAL pipeline_dop = 1;
    -- Or try 2 or 4 depending on your BE core count, but 1 is best for very high concurrency.
2. Configure Query Queues If your cluster is being overwhelmed, you should enable the Query Queue feature. Instead of allowing all 100 queries to fight for threads and memory at once (causing the 36s "ScheduleTime" delay), StarRocks can queue the queries and process them in an orderly fashion. * Action: Set the concurrency limit for the query queue:
Copy code
sql
    SET GLOBAL query_queue_concurrency_limit = 50; -- Adjust based on cluster capacity
    SET GLOBAL enable_query_queue = true;
This will ensure that only 50 queries run at once, while the others wait in a managed queue, preventing the system from entering a "thrashing" state. 3. Check Resource Group Settings If you are using Resource Groups, ensure that the 
concurrency_limit
and 
cpu_core_limit
are appropriately set for the group your Locust users are hitting. If the limit is too low, queries will spend time in 
Pending
state; if too high, they will saturate the BE and increase 
ScheduleTime
. 4. BE Configuration (Advanced) If your BE nodes have many CPU cores but are still showing high schedule times, check the following BE configuration in `be.conf`: * 
pipeline_exec_thread_pool_thread_num
: This defaults to the number of logical CPU cores. If your BE is under-utilized but 
ScheduleTime
is high, ensure this hasn't been manually restricted to a very low value. (Note: Usually, the default is fine, and adjusting 
pipeline_dop
is the correct lever). 5. Why is the State "Error"? Your output says 
State: Error
. Given the 
TotalTime
of ~37s, it is likely that: * The Locust client timed out and closed the connection before StarRocks finished. * The StarRocks 
query_timeout
 was reached (though the default is usually 300s). * The Query Queue rejected the query because the queue was full. Summary of Recommendations: 1. 
SET GLOBAL pipeline_dop = 1;
 (Most impactful) 2. Enable and tune Query Queues to manage the 100-user load. 3. Check if the BE CPU is hitting 100%—if it is, you may simply need more BE nodes to handle 100 concurrent users for this specific query complexity. References • administration/management/BE_configuration.md
c
@Rocky right now we can only use 1 2vCPU CN node
r
With only one 2vCPU CN node and 100 concurrent users, your cluster is severely "over-subscribed." In StarRocks, the default execution behavior is optimized for high-performance hardware (e.g., 16+ cores). On a 2vCPU machine, the pipeline engine is trying to juggle 100 queries with only 2 physical execution threads, leading to the 36.5s Schedule Time (99.99%) you see. To make this work on such limited hardware, you must force StarRocks into a "low-concurrency/high-queue" mode. 1. Reduce Parallelism per Query (Crucial) By default, StarRocks may try to use multiple "drivers" (parallel tasks) per query. On a 2-core machine, this causes massive context switching. * Action: Force every query to use only 1 thread.
Copy code
sql
    SET GLOBAL pipeline_dop = 1;
2. Enable Query Queuing With 100 users, if you let all 100 queries enter the execution engine at once, the 2 CPUs will spend more time switching between tasks than actually processing data. You need to limit the number of active queries to 2–4 and make the other 96–98 wait in a queue. * Action:
Copy code
sql
    SET GLOBAL enable_query_queue = true;
    SET GLOBAL query_queue_concurrency_limit = 2; -- Match your vCPU count
    SET GLOBAL query_queue_max_queued_queries = 200; -- Allow Locust to queue up
3. Manage Memory to Prevent OOM Your profile shows one query uses 1.17 GB. If a 2vCPU node typically has 4GB or 8GB of RAM, running even 5-10 of these queries simultaneously will crash the node or trigger heavy Linux swapping. * Action: Limit memory per query so the node stays stable.
Copy code
sql
    SET GLOBAL query_mem_limit = 2147483648; -- 2GB limit per query (adjust based on your RAM)
4. Optimize CN Configuration (
cn.conf
) Since you are using a CN (Compute Node), check your 
cn.conf
for the following. On a 2-core machine, we want to ensure the thread pools are not oversized. * 
pipeline_exec_thread_pool_thread_num
: This should be set to 2 (matching your vCPU). If it's higher, the kernel will waste time context switching. * 
mem_limit
: Ensure this is set to a safe value (e.g., 
80%
) to prevent the CN process from being killed by the OS OOM killer. 5. Why is the time so high? The 36s Schedule Time is exactly what happens when the 2 available CPU threads are "pinned" by other queries, and your specific query is sitting in a software queue inside StarRocks waiting for its turn. Recommendation Summary A 2vCPU node is generally recommended only for functional testing, not for 100 concurrent users. StarRocks recommends at least 8–16 cores per BE/CN for production-level concurrency. If you cannot increase the CPU count, the settings above will stop the "Error" states (likely caused by timeouts or resource exhaustion), but your latency will remain high because 100 users must "take turns" using only 2 CPU cores. To lower the 36s delay, you will eventually need to scale the CN node to more vCPUs or add more CN nodes to the cluster. References • deployment/environment_configurations.md • administration/management/resource_management/Memory_management.md • faq/Others.md • loading/loading_introduction/loading_considerations.md • deployment/plan_cluster.md • knowledge/usage/load/shared_data_batch_ingestion.md
c
@Rocky StarRocks 4.0.9 shared-data, single FE (1.8 vCPU / 7 GB pod, Xmx5500m, G1GC) + single CN (1.8 vCPU / 14 GB), shared-data S3-backed, run on AWS EKS. Workload: BI-style analytical query (4-table join with 2 RIGHT OUTER JOINs, big-tenant location IN list, partition pruning verified working). Single-shot uncontended ~500 ms, peak healthy ~7 q/s sustained. Observations under load test (Locust, 60s, ramp 5→100 users): 1. Throughput plateaus at ~7 q/s past 25 concurrent users (admission`query_queue_concurrency_limit=8`). 2. At 50+ concurrent users we get ~5–8% failures, all the same error: planner exceeded`new_planner_optimize_timeout`(3000 ms) in`memo`or`logical`phase, with the message including "FE Full GC". 3. FE JVM metrics endpoint says: zero full GCs lifetime; old-gen used 1 GB / peak 3.2 GB; heap committed 5.3 GB / used 1.9 GB; threads 267 / peak 1440. 4. FE pod RSS is 6.5 GB on a 7 GB cgroup limit — ~470 MB headroom. 5. CN pod is 7.3 GB / 14 GB and 1% idle CPU — clearly not the bottleneck. Current planned tunings (haven't applied yet): • `thrift_server_max_worker_threads = 256`in fe.conf • JVM:`-Xms4500m -Xmx4500m`,
-XX:MaxGCPauseMillis=100
,
-XX:InitiatingHeapOccupancyPercent=40
,
-XX:G1HeapRegionSize=16m
• Runtime via SET GLOBAL:`query_queue_concurrency_limit=8`,
new_planner_optimize_timeout=10000
Questions: 1. The "FE Full GC" wording in the planner timeout error is misleading given metrics show zero full GCs — what's actually triggering it on 4.0.9? 2. For a small-FE deployment like this, is shrinking Xmx (5500m → 4500m) to grow off-heap headroom the right call, or is the underlying cause different? 3. Is`thrift_server_max_worker_threads = 256`a safe cap given peak observed was 1440? Will requests get rejected under burst? 4. Are there tunings specific to 4.0.x shared-data / single-FE preprod that we're missing? 5. Is`new_planner_optimize_timeout`the right knob to raise, or is the planner taking 3s in memo phase symptomatic of something else (e.g. statistics, cache warmup, MV rewrite logic)?
r
This is a classic "starvation" scenario common in small Kubernetes deployments. With 1.8 vCPU and a 7 GB pod, you are running StarRocks FE at its absolute minimum viable specs while pushing a production-level concurrency (100 users). Here is the breakdown of your observations and specific answers to your questions: 1. The "FE Full GC" Mystery The "FE Full GC" message in the planner error is often a heuristic, not a direct report from the JVM's Event Log. * What's actually happening: StarRocks monitors the 
GarbageCollectorMXBean
. If a planning task times out (reaches 3000ms), it checks if GC cumulative time increased significantly during that window. * The Culprit: Since your RSS (6.5 GB) is nearly at your Cgroup Limit (7 GB), the Linux kernel is likely performing Node-level memory reclamation or throlling the process. When the Cgroup is that tight, the JVM's memory allocation requests can stall, or the OS may pause the process to reclaim file cache pages. StarRocks sees a "pause," checks the GC metrics, and assumes a Full GC occurred because the clock jumped while it was "frozen." * CPU Starvation: 1.8 vCPU shared across 267+ active threads (and 100 users) means your threads are spending more time in the "Ready" state waiting for a CPU slice than actually executing. This latency is added to the "Planning Time," triggering the 3s timeout. 2. Xmx vs. Pod Memory (The "470 MB Headroom" Problem) Yes, shrinking Xmx to 4500m (or even 4096m) is the right call. * StarRocks FE has significant off-heap overhead: Thread stacks (267 threads @ 1MB each = 267MB), Metaspace, Code Cache, and internal Direct Memory buffers. * In K8s, if 
RSS
approaches 
Limit
, the 
OOMKiller
is the final step, but "Invisible Throttling" happens much earlier. * Recommendation: Use 
-Xms4096m -Xmx4096m
. This gives the OS and off-heap ~3GB of breathing room. 3. Thrift Worker Threads Setting it to 256 is likely too low for 100 concurrent users. * Each client connection and several internal FE-to-CN heartbeats consume these threads. If you cap this at 256, and 100 users are active + internal housekeeping + metadata operations, you will see 
Connection Refused
or 
Socket Timeout
at the Locust/Client level. * Why was it 1440? High thread counts usually indicate that requests are "piling up" because the backend (the Planner or the CN) is slow. The threads aren't the cause of the memory usage; they are a symptom of the bottleneck. * Action: Leave it at the default or set it to 1024. To prevent memory bloat, focus on the Query Queue instead of capping threads. 4. Why the Planner times out (The "IN" List & CPU) The 
memo
phase is the cost-based search. * Large IN lists: These are transformed into 
OR
predicates or 
InPredicate
objects which explode the search space for the CBO. * The Math: You have 1.8 vCPU. If 8 queries (
query_queue_concurrency_limit=8
) are planning simultaneously, each query is effectively getting 0.22 vCPU. * A complex 4-table join with a massive 
IN
list simply cannot finish the search space exploration in 3 seconds with 0.22 vCPU. 5. Recommended Adjustments Frontend (fe.conf / JVM) * Xmx: Change to 
-Xmx4096m -Xms4096m
. * G1GC: Your proposed G1 settings are excellent for low-latency planning. * Planner Timeout: Increase 
new_planner_optimize_timeout = 10000
. This is safe; it just allows the CPU-starved planner more time to finish. * Statistics: Ensure you have run 
ANALYZE TABLE
on your 4 tables. If stats are missing, the planner works significantly harder (and longer) to estimate costs. Shared-Data Specific (4.0.x) * Metadata Cache: In shared-data, FE frequently talks to the S3-backed StarMgr. Ensure your FE pod has good network throughput.
* Single FE Risk: In 4.0.x shared-data, a single FE handles all 
starmgr
(metadata) and 
lake_compaction
scheduling. With only 1.8 vCPU, the compaction scheduler might be stealing cycles from the Query Planner. * Tuning:
Copy code
sql
    SET GLOBAL query_queue_concurrency_limit = 4; -- Lower this to give each query more CPU share
Summary Recommendation 1. Increase FE vCPU to 4 or 8 if possible. StarRocks is designed for "wide" hardware; 1.8 vCPU is extremely tight for 100 concurrent users. 2. Lower 
query_queue_concurrency_limit
to 4. Counter-intuitively, allowing fewer queries to run at once will make them finish faster because they won't fight for the 1.8 vCPU, preventing the 3s planner timeout. 3. Fix the Memory Ratio: Shrink Xmx to 4GB to stop the Cgroup/OS from throttling your process. References • knowledge/trouble_shooting/fe_memory_problems.md • deployment/plan_cluster.md • faq/Others.md • deployment/shared_data/feature-support-shared-data.md • knowledge/admin_and_operation/shared_data_compaction.md
c
does
Copy code
SET GLOBAL query_queue_concurrency_limit = 4;
affect streaming data coming from kafka via starrocks connector?
@Rocky does
Copy code
SET GLOBAL query_queue_concurrency_limit = 4;
affect streaming data coming from kafka via starrocks connector?
r
No, setting 
query_queue_concurrency_limit = 4
will not affect streaming data coming from Kafka via the StarRocks Kafka Connector or Routine Load. Here is the technical breakdown of why: 1. Different Internal Managers In StarRocks, "Queries" and "Loads" are often managed by separate subsystems to ensure that heavy analytical queries don't accidentally block background data ingestion: * Query Queue: Governed by the 
QueryMgr
on the FE. It specifically targets SQL statements that go through the full planning and execution lifecycle (primarily 
SELECT
and 
INSERT INTO
). * Stream Load (Kafka Connector): The Kafka Connector uses the Stream Load (HTTP PUT) interface. Stream Load is handled by the BE's 
StreamLoadMgr
and generally bypasses the FE's query queue logic. * Routine Load: Managed by the FE's 
RoutineLoadManager
. It is a persistent background job that schedules its own "Load Tasks." 2. Selective Enablement The 
query_queue_concurrency_limit
only takes effect if you explicitly enable it for a specific type of traffic using these global variables: * 
enable_query_queue_select = true
(Queues SELECT queries) * 
enable_query_queue_load = true
 (Queues SQL-based 
INSERT INTO
and 
Broker Load
) Even if you set 
enable_query_queue_load = true
, it is primarily designed for SQL-driven loads (like 
INSERT INTO SELECT
or 
Broker Load
). The Stream Load path used by the Kafka Connector is optimized for high-frequency ingestion and is not subject to these query-level concurrency limits. 3. Isolation of Ingestion Resources StarRocks specifically isolates streaming ingestion to prevent interference: * Thread Pool Isolation: Since StarRocks v3.2, there is a BE configuration 
enable_streaming_load_thread_pool
(default: 
true
). This ensures that Stream Load "scanners" run in a dedicated, effectively unbounded thread pool, keeping them separate from the thread pools used by your BI queries. * Routine Load Concurrency: For Routine Load, the concurrency is controlled by its own job-specific properties (e.g., 
desired_concurrent_number
) and the FE dynamic parameter 
max_routine_load_task_concurrent_num
, not the global query queue. Summary of Impact * BI Queries (Locust/Users): Will be strictly queued (only 4 at a time), which is exactly what you want for your 1.8 vCPU environment to prevent CPU thrashing and "FE Full GC" errors. * Kafka Ingestion: Will continue to run normally in the background. It will still compete for the same physical CPU and Memory on your CN node, but it will not be forced to wait in the software queue created by the 
query_queue_concurrency_limit
. Tip: If you find that Kafka ingestion is starving your BI queries for CPU on that tiny 1.8 vCPU node, you should look into Resource Groups rather than the Query Queue to strictly carve out a percentage of CPU specifically for ingestion vs. Loading. References • sql-reference/System_variable.md • administration/management/BE_configuration.md • administration/management/resource_management/query_queues.md • sql-reference/sql-statements/loading_unloading/BROKER_LOAD.md • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md
c
@Rocky instead of setting this every fe pod restart, is it possible to add in the cluster setup?
Copy code
SET GLOBAL query_queue_concurrency_limit = 4;
SET GLOBAL query_queue_max_queued_queries = 200;
SET GLOBAL query_queue_pending_timeout_second = 60;
SET GLOBAL new_planner_optimize_timeout = 10000;
r
In StarRocks, there is a technical distinction between FE Configuration Parameters (found in 
fe.conf
) and System Variables (set via 
SET GLOBAL
). The variables you listed (
query_queue_concurrency_limit
, etc.) are System Variables. By design, they are stored in the FE's internal metadata (BDBJE) rather than a flat text file. 1. Are they already persistent? If your FE pod has a Persistent Volume (PV) mounted to 
/opt/starrocks/fe/meta
, you only need to run 
SET GLOBAL
once. These settings are written to the metadata and will survive pod restarts, upgrades, and crashes. If you find yourself running these every restart, it likely means: * Your FE pod is stateless (using 
emptyDir
or no volume), meaning it loses all data/settings on restart. This is highly discouraged for production. * You are performing a "clean" redeploy every time (e.g., 
helm delete
followed by 
helm install
). 2. How to include them in the Cluster Setup (Kubernetes) If you are using GitOps (ArgoCD, Terraform) and want these settings to be part of your declarative configuration, there are three common ways to do it in K8s: Option A: Kubernetes "Init Job" (Recommended) Create a simple Kubernetes 
Job
that runs a container with a MySQL client. Configure it to wait for the FE to be "Ready" and then execute your SQL. This is the standard "Day 0" automation path.
Copy code
yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: starrocks-init-config
spec:
  template:
    spec:
      containers:
      - name: mysql-client
        image: mysql:8.0
        command: ["/bin/sh", "-c"]
        args:
        - |
          mysql -h starrocks-fe-service -P 9030 -u root -e "
          SET GLOBAL query_queue_concurrency_limit = 4;
          SET GLOBAL query_queue_max_queued_queries = 200;
          SET GLOBAL query_queue_pending_timeout_second = 60;
          SET GLOBAL new_planner_optimize_timeout = 10000;"
      restartPolicy: OnFailure
Option B: PostStart Hook (Helm/Operator) If you are using the StarRocks Helm chart or Operator, you can add a 
postStart
hook to the FE spec. This runs immediately after the container starts.
Copy code
yaml
# In your StarRocksCluster CRD or Helm Values
fe:
  lifecycle:
    postStart:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            sleep 30; # Wait for FE to initialize
            mysql -h 127.0.0.1 -P 9030 -u root -e "SET GLOBAL query_queue_concurrency_limit = 4; ..."
Option C: Custom Startup Script (Advanced) You can mount a custom script into the FE pod and change the entrypoint to run your script before/after starting the FE process. However, this is more complex to maintain than Option A. 3. Can I put them in 
fe.conf
? No. System variables like 
query_queue_concurrency_limit
are not recognized by the FE parser when reading 
fe.conf
. If you add them there, the FE will simply ignore them or fail to start because it doesn't recognize the keys. Summary Checklist 1. Check Volume Persistence: Ensure your 
StarRocksCluster
YAML (or Helm chart) has a 
storageSpec
for the FE metadata. If it does, 
SET GLOBAL
is already "permanent." 2. Use a Job: If you need to ensure these are applied even if the database is wiped/recreated, use the Kubernetes Job (Option A). 3. Resource Groups: Since you are on 4.0.x, consider using Resource Groups (
CREATE RESOURCE GROUP
) instead of global query queues. Resource groups allow you to set these same limits more granularly and are also stored permanently in the metadata. References • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md
c
@Rocky does the proposed changes here make sense? Findings • *Spill is enabled but never fires.*BE log shows`spilled bytes:0`on every query. • *Per-query memory is tiny (~8 MB peak).*OOM is*concurrency-driven*, not query-size-driven — hundreds of in-flight queries × 8 MB + standing caches = process wall hit. • *Datacache hit ratio = 100%*across 200 sampled queries. Cache is oversized for current working set. • Disk: 38/49 GiB used, dominated by datacache blockfiles (4 × 10 GiB). Proposed changes (cn.conf) KnobFromToWhy`disable_storage_page_cache`false*true*Redundant with datacache in shared-data mode → frees 640 MiB`datacache_mem_size`1280M*640M*100% hit rate gives headroom → frees 640 MiB`spill_local_storage_dir`(unset)*`/opt/starrocks/cn/storage/spill`*Known path for diagnostics; no behavior change today`lake_metadata_cache_limit`1G(unchanged)Risky to cut without metric Net effect: ~1.28 GiB more memory available for query exec. Doesn't solve concurrency=100 OOM by itself, but raises the ceiling. Assumptions • 100% datacache hit rate is representative of prod query mix (only validated on the load-test queries). • Spill won't trigger meaningfully because per-query peak is ~8 MB — far below any spill threshold. So spill path is hygiene, not a fix. • We're*not*resizing the CN,*not*using resource groups,*not*comparing with prod (prod isn't deployed yet). • Concurrency=100 on a single CN is a known-stressful scenario; OOM at that level is partly expected. Lower-concurrency points (1, 5, 25) are not in scope here.
r
Your proposed changes are a logical "survival" response to the resource constraints you're facing. However, on a node with only 1.8 vCPU, there is a significant risk that saving memory will come at a high cost to CPU—which is already your primary bottleneck. Here is the technical review of your proposed 
cn.conf
changes: 1. 
disable_storage_page_cache = true
 * The Risk: In StarRocks 4.0.x shared-data mode, the "Data Cache" is divided into a Block Cache (raw data on disk) and a Page Cache (metadata, Bloom filters, and ordinal indexes in memory). * The Impact: Setting this to 
true
disables the memory-resident cache for indexes and filters. While you reclaim ~640 MB of RSS, every query will now have to spend CPU cycles to fetch, decompress, and parse these indexes from the local block-cache or S3. * Verdict: Since your CN is already at 1% idle CPU, this change might actually worsen your performance. The CPU cycles required to repeatedly process indexes are likely more valuable than the 640 MB of RAM. I recommend keeping this 
false
but reducing its limit. 2. 
datacache_mem_size = 640M
 * The Reasoning: Given your 100% hit rate and tiny working set, this is a very safe and effective "shrink." * The Math: StarRocks 4.0 simplified cache management. 
datacache_mem_size
is the primary knob here. * Verdict: Proceed. This is the best way to reclaim memory for query execution without the CPU penalty of disabling the index cache entirely. 3. 
spill_local_storage_dir
 * The Reality: As you noted, with 8 MB per query, you are not hitting the spill threshold. * Verdict: Proceed. It’s good hygiene. However, note that if you did hit OOM due to concurrency, spilling wouldn't help much because the OOM is caused by the number of small queries, not the size of one large query. 4. 
lake_metadata_cache_limit
 * The Reasoning: In 4.0 shared-data, BE/CN must cache "tablet/rowset metadata" to avoid constant S3 metadata lookups. * Verdict: Keep as is (1GB). Cutting this can lead to "Metadata Misses," which trigger synchronous S3 calls that will hang your query threads and make the "FE Full GC" (timeout) issue even more frequent. ────────── Revised Recommendation for 1.8 vCPU / 14 GB CN Instead of your proposed set, I suggest this "CPU-First" memory reduction: Knob | Proposed Value | Why? 
datacache_mem_size
 | 
640M
| Safe. Reclaims RAM without increasing CPU overhead for your specific workload. 
disable_storage_page_cache
 | 
false
| Keep enabled. Using CPU to re-parse indexes on every query will kill your 1.8 vCPU node. 
storage_page_cache_limit
 | 
512M
| Limit it instead. (In 4.x, this is still respected as a sub-limit). It keeps hot indexes in RAM but caps the growth. 
query_queue_concurrency_limit
 | 
4
or 
8
 | Mandatory. This is the only "fix" for your scenario. 100 users on 1.8 vCPU is 0.018 vCPU/query. Queuing is your friend here. Summary of Assumptions Validation * Spill: Correct. It won't trigger. You are dying from "death by a thousand cuts" (many small allocations). * Memory Ceiling: These changes will indeed give you ~1GB more headroom, which helps mitigate the 6.5GB/7GB RSS pressure on the FE/CN. * CPU Bottleneck: This remains your "hard wall." If queries start failing with timeouts even after the memory changes, it confirms that the 100 users are simply over-subscribing the physical vCPUs beyond what the Linux scheduler can handle. Final Tip: Since you are on EKS, if this is a production-intent test, consider moving to a 4 vCPU / 16 GB instance (like 
m5.xlarge
or 
c5.xlarge
). StarRocks' multi-threaded vectorized engine really starts to shine only when it has at least 4 physical cores to play with. References • deployment/shared_data/hdfs.md
c
@Rocky Has the 
storage_page_cache_limit
knob been deprecated in shared-data 4.0? Is query queue (
enable_query_queue_select
) what he means by 
query_queue_concurrency_limit
?
r
1. Has 
storage_page_cache_limit
been deprecated? The short answer: Yes, effectively. In StarRocks 4.0 (shared-data mode), the memory management for caching has been consolidated. Here is the technical transition: * Before (Shared-Nothing/Old Versions): You used 
storage_page_cache_limit
(defined as a percentage of memory, like 
20
) to control the internal LRU cache for data pages. * Now (Shared-Data 4.0): The primary knob is 
datacache_mem_size
. * When you are in shared-data mode, the Data Cache (StarCache) takes over the responsibility of caching both data and metadata from remote storage (S3/HDFS). * The 
datacache_mem_size
parameter specifically sets the absolute memory limit (e.g., 
640M
) for the in-memory portion of this cache. * If you still have 
storage_page_cache_limit
in your 
cn.conf
, the system might still respect it for certain internal query engine caches, but it is not recommended to use both. You should rely on 
datacache_mem_size
as your single source of truth for cache memory in 4.0. Recommendation: Remove 
storage_page_cache_limit
and use 
datacache_mem_size
exclusively to avoid conflicting memory reservations. ────────── 2. 
enable_query_queue_select
vs 
query_queue_concurrency_limit
 Yes, they are two halves of the same feature. You cannot use one effectively without the other. Think of it like this: * 
enable_query_queue_select = true
: This is the Master Switch. It tells the FE: "Please start monitoring the concurrency of SELECT queries and put them in a queue if they hit the limit." Without this being 
true
, StarRocks will try to execute every query immediately until the system crashes (OOM). * 
query_queue_concurrency_limit = 4
: This is the Threshold. It tells the switch when to start queuing. Why this matters for your 1.8 vCPU node: If you set 
query_queue_concurrency_limit = 4
but forget 
SET GLOBAL enable_query_queue_select = true
, the limit will be ignored, and your 100 concurrent queries from Locust will all hit the BE at once, causing the OOM you are seeing. Correct setup for your test:
Copy code
sql
-- 1. Turn on the queuing logic for SELECTs
SET GLOBAL enable_query_queue_select = true;

-- 2. Define the concurrency "wall" (the limit)
SET GLOBAL query_queue_concurrency_limit = 4;

-- 3. Define the queue depth (how many can wait before failing)
SET GLOBAL query_queue_max_queued_queries = 200;
Summary for your findings Your logic in the "Proposed Changes" is sound, but ensure you: 1. Use absolute values (
datacache_mem_size = 640M
) instead of percentage limits for the cache. 2. Explicitly enable the switch (
enable_query_queue_select
) so that your concurrency limit actually does something. of 4 actually stops the other 96 queries from entering the execution engine. References • data_source/data_cache.md • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • administration/management/BE_configuration.md • deployment/shared_data/feature-support-shared-data.md • faq/resource_isolation_faq.md References • administration/management/resource_management/query_queues.md • sql-reference/System_variable.md • faq/resource_isolation_faq.md
Open in Slack
PreviousNext
Powered by