@Rocky I have installed StarRocks version

4.1.4

via operator helm, is there a way in the helm to auto-scale fe pods, as today I have 3 pods with 8 core, 32 GB per pod, what's the max concurrency I can have for queries. I have external iceberg catalogs also if there is way to auto-scale FE so that it can handle my burst loads during certain times

Based on the StarRocks documentation and the capabilities of the StarRocks Operator (as of version 4.1.x), here is the information regarding FE auto-scaling and concurrency for your setup. 1. FE Auto-scaling in Helm/Operator Currently, StarRocks FE pods do not support automatic scaling (HPA) via the Helm chart or Operator in the same way that Compute Nodes (CN) do. * Why? The FE (Front-End) follows a specific 1-leader/N-follower/N-observer architecture. Followers must maintain an odd number (usually 1, 3, or 5) to ensure a quorum for metadata consistency. Dynamic auto-scaling is complex for FEs because adding or removing nodes requires explicit metadata sync and election participation. * Best Practice for Burst Loads: For bursty workloads, especially with Iceberg external catalogs, you should focus on auto-scaling CN (Compute Nodes). FEs handle the query planning, but the actual data scanning and processing for Iceberg are done by BEs or CNs. * Scaling FEs: If your FE planning becomes a bottleneck, you should manually scale by adding Observer FE nodes. Observers can scale out the query planning and metadata read capacity without affecting the election quorum. You can update the

replicas

count for

starRocksFeSpec

in your Helm

values.yaml

, but the Operator currently expects you to manage the number of Followers manually (to keep it at an odd number like 3 or 5). 2. Max Query Concurrency For your current setup (3 FEs, each with 8 cores and 32 GB RAM), the "max concurrency" depends on how you define it: * Maximum Connections: Each FE defaults to 4096 connections (

qe_max_connection

). With 3 FEs, you can theoretically have up to 12,288 simultaneous client connections (assuming you have a load balancer like HAProxy or an Ingress in front of them). * Maximum Active Queries: The number of concurrently executing queries is typically limited by your BE/CN resources and your Query Queue configuration. * By default, StarRocks 3.1+ uses a leader-coordinated query queue. * Formula-based limit: The system often uses

query_queue_v2_concurrency_level

(default 4) which scales with BE core count. * Recommendation: For 8-core FE nodes, you are well-provisioned for planning. A single FE node with 8 cores can typically handle hundreds of concurrent query plans per second. If you hit limits, check the session variable

max_user_connections

(default 100 per user) and increase it:

sql
        SET PROPERTY FOR 'your_user' 'max_user_connections' = '1000';

3. Handling Iceberg Burst Loads Since you are using Iceberg, the most effective way to handle burst loads is to use Compute Nodes (CN) with the Auto-scaling Policy enabled in your Helm values:

yaml
starRocksCnSpec:
  autoScalingPolicy:
    minReplicas: 2
    maxReplicas: 20
    hpaPolicy:
      metrics:
        - type: Resource
          resource:
            name: cpu
            target:
              averageUtilization: 70
              type: Utilization

This allows the cluster to spin up more workers to scan Iceberg data during peaks and scale down afterward, while the 3 FE nodes remain as a stable management layer. Summary * Auto-scale FE? No, scale CN instead. * Max Concurrency? ~12k connections (total across 3 FEs). Active query concurrency is limited by

query_queue_concurrency_limit

and BE/CN capacity. * Recommendation: Keep your 3 FEs as they are (it's a solid production config) and implement HPA for CN nodes to handle the Iceberg processing bursts. References • release_notes/release-2.5.md • release_notes/release-4.0.md

@Rocky What if I query external hive catalog not iceberg does something change in that scenario, as currently if I am hitting around 250 QPS my pods are running out of CPU, I have

query_queue_concurrency_limit = 40

.

Moving from Iceberg to a Hive Catalog does not fundamentally change the auto-scaling limitation of the FE, but it changes how the FE handles metadata, which is likely the cause of your CPU spike at 250 QPS. 1. Why Hive is different for the FE While Iceberg stores metadata in manifest files (which StarRocks can process in a distributed way using BEs/CNs), Hive relies heavily on the Hive Metastore (HMS). At 250 QPS, your FE pods are likely hitting CPU limits because: * Metadata Fetching: Every query must check partition information. Even with caching, the FE spends significant CPU cycles managing the Hive metadata cache, calculating splits, and communicating with HMS. * Planning Overhead: Hive planning (especially if there are thousands of partitions) is performed entirely on the FE. 250 QPS is a high volume for query planning; at 8 cores, each FE is only getting a few milliseconds of CPU time per query. * Query Queue Coordination: Since StarRocks 3.1+, the Leader FE manages the query queue globally. At high concurrency, the Leader FE has the additional overhead of managing "slots" for every incoming query across the cluster. 2. Can you Auto-Scale FE for Hive? No, there is still no HPA (Horizontal Pod Autoscaler) support for FE nodes in the Operator/Helm. Adding FE nodes dynamically is risky because "Follower" nodes must maintain a quorum. However, you can manually scale out Observer FEs to handle the 250 QPS load. Observers can handle query planning and metadata reads, effectively offloading the CPU pressure from your Leader/Follower nodes. How to scale in Helm: In your

values.yaml

, you can increase the replica count. If you already have 3 followers, any additional nodes will act as Observers.

yaml
starRocksFeSpec:
  replicas: 5  # 3 Followers + 2 Observers to distribute planning load

3. Tuning to reduce FE CPU at 250 QPS If your FE pods are running out of CPU, try these Hive-specific and general FE optimizations: * Increase Hive Metadata Refresh Interval: If you don't need second-level data freshness, increase the cache TTL to reduce HMS polling.

sql
    -- In your Catalog properties
    "hive.metadata_cache_ttl" = "3600"

* Enable Parallel Metadata Loading: This can speed up planning but might use more CPU temporarily. However, if the bottleneck is waiting on HMS, this helps throughput.

sql
    -- FE Config (fe.conf)
    enable_hms_parallel_load = true

* Check `query_queue_concurrency_limit`: Your limit of

40

is actually quite low for 250 QPS. This means a large number of queries are being queued on the FE. The management of this queue (holding connections, checking slot availability) consumes FE CPU. * Recommendation: If your BE/CN nodes have spare capacity, increase this limit to

100

or more to allow queries to pass through to the execution layer faster, reducing the "bookkeeping" load on the FE. * Use Query Result Cache: For high QPS of repetitive queries, enabling the Result Cache can prevent the FE from having to re-plan the query entirely.

sql
    SET GLOBAL enable_query_result_cache = true;

Summary Recommendation 1. Scale FEs Manually: Increase

replicas

to 5 in Helm. The 2 new nodes will be Observers and can share the planning load. 2. Adjust the Queue: Increase

query_queue_concurrency_limit

. A tight queue at high QPS causes more FE overhead than it saves. 3. Check HMS Latency: Ensure your Hive Metastore isn't the one slowing down the FE planning phase, causing threads to hang and CPU to climb. References • deployment/sr_operator.md • quick_start/helm.md • administration/management/Scale_up_down.md • deployment/deploy_manually.md

@Rocky I have approx 50000 partitions on my hive table, what's the best approach to query this table without hitting CPU limits. All my queries include the partition columns in where clause. Only one table in my cluster is hive rest other tables are iceberg, most of my queries on hive table are purely on this table but some like (10-15%) queries are joins between iceberg and hive table

With 50,000 partitions and 250 QPS, your FE (Front-End) pods are likely hitting a bottleneck in Query Planning and Metadata Management. Even with partition pruning, the FE must maintain a massive cache of partition names and file locations. Since FE auto-scaling is not supported in the Operator, here is the best approach to stabilize your CPU and handle high concurrency for Hive. 1. Distribute Planning Load (Manual Scale-out) Since you cannot use HPA for FEs, the most effective "scaling" is to manually add Observer FE nodes. * Recommendation: Change your

replicas

from 3 to 5 in the Helm chart. * Why: In a 3-node setup, all nodes are "Followers" participating in the BDBJE quorum. By increasing to 5, the 2 new nodes act as Observers. You can then point your BI tools or Load Balancer (Service) to distribute queries across all 5 nodes, spreading the CPU-intensive planning tasks. 2. Hive Catalog Metadata Tuning At 250 QPS, the FE's background metadata refresh might be competing with active query planning. You should "relax" the refresh intervals to reduce CPU spikes. Update your Hive Catalog properties:

sql
ALTER CATALOG hive_catalog SET PROPERTIES (
  -- Increase freshness tolerance from 1 min to 5 or 10 mins
  "metastore_cache_refresh_interval_sec" = "600",
  "remote_file_cache_refresh_interval_sec" = "600",
  -- Ensure partition names are cached (default is true)
  "enable_cache_list_names" = "true"
);

Global FE Config (via

ADMIN SET FRONTEND CONFIG

): * `background_refresh_metadata_interval_millis`: Increase this (default 600,000ms / 10 mins) to a higher value if your Hive data doesn't change every minute. * `query_queue_concurrency_limit`: Increase this from 40 to 100+. A low limit at high QPS causes the FE to spend excessive CPU managing a massive "waiting" queue. 3. Collect Statistics for Hive If the FE doesn't have statistics for those 50,000 partitions, the Cost-Based Optimizer (CBO) works harder to guess the best plan, especially for Hive-Iceberg joins. * Action: Run a manual analysis on your Hive table:

sql
    ANALYZE TABLE hive_catalog.your_db.your_table;

* This stores metadata in StarRocks' internal

_statistics_

database, allowing the FE to plan joins much faster without repeatedly hitting the Hive Metastore (HMS). 4. Optimize Hive-Iceberg Joins For the 10-15% of queries joining Hive and Iceberg: * Query Result Cache: If these queries are repetitive, enable the Result Cache. This allows StarRocks to skip planning and execution entirely if the underlying data hasn't changed.

sql
    SET GLOBAL enable_query_result_cache = true;

* Materialized Views (The "Gold" Standard): For heavy burst loads, create an Asynchronous Materialized View that joins the Hive table and Iceberg table.

sql
    CREATE MATERIALIZED VIEW hive_iceberg_aggr_mv
    REFRESH ASYNC EVERY (INTERVAL 1 HOUR)
    DISTRIBUTED BY HASH(join_key)
    AS
    SELECT ... FROM hive_table JOIN iceberg_table ON ...;

Queries will then automatically rewrite to use the MV, which is stored in StarRocks' high-performance OLAP format, bypassing the Hive Metastore bottlenecks entirely. 5. FE Hardware Check With 50,000 partitions, the JVM Heap is as important as the CPU. * Check: Monitor

fe.gc.log

or JVM metrics. If the FE is spending 20%+ of its time in Garbage Collection (GC) due to the partition metadata, it will manifest as high CPU usage. * Adjustment: Ensure your

JAVA_OPTS

in Helm allocates enough heap (e.g.,

-Xmx24g

for your 32GB pods) to hold the 50k partition metadata comfortably. References • data_source/feature-support-data-lake-analytics.md • data_source/catalog/hive_catalog.md • data_source/catalog/iceberg/iceberg_catalog.md