http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Suggest tuning configs for starrock...
# questions-and-troubleshooting
s
@Rocky Suggest tuning configs for starrocks self hosted with helm on k8s deploy with latest version. My usecase is to have low latency reads from S3 iceberg tables using external glue catalog. Here is my current starrocks values.yaml file.
Copy code
operator:
  global:
    rbac:
      create: true
      serviceAccount:
        name: 'starrocks'
        annotations:
          <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: 'arn:aws:iam::<amazon_id>:role/starrocks-role'

  timeZone: 'UTC'

starrocks:
  starrocksCluster:
    enabledBe: true
    enabledCn: true

  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass

  starrocksFESpec:
    replicas: 3
    serviceAccount: 'starrocks'
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8030"
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    resources:
      requests:
        cpu: 1
        memory: 3Gi
      limits:
        cpu: 4
        memory: 8Gi
    storageSpec:
      name: fe
      storageClassName: 'gp3'
      storageSize: 20Gi
      logStorageSize: 5Gi

  starrocksBeSpec:
    replicas: 3
    serviceAccount: 'starrocks'
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8040"
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    resources:
      requests:
        cpu: 1
        memory: 2Gi
      limits:
        cpu: 4
        memory: 8Gi
    storageSpec:
      name: be
      storageClassName: 'gp3'
      storageSize: 50Gi
      logStorageSize: 5Gi
    config: |
      be_port = 9060
      webserver_port = 8040
      heartbeat_service_port = 9050
      brpc_port = 8060
      sys_log_level = INFO

      datacache_enable = true
      datacache_disk_size = 30%
      datacache_mem_size = 1G
      datacache_eviction_policy = slru
      mem_limit = 90%
      enable_spill = true
      spill_local_storage_dir = ${STARROCKS_HOME}/spill

  starrocksCnSpec:
    replicas: 1
    serviceAccount: 'starrocks'
    autoScalingPolicy:
      version: v2
      minReplicas: 1
      maxReplicas: 5
      hpaPolicy:
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 70
          - type: Resource
            resource:
              name: memory
              target:
                type: Utilization
                averageUtilization: 80
        behavior:
          scaleUp:
            stabilizationWindowSeconds: 30
            policies:
              - type: Pods
                value: 2
                periodSeconds: 30
          scaleDown:
            stabilizationWindowSeconds: 600
            policies:
              - type: Pods
                value: 1
                periodSeconds: 120
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: '/metrics'
        <http://prometheus.io/port|prometheus.io/port>: '8040'
        <http://prometheus.io/scrape|prometheus.io/scrape>: 'true'
    resources:
      requests:
        cpu: 1
        memory: 3Gi
      limits:
        cpu: 4
        memory: 8Gi
    storageSpec:
      name: cn
      storageClassName: 'gp3'
      storageSize: 50Gi
      logStorageSize: 5Gi
    persistentVolumeClaimRetentionPolicy:
      whenScaled: Delete
      whenDeleted: Delete
    config: |
      be_port = 9060
      webserver_port = 8040
      heartbeat_service_port = 9050
      brpc_port = 8060
      sys_log_level = INFO
      datacache_enable = true
      datacache_disk_size = 80%
      datacache_mem_size = 1G
      datacache_eviction_policy = slru
      mem_limit = 90%
      enable_spill = true
      spill_local_storage_dir = ${STARROCKS_HOME}/spill
r
To achieve low-latency reads from S3 Iceberg tables using AWS Glue, your configuration needs to focus on Metadata Caching (to reduce Glue/S3 API calls) and Data Caching (to minimize S3 I/O). Based on your current 
values.yaml
, here are the suggested tuning configurations for the latest StarRocks version (v3.3+): 1. Increase Resource Allocations Your current limits (4 CPU, 8Gi RAM) are below the recommended production baseline for Data Lake scenarios. For low latency, StarRocks requires more memory to store metadata and cache hot data blocks effectively. * Recommendation: Increase FE RAM to at least 16Gi and BE/CN RAM to at least 32Gi - 64Gi. * Why: Data Cache and Iceberg metadata caches consume a significant portion of the process memory. 2. FE Configuration (Metadata Tuning) Add these to 
starrocksFESpec.config
to cache Iceberg/Glue metadata on the FE's local disk, which significantly speeds up query planning.
Copy code
yaml
starrocksFESpec:
  config: |
    # Enable disk-based metadata caching for Iceberg
    enable_iceberg_metadata_disk_cache = true
    # Increase memory for metadata (Default 512MB)
    iceberg_metadata_memory_cache_capacity = 2147483648
    # Increase disk cache for metadata (Default 2GB)
    iceberg_metadata_disk_cache_capacity = 10737418240
    # Enable background refresh to keep Glue metadata warm
    enable_background_refresh_connector_metadata = true
    # Refresh metadata every 2 minutes (Default 10 mins)
    background_refresh_metadata_interval_millis = 120000
3. BE & CN Configuration (Data Cache Tuning) Since you are using S3, the Data Cache is your most critical component. Your current 
datacache_mem_size
of 1G is small. If you increase the pod memory, you should increase the cache size.
Copy code
yaml
starrocksBeSpec: # Apply similar changes to starrocksCnSpec
  config: |
    # ... existing ports ...
    datacache_enable = true
    # Use a larger portion of RAM for block cache (if you increase pod memory to 32Gi+)
    datacache_mem_size = 4G 
    # Enable automatic adjustment or set specific disk quota
    datacache_auto_adjust_enable = false
    datacache_disk_size = 40G 
    # Use SLRU for better hot/cold data separation
    datacache_eviction_policy = slru
    # Avoid spilling to disk for low latency queries
    enable_spill = false
4. Catalog & Session Variables When creating your Iceberg catalog, ensure metadata caching is enabled at the catalog level. You can also set session variables to enable the Query Cache, which caches intermediate results for repetitive queries. Catalog Creation SQL:
Copy code
sql
CREATE EXTERNAL CATALOG iceberg_glue
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "glue",
    "aws.s3.use_instance_profile" = "true",
    "aws.s3.region" = "us-east-1",
    "enable_iceberg_metadata_cache" = "true",
    "iceberg_table_cache_refresh_interval_sec" = "60"
);
Global Session Variables (Run as admin):
Copy code
sql
-- Enable per-tablet query results caching
SET GLOBAL enable_query_cache = true;
-- Allow StarRocks to choose the best plan mode for Iceberg (local vs distributed)
SET GLOBAL plan_mode = 'auto';
5. Storage Class Note You are using 
gp3
. For the Data Cache to perform well, ensure your 
gp3
volumes are provisioned with sufficient throughput (at least 250 MB/s) and IOPS (at least 3000), as the cache performance is bound by local disk I/O. Summary Checklist 1. Memory: Increase Pod memory limits (8Gi is very tight for v3.3+ with Data Cache). 2. FE Cache: Enable 
enable_iceberg_metadata_disk_cache
. 3. Data Cache: Increase 
datacache_mem_size
to at least 4G-8G if RAM permits. 4. Query Cache: Enable 
enable_query_cache
via SQL for sub-second repeat queries. References • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/caching/query_cache.md
s
@Rocky I am also running data marts with dbt (Basically creating aggregate and fact tables for data mart on iceberg using dbt). Then loading dashboards from these aggregates in very low latency via cubejs layer. Give me suggestions around these
r
For your dbt + Iceberg + Cube.js stack, the main challenge is that even aggregated data on S3/Iceberg has physical latency limits (S3 I/O + Glue metadata). To reach the sub-second performance usually required for dashboards, you should move from a "Direct Lake" approach to an "Internal Acceleration" approach using Asynchronous Materialized Views (MVs). Here is the recommended architecture for your data marts: 1. dbt: Optimize the Iceberg Marts When dbt creates your aggregate/fact tables in Iceberg, ensure the physical layout is optimized for StarRocks to scan. * Partitioning: Use 
partition_by
in your dbt config to match the most common dashboard filters (e.g., 
event_date
). * Sorting: Ensure your dbt model includes an 
ORDER BY
clause. StarRocks will use this to sort the data before writing to Iceberg, which improves the effectiveness of Zone Maps (min/max pruning) when you read it back.
Copy code
sql
-- dbt model config for Iceberg
{{ config(
    materialized='table',
    catalog='iceberg_glue',
    database='mart',
    partition_by=['event_date'],
    properties={
      'write.format.default': 'parquet',
      'write.metadata.compression-codec': 'gzip'
    }
) }}

SELECT ... FROM ... ORDER BY filter_column  -- Sort for better pruning
2. StarRocks: The "Acceleration Layer" (Internal MVs) Instead of Cube.js querying the Iceberg table directly, create an Asynchronous Materialized View inside StarRocks. This MV will physically store the data in StarRocks' local storage (internal disks) while staying synced with your Iceberg mart. Benefits: * Local Performance: Queries hit the StarRocks OLAP engine (C++) instead of S3. * Automatic Refresh: StarRocks can detect partition changes in Iceberg and incrementally refresh the MV. * Transparent Rewrite: If Cube.js queries the Iceberg table, StarRocks can automatically rewrite the query to use the MV.
Copy code
sql
CREATE MATERIALIZED VIEW mv_mart_acceleration
PARTITION BY (event_date) -- Align with Iceberg partition
DISTRIBUTED BY HASH(id)
REFRESH ASYNC
AS 
SELECT * FROM iceberg_glue.mart.my_dbt_table;
Workflow Integration: After your dbt job finishes, trigger the MV refresh via the StarRocks API or a SQL command: 
REFRESH MATERIALIZED VIEW mv_mart_acceleration;
3. Cube.js & Query Cache Tuning Since Cube.js often sends identical or high-frequency aggregate queries, enable the Query Cache. This caches intermediate aggregation results in the BE's memory. FE Configuration (
starrocksFESpec.config
):
Copy code
yaml
# Enable the Query Cache feature globally
query_cache_capacity = 2147483648 # 2GB for cache metadata/results
Session Variables (Apply to the user Cube.js uses):
Copy code
sql
-- Enable per-tablet aggregation caching
SET GLOBAL enable_query_cache = true;

-- If using MVs, allow a small amount of staleness for massive speedup
SET GLOBAL mv_rewrite_staleness_second = 300;
4. Summary of Suggested 
values.yaml
Tweaks To support this dbt+Cube workflow, your pods need more "headroom" for caching and MV refreshes:
Copy code
yaml
starrocksBeSpec:
  resources:
    limits:
      cpu: 8        # Increased for concurrent refreshes
      memory: 32Gi  # Increased to hold Data Cache + Query Cache
  config: |
    # Increase memory for Query Cache
    query_cache_max_capacity_mb = 4096
    # Increase threads for writing to S3 if dbt is heavy
    connector_io_tasks_per_scan_operator = 16
Pro-Tip for Cube.js If you use the StarRocks Materialized View approach, you can often disable Cube.js Pre-aggregations. Maintaining pre-aggregations in Cube's own storage adds complexity; StarRocks MVs provide the same performance benefit natively within the database layer, but with better SQL support and easier maintenance through a single engine. References • using_starrocks/caching/query_cache.md • faq/Sql_faq.md
s
@Rocky While I agree on we can disable cube pre-aggregates, that we are not using either way but would simple result caching at cube layer help me. Also my use case is high concurrency (40-50 normal, 100 at burst should not break) as well along with low latency so any specific tuning I have to do for that.
r
For a high-concurrency (100+ burst) dashboard use case with Cube.js and Iceberg, your tuning should focus on Connection Management, Query Queuing, and Multi-layer Caching. 1. Caching Strategy: Cube.js vs. StarRocks Yes, result caching at the Cube layer is still highly recommended. StarRocks' internal Query Cache (tablet-level aggregation cache) is powerful because it allows partial reuse of data even if the query isn't an exact match. However, Cube.js result caching provides: * Zero Latency for Identical Hits: Skips the entire SQL parsing and planning overhead on the StarRocks FE. * Reduced Connection Load: A hit in the Cube cache means one fewer active connection/thread being used in StarRocks. Recommendation: Use Cube.js Redis-based caching for exact widget hits, and enable StarRocks Query Cache for "near-miss" queries (e.g., different date ranges that overlap). ────────── 2. High Concurrency Tuning (100+ Burst) To handle 100 concurrent queries without breaking, you must adjust the default limits and enable the Query Queue to prevent the cluster from being overwhelmed. FE Configuration (
starrocksFESpec.config
) Add these to handle more simultaneous connections and metadata lookups:
Copy code
yaml
starrocksFESpec:
  config: |
    # Increase maximum connections (Default is 1024, but 8Gi RAM may need monitoring)
    qe_max_connection = 2000
    # Increase threads for handling metadata for Iceberg/S3
    metadata_refresh_poll_threads = 16
Global SQL Variables (Critical for Concurrency) Run these once as 
root
to prepare the engine for high load:
Copy code
sql
-- 1. Increase user-specific connection limits
ALTER USER 'cubejs_user' SET PROPERTIES ("max_user_connections" = "500");

-- 2. Enable Global Query Queue (Safeguard for bursts)
-- This ensures that if 100+ heavy queries arrive, they queue rather than OOM-ing the nodes.
SET GLOBAL enable_query_queue_select = true;
SET GLOBAL query_queue_concurrency_limit = 100; -- Concurrent queries per BE
SET GLOBAL query_queue_max_queued_queries = 512;

-- 3. Adaptive Parallelism
-- Automatically reduces parallelism per query as concurrency increases to keep latency stable.
SET GLOBAL pipeline_dop = 0; -- 0 means adaptive
────────── 3. BE/CN Configuration for Burst Stability For your 
values.yaml
, increase the BRPC threads and ensure the memory limit is high enough to support both the Data Cache and the Query Cache.
Copy code
yaml
starrocksBeSpec:
  config: |
    # Increase threads for internal communication
    brpc_num_threads = 256 
    # Enable and tune Query Cache (BE RAM)
    query_cache_capacity = 2147483648 # 2GB
    # Ensure memory limit is safe but aggressive
    mem_limit = 90%
────────── 4. Summary for the Data Mart + Cube.js Workflow 1. Materialized Views: If your Iceberg aggregate tables are still slightly slow for the burst load, create an Async MV inside StarRocks that points to the Iceberg mart. This moves the data to local disks, which is significantly faster for 100+ concurrent scans than S3 I/O. 2. Resource Groups: If you have other workloads (like dbt refreshes) happening simultaneously, put Cube.js in its own resource group:
Copy code
sql
    CREATE RESOURCE GROUP cube_dashboard_group
    PROPERTIES (
        "cpu_core_limit" = "10",
        "mem_limit" = "40%",
        "concurrency_limit" = "100"
    );
    -- Bind your Cube.js DB user to this group
    ALTER USER 'cubejs_user' SET PROPERTIES ("resource_group" = "cube_dashboard_group");
Final Note on Resources: Your current 8Gi RAM limit per node is the biggest bottleneck for 100 concurrent queries. Each query context and the caches consume memory. If you experience crashes during bursts, increase the pod memory to 16Gi or 32Gi. References • administration/management/resource_management/query_queues.md
2 Views
Open in Slack
PreviousNext
Powered by