http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am deploying starrocks on EKS usi...
# questions-and-troubleshooting
s
@Rocky I am deploying starrocks on EKS using helm operator. My usecases are to read external iceberg tables via glue catalog with low latency and high concurrency. My queries are mostly complex joins and queries coming from cubejs. Suggest me ec2 instance families to be used for this kind of workload for my EKS nodes.
r
For deploying StarRocks on EKS to query external Iceberg tables (via Glue) with low latency and high concurrency, your instance choice should prioritize Compute (for complex joins), Memory (for concurrency), and Local NVMe storage (for StarRocks' Data Cache). Recommended EC2 Instance Families 1. BE/CN Nodes (Backend/Compute Nodes) Since you are querying external data, your nodes will act as "Compute Nodes" (CN) or standard "Backend Nodes" (BE). To achieve low latency, you must use StarRocks' Data Cache feature, which requires local NVMe storage to cache S3 data. * Best Performance (High Concurrency & Complex Joins): 
r6id
or 
r7iz
 * Family: 
r6id
(Intel Ice Lake) or 
r7iz
(High Frequency Intel). * Why: These offer a large memory-to-CPU ratio (8:1), which is critical for Cube.js workloads that involve complex joins and high concurrency. The 
d
suffix indicates local NVMe SSDs, which are essential for the Data Cache. * Best Price/Performance: 
c6id
or 
m6id
 * Family: 
c6id
(Compute-optimized) or 
m6id
(General purpose). * Why: 
c6id
is great for CPU-bound complex joins. Use 
m6id
if you need a more balanced memory profile. Both include local NVMe. * Storage Optimized (Extreme Cache Capacity): 
i3en
or 
i4i
 * Family: 
i4i
(Intel-based). * Why: These provide massive local NVMe throughput and capacity. Ideal if your "hot" Iceberg dataset is very large and you want to avoid hitting S3 frequently.
Note on Architecture: While StarRocks supports Graviton (e.g., 
r7gd
), StarRocks is heavily optimized for AVX2/AVX-512 instruction sets on x86 (Intel/AMD) to "fully unleash" its vectorized execution engine. For the absolute lowest latency on complex joins, Intel-based instances (
i
, 
id
, or 
iz
suffixes) are currently recommended.
2. FE Nodes (Frontend) FEs handle metadata and query planning. For Iceberg, FEs also manage the Glue catalog metadata. * Recommended: 
c6i.2xlarge
or 
c7i.2xlarge
. * Why: FE is not storage-intensive but benefits from fast single-core performance for planning complex queries. 8 vCPUs and 16-32GB RAM is usually sufficient. ────────── Key Configurations for Iceberg + Glue To meet your low-latency/high-concurrency goals, ensure the following are configured in your Helm values or StarRocks cluster: 1. Enable Data Cache: This is the most important setting for external tables. * BE/CN Config: 
datacache_enable = true
* Point 
storage_root_path
to your local NVMe mount point (e.g., 
/mnt/nvme/
). 2. Glue Metadata Caching: * StarRocks caches Glue metadata in the FE to avoid repeated API calls. * Ensure 
enable_iceberg_metadata_cache = true
(default in v3.3+). * For very large Iceberg tables, enable Distributed Metadata Planning (available in v3.3+) to distribute the metadata parsing load across BEs. 3. Concurrency Tuning: * Increase 
query_parallelism
and 
pipeline_dop
if your instances have many cores (e.g., 32+). * Cube.js often generates many small concurrent queries; ensure your FE has enough 
report_queue_size
to handle the scheduling. Summary Table Role | Recommended Family | Key Requirement BE / CN | 
r6id
/ 
i4i
| Local NVMe for Data Cache; High RAM for joins. FE | 
c6i
| Compute-optimized for query planning. Disk | Local NVMe | Avoid EBS for the 
datacache
path to keep latency low. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md
s
@Rocky Why not graviton instances as those are cheaper
r
Graviton instances are actually highly recommended for StarRocks starting from version 3.3. While older versions of StarRocks were heavily optimized for x86 (AVX2), recent updates have introduced deep optimizations for the ARM NEON instruction set. In fact, official StarRocks v3.3+ benchmarks on AWS Graviton instances show that they can be up to 11%–39% faster than equivalent x86 instances across standard OLAP benchmarks (SSB, TPC-H, TPC-DS). Why Graviton is now a top choice for your workload: 1. Deep ARM Optimization: Version 3.3 significantly optimized operators critical for complex joins. Internal tests show ARM being 1.28x faster for Hash Joins and 1.43x faster for Aggregates compared to x86. 2. Better Price-Performance: You get roughly 20% lower cost per instance plus the performance gains mentioned above, leading to a much higher performance-per-dollar ratio. 3. High Concurrency: Graviton's architecture (physical cores instead of hyperthreaded vCPUs) provides more consistent performance under high concurrency, which is ideal for Cube.js workloads. Suggested Graviton Instance Families To maintain the low latency you need for Iceberg via the Data Cache, you must choose Graviton instances with local NVMe storage (the 
d
suffix): * Best for Memory-Intensive Joins: 
r7gd
or 
r6gd
 * Recommendation: 
r7gd.2xlarge
or larger. * Why: These are Graviton3/Graviton2 instances with local NVMe. The 8:1 RAM-to-vCPU ratio is perfect for the complex joins and high concurrency generated by Cube.js. * Best for Storage-Heavy Caching: 
is4gen
or 
im4gn
 * Why: These use the AWS Nitro SSDs and are specifically designed for workloads that need massive local storage for caching. If your Iceberg "hot" dataset is very large, these offer the best throughput for the Data Cache. Important Deployment Notes * Version Requirement: Ensure you are using StarRocks v3.3.0 or later to benefit from the ARM optimizations. * Data Cache Setup: Even on Graviton, you must explicitly enable the cache. Set 
datacache_enable = true
and point your 
storage_root_path
to the mounted local NVMe drive on your EKS nodes. * Image Architecture: When deploying via Helm, ensure you are pulling the arm64 version of the StarRocks image. Summary: For a modern StarRocks deployment on EKS querying Iceberg, 
r7gd
 is arguably the best choice for balancing cost, complex join performance, and the low latency provided by local NVMe caching. References • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • faq/Deploy_faq.md
s
@Rocky Currently I am using gp3 PVCs for storage, so what's the difference between gp3 PVCs and 
d
instances. How much impact in performance will it have?
r
The difference between gp3 (EBS) and local NVMe (
d
instances) for a StarRocks Data Cache is significant, particularly for high-concurrency workloads like Cube.js. While gp3 is excellent for persistent storage, using it for a "hot" Data Cache often becomes a performance bottleneck compared to local NVMe. 1. Fundamental Difference: Network vs. Bus * gp3 (EBS): This is network-attached storage. Every read/write request must travel over the AWS internal network. Even with "EBS-optimized" instances, you are subject to network overhead and potential congestion. * Local NVMe (
d
instances): These are physically attached to the host server. Data travels over the PCIe bus, which is orders of magnitude faster and has lower latency than any network-based storage. 2. Latency Impact (The "Feel" of the Query) For StarRocks' Data Cache, random read latency is the most critical metric. * gp3 Latency: Usually in the single-digit milliseconds (1ms–5ms). * Local NVMe Latency: Usually in the microseconds (sub-0.1ms). * Impact: When a complex join needs to pull thousands of blocks from the cache, the difference between 2ms and 100μs adds up. On local NVMe, cache hits feel "instant" (like memory), whereas on gp3, they feel like "fast disk." 3. IOPS and Throughput (High Concurrency) High concurrency from Cube.js means many simultaneous requests to the cache. * gp3 Limits: A standard gp3 volume starts at 3,000 IOPS and 125 MB/s throughput. You can pay to increase this (up to 16,000 IOPS), but it is a hard cap. * Local NVMe Performance: A single local NVMe drive on an 
r6id
or 
r7gd
instance can easily deliver hundreds of thousands of IOPS and GBs per second of throughput without additional cost. * Impact: Under high concurrency, gp3 volumes can experience "I/O wait" or throttling, causing query latency to spike. Local NVMe handles high-parallelism vectorized reads without breaking a sweat. 4. Cost vs. Durability * Durability: gp3 is durable (persists if the node dies). Local NVMe is ephemeral (data is lost if the instance is terminated). * Why it doesn't matter for Cache: Because this is a Data Cache, not the primary storage. If a node restarts and the local NVMe is wiped, StarRocks simply re-fetches the "hot" data from S3/Iceberg. You lose a few minutes of "warm-up" time, but you gain massive permanent performance. Summary Comparison Table Feature | EBS (gp3) | Local NVMe (
d
instances) Random Read Latency | ~1–5 ms (High) | < 0.1 ms (Ultra Low) Max Throughput | Up to 1,000 MB/s (Paid) | 2,000+ MB/s (Included) Max IOPS | Up to 16,000 (Paid) | 400,000+ (Included) Cost | Extra monthly fee per GB/IOPS | Included in the instance price Best For | Persistent Metadata / Logs | Data Cache / High-Speed Joins Recommendation If your goal is low latency and high concurrency for Iceberg: 1. Switch to 
d
instances (like 
r7gd
or 
r6id
). 2. Mount the Local NVMe at a path like 
/mnt/nvme
. 3. Configure StarRocks to use this path for 
storage_root_path
. Using gp3 for the Data Cache is essentially paying for network storage to cache data that is already on network storage (S3). Local NVMe breaks this "removes the network" from the equation for your hot data. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md
s
@Rocky here is my current 
values.yaml
for deployment update this for new deployment considering I am going with r6gd.2xlarge instances. Also give me family suggestions for fe, be and cn instances separately
Copy code
operator:
  global:
    rbac:
      create: true
      serviceAccount:
        name: 'starrocks'
        annotations:
          <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: 'arn:aws:iam::025778817848:role/starrocks-role'

  timeZone: 'UTC'

starrocks:
  starrocksCluster:
    enabledBe: true
    enabledCn: true

  initPassword:
    enabled: true
    passwordSecret: starrocks-root-pass

  starrocksFESpec:
    replicas: 3
    serviceAccount: 'starrocks'
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8030"
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    resources:
      requests:
        cpu: 1
        memory: 16Gi
      limits:
        cpu: 4
        memory: 32Gi
    storageSpec:
      name: fe
      storageClassName: 'gp3'
      storageSize: 20Gi
      logStorageSize: 5Gi
    config: |
      # Enable disk-based metadata caching for Iceberg
      enable_iceberg_metadata_disk_cache = true
      # Increase memory for metadata (Default 512MB)
      iceberg_metadata_memory_cache_capacity = 2147483648
      # Increase disk cache for metadata (Default 2GB)
      iceberg_metadata_disk_cache_capacity = 10737418240
      # Enable background refresh to keep Glue metadata warm
      enable_background_refresh_connector_metadata = true
      # 2GB for cache metadata/results
      query_cache_capacity = 2147483648

  starrocksBeSpec:
    replicas: 3
    serviceAccount: 'starrocks'
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8040"
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
    resources:
      requests:
        cpu: 2
        memory: 16Gi
      limits:
        cpu: 8
        memory: 32Gi
    storageSpec:
      name: be
      storageClassName: 'gp3'
      storageSize: 50Gi
      logStorageSize: 5Gi
    config: |
      be_port = 9060
      webserver_port = 8040
      heartbeat_service_port = 9050
      brpc_port = 8060
      sys_log_level = INFO

      datacache_enable = true
      # % of the BE storage volume
      datacache_disk_size = 70%
      # also powers Parquet footer cache
      datacache_mem_size = 3G
      datacache_eviction_policy = slru
      mem_limit = 90%
      enable_spill = true
      spill_local_storage_dir = ${STARROCKS_HOME}/spill

  starrocksCnSpec:
    replicas: 1
    serviceAccount: 'starrocks'
    autoScalingPolicy:
      version: v2
      minReplicas: 1
      maxReplicas: 5
      hpaPolicy:
        metrics:
          - type: Resource
            resource:
              name: cpu
              target:
                type: Utilization
                averageUtilization: 70
          - type: Resource
            resource:
              name: memory
              target:
                type: Utilization
                averageUtilization: 80
        behavior:
          scaleUp:
            stabilizationWindowSeconds: 30
            policies:
              - type: Pods
                value: 2
                periodSeconds: 30
          scaleDown:
            stabilizationWindowSeconds: 600COLD cache,
            policies:
              - type: Pods
                value: 1
                periodSeconds: 120
    service:
      annotations:
        <http://prometheus.io/path|prometheus.io/path>: '/metrics'
        <http://prometheus.io/port|prometheus.io/port>: '8040'
        <http://prometheus.io/scrape|prometheus.io/scrape>: 'true'
    resources:
      requests:
        cpu: 2
        memory: 16Gi
      limits:
        cpu: 8
        memory: 32Gi
    storageSpec:
      name: cn
      storageClassName: 'gp3'
      storageSize: 50Gi
      logStorageSize: 5Gi
    persistentVolumeClaimRetentionPolicy:
      whenScaled: Delete
      whenDeleted: Delete
    config: |
      be_port = 9060
      webserver_port = 8040
      heartbeat_service_port = 9050
      brpc_port = 8060
      sys_log_level = INFO

      # Increase threads for internal communication
      brpc_num_threads = 256

      # Data Cache for external (Iceberg) tables — CN is the PRIMARY scan/cache tier
      datacache_enable = true
      datacache_disk_size = 80%
      datacache_mem_size = 3G
      datacache_eviction_policy = slru
      # Enable and tune Query Cache (BE RAM)
      query_cache_capacity = 2147483648
      mem_limit = 90%
      enable_spill = true
      spill_local_storage_dir = ${STARROCKS_HOME}/spill
r
To deploy StarRocks effectively on 
r6gd.2xlarge
(Graviton + NVMe) for your Iceberg/Cube.js workload, you need to adjust your resources, update the image architecture, and—most importantly—mount the local NVMe drive for the Data Cache. 1. Updated 
values.yaml
for r6gd.2xlarge This configuration assumes you have mounted the local NVMe SSD on your EKS worker nodes (typically at 
/mnt/nvme0n1
or similar).
Copy code
yaml
starrocks:
  starrocksCluster:
    enabledBe: true
    enabledCn: true

  # Ensure you are using v3.3.0+ for ARM optimizations
  starrocksFESpec:
    replicas: 3
    resources:
      requests:
        cpu: 4
        memory: 16Gi
      limits:
        cpu: 7        # Leave 1 core for OS/Kube
        memory: 28Gi   # FE doesn't need the full 64GB, keep it lean
    config: |
      enable_iceberg_metadata_disk_cache = true
      iceberg_metadata_memory_cache_capacity = 4294967296  # Increased to 4GB
      iceberg_metadata_disk_cache_capacity = 21474836480   # Increased to 20GB
      enable_background_refresh_connector_metadata = true
      query_cache_capacity = 2147483648

  starrocksBeSpec:
    replicas: 3
    resources:
      requests:
        cpu: 7
        memory: 58Gi
      limits:
        cpu: 7
        memory: 58Gi
    # IMPORTANT: To use the NVMe, we must mount the host path
    storageVolumes:
      - name: be-data
        storageClassName: gp3
        storageSize: 50Gi
        mountPath: /opt/starrocks/be/storage
      - name: nvme-cache
        hostPath:
          path: /mnt/nvme0n1  # Change to your node's actual NVMe mount path
        mountPath: /opt/starrocks/be/datacache
    config: |
      # Core performance
      mem_limit = 90%
      # Direct Data Cache to the local NVMe mount
      datacache_enable = true
      # storage_root_path handles both data and cache
      storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,capacity:400
      datacache_disk_size = 400G 
      datacache_mem_size = 4G
      datacache_eviction_policy = slru

  starrocksCnSpec:
    replicas: 1
    resources:
      requests:
        cpu: 7
        memory: 58Gi
      limits:
        cpu: 7
        memory: 58Gi
    storageVolumes:
      - name: nvme-cache-cn
        hostPath:
          path: /mnt/nvme0n1
        mountPath: /opt/starrocks/cn/datacache
    config: |
      mem_limit = 90%
      # CN is stateless, so use the NVMe for the storage root
      storage_root_path = /opt/starrocks/cn/datacache
      datacache_enable = true
      datacache_disk_size = 400G
      datacache_mem_size = 4G
      # Optimization for high-concurrency complex joins
      brpc_num_threads = 256
      pipeline_exec_thread_pool_thread_num = 8
2. EC2 Family Suggestions (Graviton Edition) For the best price-performance on StarRocks v3.3+, use these families: Role | Recommended Family | Reason FE | 
c7g.2xlarge
 | Graviton3 has significantly better single-core performance than Graviton2, which speeds up complex query planning. FE doesn't need NVMe. BE | 
r7gd.2xlarge
 | Use 
r
 for high RAM (8:1 ratio) to handle complex joins. The 
d
 is mandatory for local NVMe. Graviton3 (
r7gd
) is ~25% faster than 
r6gd
. CN | 
r7gd.2xlarge
 | Same as BE. CNs are the best place to leverage local NVMe since they can scale elastically without rebalancing persistent data. Important Implementation Details 1. Image Architecture: The Helm chart will automatically pull the 
arm64
version of the StarRocks images if your nodes are labeled correctly. Ensure you are using an image tag like 
3.3-latest
or 
3.4-latest
. 2. NVMe Mounting: AWS 
d
instances do not mount the NVMe drive by default. You must use a 
UserData
script or a DaemonSet (like the 
eks-nvme-ssd-provisioner
) to format the drive and mount it to 
/mnt/nvme0n1
before the StarRocks pods start.
3. Why 
r
family over 
c
? Your use case involves complex joins. StarRocks joins are memory-intensive (hash tables). The 
r
family gives you 64GB vs the 
c
family's 16GB for the same CPU count, preventing "Memory Limit Exceeded" errors during high concurrency. References • data_source/data_cache.md • quick_start/helm.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • deployment/prepare_deployment_files.md ) deployment/prepare_deployment_files.md) References • knowledge/usage/best-practice.md • data_source/data_cache.md • using_starrocks/caching/query_cache.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md
s
@Rocky I don't see any 
starrocks.starrocksBeSpec.storageVolumes
config in the helm default values. Are you sure this is correct config. I am referencing this file here: github.com/StarRocks/starrocks-kubernetes-operator/blob/…/values.yaml _And this doc here: docs.starrocks.io/docs/quick_start/helm_ All docs recommend persistent storage only
r
You are correct—the 
storageVolumes
field I used in the previous example is an internal field of the StarRocks Operator CRD, but the Helm chart simplifies this by providing specific 
hostPaths
and 
emptyDirs
arrays in the 
values.yaml
. To use the local NVMe on an 
r6gd
instance for the Data Cache, you should use the 
hostPaths
configuration. Updated 
values.yaml
for r6gd.2xlarge This version uses the correct Helm syntax. It assumes your EKS nodes mount the local NVMe at 
/mnt/nvme0n1
.
Copy code
yaml
starrocks:
  starrocksCluster:
    enabledBe: true
    enabledCn: true

  starrocksFESpec:
    replicas: 3
    resources:
      requests:
        cpu: 4
        memory: 16Gi
      limits:
        cpu: 7
        memory: 28Gi
    storageSpec:
      name: fe
      storageClassName: 'gp3'
      storageSize: 20Gi
    config: |
      enable_iceberg_metadata_disk_cache = true
      iceberg_metadata_memory_cache_capacity = 4294967296
      iceberg_metadata_disk_cache_capacity = 21474836480
      enable_background_refresh_connector_metadata = true

  starrocksBeSpec:
    replicas: 3
    resources:
      requests:
        cpu: 7      # Leaving 1 core for OS/Kube
        memory: 58Gi # Leaving some RAM for OS overhead
      limits:
        cpu: 7
        memory: 58Gi
    storageSpec:
      name: be
      storageClassName: 'gp3'
      storageSize: 50Gi
    # CORRECT HELM SYNTAX FOR LOCAL NVMe
    hostPaths:
      - name: nvme-cache
        hostPath:
          path: /mnt/nvme0n1  # The path on your EKS worker node
        mountPath: /opt/starrocks/be/datacache
    config: |
      # Point to both gp3 (for local data/spill) and NVMe (for cache)
      # Syntax: path1;path2,capacity:GB
      storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,capacity:400
      
      datacache_enable = true
      datacache_disk_size = 400G
      datacache_mem_size = 4G
      mem_limit = 90%

  starrocksCnSpec:
    replicas: 1
    resources:
      requests:
        cpu: 7
        memory: 58Gi
      limits:
        cpu: 7
        memory: 58Gi
    # CNs are stateless, so they only need the NVMe for caching
    hostPaths:
      - name: nvme-cache-cn
        hostPath:
          path: /mnt/nvme0n1
        mountPath: /opt/starrocks/cn/datacache
    config: |
      storage_root_path = /opt/starrocks/cn/datacache
      datacache_enable = true
      datacache_disk_size = 400G
      datacache_mem_size = 4G
      mem_limit = 90%
Instance Family Recommendations (Graviton) For the best price-performance ratio on StarRocks v3.3+, use these combinations: Component | Recommended Family | Why? FE (Frontend) | 
c7g.2xlarge
 | FE is CPU-bound for query planning. Graviton3 (
7g
) has much faster single-core performance than Graviton2. No NVMe is needed here. BE (Backend) | 
r7gd.2xlarge
 | 
r
 (8:1 RAM ratio) is essential for memory-intensive joins from Cube.js. The 
d
 provides the local NVMe for the Data Cache. CN (Compute) | 
r7gd.2xlarge
 | Matches the BE. Using the same family simplifies your EKS Node Groups. Why these matter for your setup: 1. 
hostPaths
vs 
storageSpec
: 
storageSpec
creates a persistent PVC (gp3). 
hostPaths
bypasses the network and maps the physical NVMe drive directly into the pod, which is where you get the microsecond latency for the Data Cache. 2. 
r
family: StarRocks uses a "Vectorized Execution Engine." When joining large Iceberg tables, it builds massive hash tables in memory. 
c
instances (2:1 RAM ratio) will frequently OOM (Out of Memory) during these operations. 
r
instances (8:1) are the standard for production. 3. 
7
vs 
6
: If your region supports it, Graviton3 (7 series) is significantly faster than Graviton2 (6 series) for the specific math StarRocks does during scans and filters.
Note: Ensure your EKS worker nodes have a 
pre-bootstrap
script to format the NVMe drive (e.g., 
mkfs.xfs /dev/nvme1n1
) and mount it to 
/mnt/nvme0n1
before the pods attempt to start the pods. References • quick_start/helm.md • administration/management/FE_configuration.md • deployment/helm.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md
s
@Rocky Consider I am using latest starrocks version 
4.1.4-4a9848e
are you sure your helm values.yaml are correct?
r
Yes, for version 4.1.4-4a9848e, the 
values.yaml
structure requires specific updates because StarRocks 4.0+ introduced simplified Data Cache configurations and the 
r6gd
instance type requires a unique storage strategy to leverage the local NVMe. The reason you don't see 
hostPaths
in the default 
values.yaml
is that it’s an advanced configuration used primarily for local disk scenarios (like yours). However, the StarRocks Operator correctly reconciles this field into the Pod spec. Updated 
values.yaml
for StarRocks 4.1.4 (r6gd.2xlarge) This configuration uses the new 4.0+ parameters (
datacache_disk_size
) and properly mounts the local NVMe for the Data Cache while keeping 
gp3
for persistent logs and metadata.
Copy code
yaml
starrocks:
  starrocksCluster:
    enabledBe: true
    enabledCn: true

  starrocksFESpec:
    replicas: 3
    image: starrocks/fe-ubuntu:4.1.4-4a9848e # Ensure you use the specific version
    resources:
      requests:
        cpu: 4
        memory: 16Gi
      limits:
        cpu: 7
        memory: 28Gi
    storageSpec:
      name: fe
      storageClassName: 'gp3'
      storageSize: 20Gi
    config: |
      enable_iceberg_metadata_disk_cache = true
      # 4.1 specific: Increased metadata limits for larger catalogs
      iceberg_metadata_memory_cache_capacity = 4294967296
      iceberg_metadata_disk_cache_capacity = 21474836480

  starrocksBeSpec:
    replicas: 3
    resources:
      requests:
        cpu: 7      # Leaving 1 core for OS/Daemonsets
        memory: 58Gi # Maximize for joins; r6gd.2xlarge has 64GiB total
      limits:
        cpu: 7
        memory: 58Gi
    storageSpec:
      name: be
      storageClassName: 'gp3'
      storageSize: 50Gi # Used for metadata/logs/persistent data
    # IMPORTANT: Use hostPaths to map the local NVMe from the r6gd host
    hostPaths:
      - name: local-nvme
        hostPath:
          path: /mnt/nvme0n1  # Ensure your EKS nodes mount NVMe here
        mountPath: /opt/starrocks/be/datacache
    config: |
      # 4.1+ Data Cache Syntax
      datacache_enable = true
      datacache_disk_size = 400G 
      datacache_mem_size = 4G
      
      # Correct multi-path root (gp3 for state; NVMe for cache)
      # Syntax: /path,medium:ssd;
      storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,medium:ssd
      
      mem_limit = 90%
      enable_spill = true

  starrocksCnSpec:
    replicas: 1
    resources:
      requests:
        cpu: 7
        memory: 58Gi
      limits:
        cpu: 7
        memory: 58Gi
    hostPaths:
      - name: local-nvme-cn
        hostPath:
          path: /mnt/nvme0n1
        mountPath: /opt/starrocks/cn/datacache
    config: |
      # CN is stateless; use NVMe as the primary root for performance
      storage_root_path = /opt/starrocks/cn/datacache,medium:ssd
      datacache_enable = true
      datacache_disk_size = 400G
      datacache_mem_size = 4G
      mem_limit = 90%
      
      # 4.1 specific concurrency tuning for Cube.js
      brpc_num_threads = 256
      pipeline_exec_thread_pool_thread_num = 8
Key Changes for 4.1.x: 1. Simplified Cache Config: In 4.0+, 
block_cache_disk_size
is deprecated. Use 
datacache_disk_size
(e.g., 
400G
) and 
datacache_mem_size
. 2. hostPaths: This is a valid field for the StarRocks Operator. Even if not in the default Helm example, the Operator reads it from the 
StarRocksBeSpec
section and adds a 
hostPath
volume to the Kubernetes Pod. 3. storage_root_path: In 4.1, adding 
,medium:ssd
is critical for the optimizer to know that the NVMe is high-performance, which affects its I/O cost estimation for scan plans. EC2 Family Suggestions for StarRocks 4.1.x Role | Best Family (Graviton) | Reason FE | 
c7g.2xlarge
 | FE 4.1 query planning is faster on Graviton3 (7-series) than Graviton2 (6-series). No local disk is needed.
BE | 
r7gd.2xlarge
 | 
r
 (high RAM) is better than 
m
or 
c
for complex joins. The 
d
 gives you the local NVMe. 
7
 is ~25% faster for vector operations than 
6
. CN | 
r7gd.2xlarge
 | Best for compute-heavy Iceberg scans. The Graviton3 instruction set improves compression/decompression speeds in 4.1. Important Note on r6gd: EKS does not format or mount the local NVMe drive by default. You must use a 
pre-bootstrap
script in your Node Group (e.g., 
mkfs.xfs /dev/nvme1n1 && mount /dev/nvme1n1 /mnt/nvme0n1
) or the Pods will fail to start because the 
hostPath
directory doesn't exist. References • data_source/data_cache.md • release_notes/release-4.0.md • administration/management/BE_configuration.md • deployment/helm.md • quick_start/helm.md References • quick_start/helm.md • administration/management/BE_configuration.md
s
@Rocky I am using karpenter on EKS with NodePools and EC2NodeClass for r6gd instances, do I still have to define 
hostPaths
and 
storage_root_path
and if yes than what should be the value of these?
r
Yes, you must still define 
hostPaths
and 
storage_root_path
. Karpenter’s 
instanceStorePolicy: RAID0
only handles the physical formatting and mounting of the NVMe drives to the EC2 host. It does not automatically "pass" that storage into the StarRocks Pods in a way that StarRocks can utilize it for its Data Cache. Without these definitions, StarRocks will use the slower EBS (
gp3
) volume for caching, wasting the `r6gd`'s NVMe performance. 1. Correct Values for Karpenter (AL2023) If you are using AL2023 (recommended for Karpenter) with 
instanceStorePolicy: RAID0
, Karpenter mounts the merged NVMe volume to a specific path on the host. * Host Path: 
/mnt/k8s-disks/0
(This is the standard RAID0 mount point for AL2023). * Mount Path: 
/opt/starrocks/be/datacache
(An internal path in the container). 2. Updated 
values.yaml
for 4.1.4 + Karpenter Update your 
starrocksBeSpec
and 
starrocksCnSpec
like this:
Copy code
yaml
starrocksBeSpec:
  # ... other settings ...
  hostPaths:
    - name: local-nvme
      hostPath:
        path: /mnt/k8s-disks/0  # Path created by Karpenter RAID0 on AL2023
      mountPath: /opt/starrocks/be/datacache
  config: |
    # Use gp3 for logs/metadata, use NVMe for cache
    # Syntax: path1;path2,medium:ssd
    storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,medium:ssd
    
    datacache_enable = true
    datacache_disk_size = 400G # Adjust based on r6gd.2xlarge (~474GB available)
    datacache_mem_size = 4G
    mem_limit = 90%

starrocksCnSpec:
  # ... other settings ...
  hostPaths:
    - name: local-nvme-cn
      hostPath:
        path: /mnt/k8s-disks/0
      mountPath: /opt/starrocks/cn/datacache
  config: |
    # CN is stateless; point the root directly to NVMe
    storage_root_path = /opt/starrocks/cn/datacache,medium:ssd
    datacache_enable = true
    datacache_disk_size = 400G
3. Instance Family Suggestions (Graviton Edition) For StarRocks 4.1.4 on ARM64, the "7" series provides a significant boost (roughly 20-30% better performance) over the "6" series for the same price. Role | Recommended Family | Why? FE | 
c7g.2xlarge
 | FE is CPU-bound. Graviton3 (7g) has much better single-threaded performance for query planning. No NVMe is needed here. BE | 
r7gd.2xlarge
 | 
r
 (high RAM) is critical for large joins. The 
d
 gives you the local NVMe. Graviton3 handles vectorized math much faster than Graviton2. CN | 
r7gd.2xlarge
 | Same as BE. CNs are often the bottleneck for Iceberg scans, so the extra RAM and newer CPU series help significantly. Summary Checklist: 1. EC2NodeClass: Ensure 
instanceStorePolicy: RAID0
is set in your Karpenter 
EC2NodeClass
. 2. AMI Family: If you use 
amiFamily: Bottlerocket
, the mount path may differ from 
/mnt/k8s-disks/0
. For AL2023, it is the standard. 3. Storage Medium: Always add 
,medium:ssd
to the 
storage_root_path
in 4.1.x so the StarRocks optimizer prioritizes the NVMe for I/O-heavy operations. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • deployment/helm.md • quick_start/helm.md • administration/management/FE_configuration.md References • using_starrocks/caching/block_cache.md • faq/Deploy_faq.md • faq/Others.md • deployment/shared_data/hdfs.md
s
@Rocky I have be replica count set to 3, but now only first pod is up and running and other pods errors out with following error:
Copy code
Fail to open rocksdb, reason:
IO error: While lock file:
/opt/starrocks/be/datacache/datacache/meta/LOCK:
Resource temporarily unavailable

Fail to open RocksDB
path:/opt/starrocks/be/datacache/meta

terminate called without an active exception
...
starcache::StarCacheImpl::_load_all_from_meta_kv()
@Rocky Are you solving?
r
Yes, you must still define both 
hostPaths
and 
storage_root_path
. While Karpenter’s 
instanceStorePolicy: RAID0
handles the physical formatting and mounting of the NVMe drives to the EC2 host, Kubernetes does not automatically "pass" that host directory into your StarRocks Pods. You have to manually bridge that gap so the StarRocks BE process can see the high-speed local storage. 1. Why 
hostPaths
? Karpenter (using AL2023) with RAID0 mounts the combined local NVMe storage to a specific directory on the host—typically 
/mnt/k8s-disks/0
. The 
hostPaths
configuration in your Helm values maps this host directory into the StarRocks container. 2. Why 
storage_root_path
? Simply mounting the drive isn't enough; you must tell the StarRocks BE process to actually use that path for its Data Cache and storage. Recommended Values for 
r6gd
+ Karpenter (AL2023) In your 
values.yaml
for StarRocks 4.1.4, use the following configuration:
Copy code
yaml
starrocksBeSpec:
  # ... other resources ...
  
  # 1. Map the Karpenter-managed RAID0 mount into the Pod
  hostPaths:
    - name: local-nvme
      hostPath:
        path: /mnt/k8s-disks/0  # Standard RAID0 mount point for Karpenter on AL2023
      mountPath: /opt/starrocks/be/datacache # Internal path in the container

  config: |
    # 2. Tell StarRocks to use the mounted path
    # Syntax: <path1>;<path2>,medium:ssd
    # Note: We keep the default storage for logs/metadata and use NVMe for cache/data.
    storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,medium:ssd
    
    # 4.1.x Data Cache Configuration
    datacache_enable = true
    datacache_disk_size = 400G  # r6gd.2xlarge has ~474GB; leave buffer for OS/logs
    datacache_mem_size = 4G
    mem_limit = 90%
Important Version Note (4.1.4) Since you are using 4.1.4, you should be aware of a few specific improvements and changes: * 
medium:ssd
: In 4.1, adding the 
,medium:ssd
suffix to your 
storage_root_path
is highly recommended. It informs the cost-based optimizer that this specific path is backed by high-performance storage, which influences how it plans I/O-intensive queries. * Adaptive Layout: 4.1 introduces adaptive tablet distribution. Using the local NVMe for the Data Cache is critical here, as the system will rely on fast disk access to manage shifting data distributions without hitting S3/Iceberg latency. Summary of Paths Component | Value 
hostPaths.path
 | 
/mnt/k8s-disks/0
(Host location) 
hostPaths.mountPath
 | 
/opt/starrocks/be/datacache
(Container location) 
storage_root_path
 | Must include 
/opt/starrocks/be/datacache,medium:ssd
Verification Tip: After the pods start, you can run 
kubectl exec -it <be-pod-name> -- df -h
to verify that 
/opt/starrocks/be/datacache
is indeed mounted and shows the expected ~400-500GB of NVMe space. References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/helm.md • quick_start/helm.md • release_notes/release-4.0.md • faq/Deploy_faq.md References • quick_start/helm.md • deployment/shared_data/feature-support-shared-data.md • knowledge/usage/storage/hdfs_storage_volume_extended.md
s
@Rocky I have be replica count set to 3, but now only first pod is up and running and other pods errors out with following error:
Copy code
Fail to open rocksdb, reason:
IO error: While lock file:
/opt/starrocks/be/datacache/datacache/meta/LOCK:
Resource temporarily unavailable

Fail to open RocksDB
path:/opt/starrocks/be/datacache/meta

terminate called without an active exception
...
starcache::StarCacheImpl::_load_all_from_meta_kv()
r
The error you are seeing (
Resource temporarily unavailable
 on the 
LOCK
 file) occurs because multiple StarRocks BE pods are being scheduled onto the same EKS node and are trying to use the exact same host directory (
/mnt/k8s-disks/0
) for their Data Cache metadata. Since RocksDB (the engine behind the Data Cache meta) requires an exclusive lock on its files, only the first pod to start succeeds. The others fail because they cannot acquire the lock. To solve this, you have two options: Isolation or Separation. Option 1: Isolate via Subdirectories (Recommended for Flexibility) You can use 
subPathExpr
to give each pod its own unique directory (named after the pod) inside the NVMe mount point. This allows multiple BEs to live on one large node without conflicting. Update your 
values.yaml
as follows:
Copy code
yaml
starrocksBeSpec:
  # ... other settings ...
  
  # Ensure the POD_NAME environment variable is available
  envVars:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name

  hostPaths:
    - name: local-nvme
      hostPath:
        path: /mnt/k8s-disks/0  # The root of the Karpenter RAID0 mount
      mountPath: /opt/starrocks/be/datacache
      # The Operator will use this to create /mnt/k8s-disks/0/be-0, etc.
      subPathExpr: $(POD_NAME) 

  config: |
    storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,medium:ssd
    datacache_enable = true
    datacache_disk_size = 400G
Option 2: Separate via Pod Anti-Affinity (Recommended for Performance) For production workloads on 
r6gd.2xlarge
, it is best practice to have one BE pod per node. This prevents CPU/Memory contention and naturally avoids the directory conflict because each pod will be on a different physical NVMe drive. Add this to your `values.yaml`:
Copy code
yaml
starrocksBeSpec:
  # ... other settings ...
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: "<http://starrocks.com/be|starrocks.com/be>" # Adjust this label based on your release name
                operator: In
                values:
                  - "true"
          topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
Why this is happening in 4.1.4: In StarRocks 4.0+, the StarCache (Data Cache) implementation uses RocksDB to manage its metadata locally on disk for high performance. Unlike older versions where cache might have been partially in-memory or less strict about locking, 4.1.4 strictly enforces this lock to prevent data corruption in the cache. Summary Checklist: 1. Check Pod Distribution: Run 
kubectl get pods -o wide
to see if your BE pods are all on the same node. 2. Apply `subPathExpr`: If you want to allow multiple pods per node. 3. Apply `podAntiAffinity`: If you want to ensure each pod gets its own dedicated instance/NVMe. 4. Clean Up: If a pod failed previously, it might have left a "stale" lock. You may need to manually clear the 
/mnt/k8s-disks/0/datacache
directory on the host if the pods still won't start after applying the fix. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • deployment/helm.md • quick_start/helm.md References • quick_start/helm.md • administration/management/BE_configuration.md • deployment/deploy_manually.md
The error 
IO error: While lock file: /opt/starrocks/be/datacache/datacache/meta/LOCK: Resource temporarily unavailable
occurs because multiple Backend (BE) or Compute (CN) pods are trying to use the same directory on the same physical host for their Data Cache metadata. Since you are using 
hostPath: /mnt/k8s-disks/0
, every pod scheduled on that node maps to the exact same physical folder. RocksDB (used for cache metadata) requires an exclusive lock on its files; when the second pod tries to start, it sees that Pod 1 already owns the lock and crashes. The Fix: Pod Anti-Affinity For 
r6gd.2xlarge
instances (8 vCPU, 64GB RAM), you should only run one BE pod per node to avoid resource contention and to solve this locking issue. Update your 
values.yaml
to include an 
affinity
block for both Backend and Compute nodes. This forces the Kubernetes scheduler to place each pod on a unique node.
Copy code
yaml
starrocks:
  starrocksBeSpec:
    # ... previous config ...
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: "<http://starrocks.com/be|starrocks.com/be>"
                  operator: In
                  values: ["true"]
            topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"

  starrocksCnSpec:
    # ... previous config ...
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: "<http://starrocks.com/cn|starrocks.com/cn>"
                  operator: In
                  values: ["true"]
            topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
Why this is happening with Karpenter: 1. Bin-Packing: Even though your BE pod requests almost the entire node, Karpenter or the Kubernetes scheduler might still try to place a CN pod or another BE replica on the same node if the available resources (CPU/RAM) appear sufficient after accounting for overhead. 2. Shared Local Path: Because 
hostPath
is a global reference to the host's filesystem, Pod-0 and Pod-1 will both try to write to 
/mnt/k8s-disks/0/meta/LOCK
. 3. Locks: RocksDB cannot share a data directory. One process = One directory. Advanced Fix: Unique Subdirectories (if you use very large instances) If you decide to use larger instances (like 
r6gd.16xlarge
) where you want to run multiple pods per node, you must make the path unique by using the Pod name. 1. Pass the Pod Name into the container. 2. Use the variable in your 
storage_root_path
.
Copy code
yaml
starrocksBeSpec:
  env:
    - name: POD_NAME
      valueFrom:
        fieldRef:
          fieldPath: metadata.name
  config: |
    # Adding $(POD_NAME) creates a unique folder for every replica on the host disk
    storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache/$(POD_NAME),medium:ssd
Rocky's Recommendation: Stick to Pod Anti-Affinity. It is the standard for production database deployments. It ensures high availability (one node failure only takes out one BE) and guarantees that each BE has exclusive access to the local NVMe throughput. References • data_source/data_cache_troubleshooting.md • quick_start/helm.md • administration/management/BE_configuration.md • deployment/helm.md • using_starrocks/caching/block_cache.md
s
@Rocky getting the same error again. I don't think 
subPathExpr
is a valid option in helm values cause I don't see this here as well: github.com/StarRocks/starrocks-kubernetes-operator/blob/…/values.yaml
r
You are absolutely correct—the 
subPathExpr
field is a standard Kubernetes Pod feature, but it is not currently exposed in the 
hostPaths
abstraction within the StarRocks Operator's 
values.yaml
. The 
Resource temporarily unavailable
error occurs because two or more pods (likely a BE and a CN, or two BE replicas) have been scheduled onto the same physical 
r6gd
host and are both trying to lock the same 
/mnt/k8s-disks/0
directory. To fix this for StarRocks 4.1.4 on Karpenter, you should use Hard Pod Anti-Affinity and Node Selectors. This ensures that each BE/CN gets its own dedicated NVMe drive on its own node. 1. Updated 
values.yaml
(Production Grade for Karpenter) This configuration prevents pods from clashing by forcing the scheduler to place only one BE or CN per node.
Copy code
yaml
starrocksBeSpec:
  # Ensure BEs don't land on the same node
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: "<http://starrocks.com/be|starrocks.com/be>"
                operator: In
                values: ["true"]
          topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
  
  # IMPORTANT: If you use both BE and CN, they must also avoid each other
  # to prevent them from clashing on the same hostPath
  nodeSelector:
    <http://topology.kubernetes.io/instance-type|topology.kubernetes.io/instance-type>: r6gd.2xlarge # Optional: Pin to r6gd

  hostPaths:
    - name: local-nvme
      hostPath:
        path: /mnt/k8s-disks/0
      mountPath: /opt/starrocks/be/datacache
  config: |
    # Path for metadata (gp3) and path for data cache (NVMe)
    storage_root_path = /opt/starrocks/be/storage;/opt/starrocks/be/datacache,medium:ssd
    datacache_enable = true
    datacache_disk_size = 400G
    datacache_mem_size = 4G

starrocksCnSpec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: "<http://starrocks.com/cn|starrocks.com/cn>" # CNs avoid other CNs
                operator: In
                values: ["true"]
              - key: "<http://starrocks.com/be|starrocks.com/be>" # CNs also avoid BEs to prevent path clash
                operator: In
                values: ["true"]
          topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
  hostPaths:
    - name: local-nvme-cn
      hostPath:
        path: /mnt/k8s-disks/0
      mountPath: /opt/starrocks/cn/datacache
  config: |
    storage_root_path = /opt/starrocks/cn/datacache,medium:ssd
    datacache_enable = true
    datacache_disk_size = 400G
2. Why this is happening even with Anti-Affinity? If you applied 
podAntiAffinity
and still see the error, check the following: 1. Old Lock Files: Because you are using 
hostPath
, the 
/mnt/k8s-disks/0/datacache/meta/LOCK
file is persistent. If a previous failed deployment left a lock file behind, the new pod might fail to start. _ Fix: You may need to manually clear the host directory on the node or use a 
pre-bootstrap
script in Karpenter to 
rm -rf /mnt/k8s-disks/0/_
on node start. 2. BE vs. CN Clash: By default, a BE and a CN can land on the same node. Since both use 
/mnt/k8s-disks/0
, they will fight for the lock. Use the "Inter-Component" anti-affinity shown above (adding 
<http://starrocks.com/be|starrocks.com/be>
to the CN's anti-affinity list). 3. Verification Commands Run these to see exactly where your pods are landing:
Copy code
bash
# Check if pods are sharing nodes
kubectl get pods -o wide -l <http://starrocks.com/be=true|starrocks.com/be=true>
kubectl get pods -o wide -l <http://starrocks.com/cn=true|starrocks.com/cn=true>

# Check the BE logs for the specific path it is trying to lock
kubectl logs <be-pod-name> | grep "meta/LOCK"
Recommendation for `r6gd.2xlarge`: Since an 
r6gd.2xlarge
has exactly 8 vCPUs and 64GB RAM, and your BE is requesting ~7 vCPUs and 58GB RAM, there is zero physical room for another StarRocks pod on that node anyway. Using Hard Anti-Affinity (
requiredDuringScheduling...
) is the correct architecture here. It forces Karpenter to spin up 3 separate nodes for your 3 BEs.
6 Views
Open in Slack
PreviousNext
Powered by