@Rocky We are operating Starrocks on AWS EKS, we are now investigating machine type options with Graviton. Based on current knowledge and experience, what machine type is recommended and why?

StarRocks now officially supports the ARM64 architecture, and based on recent performance benchmarks and community best practices, AWS Graviton3 (7g series) instances are currently the recommended choice for operating StarRocks on EKS. Recommended Machine Types Role | Recommended Instance Family | Recommended Sizing (Min) FE (Front-End) | m7g (General Purpose) |

m7g.2xlarge

(8 vCPU, 32GB) BE (Back-End) | c7g (Compute) or r7g (Memory) |

c7g.4xlarge

/

r7g.4xlarge

(16 vCPU, 64GB+) CN (Compute Node) | c7g (Compute Optimized) |

c7g.4xlarge

or larger Note: For production, BE/CN nodes should aim for at least 16 vCPUs to effectively utilize StarRocks' vectorized execution engine. Why Graviton3 (7g) is Recommended 1. Price-Performance Boost: Official StarRocks benchmarks (TPC-DS 100G) demonstrate that Graviton3 (C7g) can deliver up to a 50% price-performance improvement compared to Intel-based instances (C6i). This is driven by both lower hourly instance costs and ARM-specific instruction set optimizations. 2. Architectural Efficiency: Graviton processors provide a dedicated physical CPU core for every vCPU (unlike x86 SMT/Hyper-threading). This results in more predictable performance for the CPU-intensive SIMD (Single Instruction, Multiple Data) operations that StarRocks relies on for its vectorization. 3. Modern Memory Standards: Graviton3 instances utilize DDR5 memory, providing higher bandwidth and lower latency, which significantly benefits StarRocks' columnar scanning and large-scale data joins. 4. Maturation over Graviton4: While Graviton4 (8g series) offers even higher raw performance, current guidance suggests prioritizing Graviton3 as the most stable and validated platform for StarRocks production workloads while optimizations for Graviton4 continue to mature. Key Considerations for EKS * Multi-Arch Images: Ensure you are using the ARM64-compatible StarRocks Docker images. StarRocks provides official images for both

amd64

and

arm64

. * Karpenter/Node Groups: If using Karpenter or EKS Managed Node Groups, ensure your node templates specifically select the

arm64

architecture. * Avoid Mixed Clusters: It is strongly recommended not to mix x86 and ARM nodes within the same StarRocks cluster. Hardware heterogeneity can lead to inconsistent query performance and potential stability issues. * Storage: Regardless of the CPU, use EBS (gp3) with high IOPS/throughput or i4g (NVMe-backed Graviton) instances if your workload is extremely I/O intensive or requires Primary Key tables with persistent indexes. References • deployment/plan_cluster.md • deployment/deployment_prerequisites.md • faq/Deploy_faq.md • faq/Others.md • quick_start/helm.md

@Rocky why not use Graviton4 with M8g?

While Graviton4 (8g series) and the m8g family are excellent choices, the recommendation for Graviton3 (7g) was based on its status as the "proven" baseline for the current StarRocks 3.3/3.4 optimization cycle. However, you can and should consider Graviton4 if it is available in your region. Here is the breakdown of why you might choose one over the other: 1. Performance vs. Cost * Performance: Graviton4 (Neoverse V2) offers roughly 30% better compute performance and 75% more memory bandwidth than Graviton3. For a vectorized engine like StarRocks, the increased memory bandwidth is particularly beneficial for large scans. * Cost: Graviton4 instances typically have a slightly higher hourly cost than Graviton3. While the price-performance ratio is often better on Graviton4, some users prefer Graviton3 for a lower absolute entry price in smaller clusters. 2. Why "C" or "R" over "M" for BE Nodes? While m8g (General Purpose) is a solid all-rounder, it is rarely the optimal choice for StarRocks Back-end (BE) or Compute Nodes (CN): * Memory-to-vCPU Ratio: StarRocks BEs are memory-hungry due to large-scale joins and caching. * m8g provides 4GB RAM per vCPU. * r8g (Memory Optimized) provides 8GB RAM per vCPU. For complex OLAP queries, you frequently hit memory limits before CPU limits, making

r8g

more cost-efficient for the performance delivered. * Compute Intensity: If your workloads are dominated by simple aggregations or high-speed data ingestion, c8g (Compute Optimized) provides the most "vCPU for your dollar," though with only 2GB RAM per vCPU. * Best Role for M8g: We typically recommend m8g for the Front-end (FE) nodes, where the workload is a mix of metadata management, query parsing, and coordination, which fits the general-purpose profile perfectly. 3. StarRocks Version Support To get the most out of Graviton4, you should be on StarRocks v3.3.0 or later. This version introduced deep optimizations for the ARM NEON instruction set (specifically for bitshuffle, CRC, and filtering), which are essential for achieving the performance gains promised by the newer architecture. Summary Recommendation * Use m8g for: Front-end (FE) nodes. * Use r8g or c8g for: Back-end (BE) and Compute Nodes (CN), depending on whether your queries are more memory-constrained or CPU-constrained. * Stick with 7g (Graviton3) if: You are in a region where 8g is not yet available or if you have existing Reserved Instances you wish to utilize. If you are starting fresh on EKS and Graviton4 is available, r8g.4xlarge nodes for your BEs will likely provide the best overall experience. References • faq/Deploy_faq.md • developers/mac-compile-run-test.md • deployment/deployment_prerequisites.md • release_notes/release-3.3.md • faq/Others.md • introduction/what_is_starrocks.md

@Rocky On which cases we scale out FE's and on which we scale up?

In StarRocks, Front-End (FE) nodes handle metadata management and query planning. Deciding whether to scale up (more resources per node) or out (more nodes) depends on whether you are hitting limits in metadata volume or query concurrency. 1. When to Scale UP (Vertical Scaling) Scaling up involves increasing the CPU and, most importantly, the RAM (JVM Heap) of your FE nodes. All FEs (Leader, Followers, and Observers) keep a full copy of the metadata in memory, so they must all be scaled up together. * Case: Growing Metadata (Primary Driver) StarRocks stores metadata for all tables, partitions, and tablets in the FE's JVM heap. As your data grows in terms of schema complexity, you must scale up the memory. Use the following official guidelines based on the number of tablets (replicas) in your cluster: * < 1 million tablets: 16 GB RAM * 1–2 million tablets: 32 GB RAM * 2–5 million tablets: 64 GB RAM * 5–10 million tablets: 128 GB RAM * Case: Frequent Full GCs or OOM Crashes If you see

java.lang.OutOfMemoryError

in your

fe.log

or frequent "Leader transfer" events caused by long Stop-The-World (STW) Garbage Collection pauses, you need to increase the

-Xmx

(Max Heap) and the instance's physical RAM. * Case: Large-Scale Query Planning If your queries involve thousands of partitions (e.g., querying years of daily-partitioned data in one go), the FE needs more CPU and memory to generate the execution plan. 2. When to Scale OUT (Horizontal Scaling) Scaling out involves adding more FE nodes. There are two types of nodes you can add: Followers and Observers. * Case: High Query Concurrency (Add Observers) If your cluster faces high QPS (queries per second), the Leader and Followers might become a bottleneck for query parsing and planning. * Action: Add Observer nodes. Observers can handle query requests but do not participate in the Raft voting process, meaning they increase read capacity without slowing down metadata writes. * Case: High Availability (Add Followers) * Action: Ensure you have exactly 3 Follower nodes (1 Leader + 2 Followers). This is the standard production setup to survive the failure of one node. Adding more than 3 followers (e.g., 5) is rarely necessary and can actually slow down metadata synchronization due to the increased overhead of the Raft consensus. * Case: Infrastructure Migration If you are moving to a new instance type (e.g., moving from x86 to Graviton), the safest path is to scale out by adding the new ARM nodes as Followers/Observers and then scaling in the old x86 nodes. Summary Table Requirement | Scaling Direction | Node Type Too many tablets/partitions | Scale Up (Memory) | All FEs High Query QPS | Scale Out | Observers Metadata Write Latency | Scale Up (CPU/IO) | Leader/Followers Fault Tolerance | Scale Out (to 3) | Followers Pro-tip for EKS: Since you are on EKS, use a Vertical Pod Autoscaler (VPA) for FE memory recommendations, but avoid using Horizontal Pod Autoscalers (HPA) for FEs, as adding/removing FE nodes requires manual SQL commands (

ALTER SYSTEM ADD/DROP

) to update the metadata correctly. References • administration/management/Scale_up_down.md • deployment/plan_cluster.md • faq/Others.md • introduction/Architecture.md • knowledge/trouble_shooting/fe_memory_problems.md References • knowledge/trouble_shooting/fe_memory_problems.md • faq/Others.md

@Rocky same question with CN's, when do we scale out and when do we scale up?

For Compute Nodes (CN), which are used in StarRocks' shared-data (cloud-native) architecture, the trade-off between scaling up and scaling out is primarily driven by query complexity versus total throughput. Since CNs are stateless and do not require data rebalancing, you have much more flexibility compared to FE or BE nodes. 1. When to Scale OUT (Horizontal Scaling) Scaling out (adding more CN pods) is the most common approach in EKS and is generally the "default" for elasticity. * Case: High Query Concurrency (QPS) If you have many users running independent queries simultaneously, scaling out is more efficient. Each CN can handle a portion of the incoming query volume, reducing queuing at the compute layer. * Case: Total Cache Size Requirements In shared-data mode, CNs use local storage (EBS or Instance Store) to cache data from S3. * Logic: If your "hot" dataset is 10TB and each CN has a 1TB cache, you need at least 10 CNs to keep the data local. Scaling out increases the aggregate cache capacity of the cluster. * Case: Blast Radius & Availability In EKS, nodes can be reclaimed (Spot instances) or fail. Having 10 medium-sized CNs is safer than having 2 massive CNs; a single node failure in a 10-node cluster only loses 10% of your compute and cache. * Case: Rapid Elasticity On EKS, it is faster to spin up 5 new

c7g.2xlarge

pods than it is to migrate a massive workload to a

c7g.16xlarge

. 2. When to Scale UP (Vertical Scaling) Scaling up (moving to larger instance types like

r7g.8xlarge

or

r7g.16xlarge

) is necessary when individual query performance hits a wall. * Case: Large Memory-Intensive Joins If a single query involves joining two massive tables that don't fit into the memory of a smaller node, StarRocks may have to "spill" to disk, significantly slowing down the query. * Action: Scale up to r7g instances (8GB RAM per vCPU) to keep the shuffle and join operations in-memory. * Case: Intra-query Parallelism StarRocks can parallelize a single query across all cores of a machine. If you have a single, massive "hero query" that must finish as fast as possible, a single large node is often faster than many small nodes due to reduced network overhead between fragments. * Case: Cache Hit Performance If you find that your cache hit rate is low because the "per-node" cache is too small for the specific partitions being scanned, a larger node with a larger attached EBS volume or NVMe disk can improve performance for that specific compute thread. Summary Table for CN Scaling Driver | *Scale OUT (More Pods)* | *Scale UP (Larger Pods)* Primary Metric | High Concurrency / QPS | Single Query Latency Resource Bottleneck | Total Cluster CPU/Cache | Single-node RAM (Joins/Spilling) EKS Strategy | Best for Spot & Karpenter | Best for "Power Users" / Big Data Ideal Instance |

c7g.2xlarge

or

4xlarge

|

r7g.8xlarge

or

16xlarge

EKS Recommendation: The "Sweet Spot" For StarRocks on EKS using Graviton, the "Sweet Spot" for a CN is typically the 4xlarge or 8xlarge size (e.g.,

c7g.4xlarge

or

r7g.8xlarge

). * Why? Instances smaller than 4xlarge often struggle with resource overhead and have limited network bandwidth. Instances larger than 16xlarge can lead to high "blast radius" if a pod is evicted and can cause complex JVM/OS contention issues. Pro-tip: If you use Karpenter on EKS, you can define a

NodePool

that allows for both. Karpenter can automatically scale out (add more nodes) during peak hours and can be configured to prefer larger instances if you find your queries are frequently hitting memory limits. References • administration/management/Scale_up_down.md • deployment/plan_cluster.md

@Rocky If we are using the starrocks kubernetes operator, when scaling out FE's - can we control how many followers and how many observers?

When using the StarRocks Kubernetes Operator, you currently cannot explicitly control the number of Followers vs. Observers directly within the

starRocksFeSpec

fields of the

StarRocksCluster

Custom Resource (CR). Here is how the scaling works under the hood with the Operator: 1. The

replicas

Field The only native control you have in the YAML is the

replicas

count:

yaml
spec:
  starRocksFeSpec:
    replicas: 3  # This is the total number of FE nodes

By default, the Operator treats all FE nodes defined in this spec as Followers (voting members of the Raft group). If you set this to 3, it creates a standard 3-node HA quorum (1 Leader + 2 Followers). 2. Scaling Beyond 3 Nodes If you scale the FE

replicas

to 4 or 5: * The Operator's Logic: The Operator typically registers new nodes as Followers by default. While you can have a 5-node Follower cluster, it is rarely recommended because adding more voting members increases the overhead of the Raft consensus protocol (metadata writes become slower). * The "Observer" Gap: As of current versions (v1.9/v1.10), the Operator does not have a separate

observerReplicas

field to automatically register nodes as Observers. 3. How to add Observers in Kubernetes Since the Operator doesn't natively support a mix of roles in one spec, you have two main options if you need Observers for high read concurrency: * Manual Registration (Recommended): 1. Increase the

replicas

in your

StarRocksCluster

YAML. 2. Once the new Pods are running, they may not be automatically joined to the cluster if the Operator logic only handles the initial 3. 3. Log in to the Leader FE and manually run:

sql
        ALTER SYSTEM ADD OBSERVER "fe-pod-hostname.svc:9010";

* Architectural Workaround: Use Compute Nodes (CN) for query scaling instead of adding FE Observers. In modern StarRocks architectures (especially on EKS), it is often more efficient to scale the CN layer for CPU-heavy query work, as the FE's main bottleneck is typically metadata memory (which scaling out Observers doesn't solve) or query parsing (which only needs Observers at extremely high QPS). Summary Recommendation * For High Availability: Set

replicas: 3

. This gives you a robust 1-Leader, 2-Follower setup managed entirely by the Operator. * For High Read Concurrency: Scale out your Compute Nodes (CN) first. If you specifically need FE Observers for SQL parsing throughput, you will likely need to register them manually via SQL, as the current Operator CRD treats the FE group as a unified set of Followers. References • deployment/sr_operator.md • administration/management/Scale_up_down.md • deployment/plan_cluster.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • deployment/plan_cluster.md • faq/Deploy_faq.md