@Rocky In StarRocks Shared Data architecture, is Scale-Out generally recommended over Scale-Up? I'm wondering whether continuously scaling out Compute Nodes (CNs) eventually leads to higher network overhead. Once the cluster reaches a certain number of CN replicas, wouldn't it be more efficient to scale up existing CNs instead of continuing to add more nodes?

In StarRocks' Shared Data (Compute-Storage Separation) architecture, whether to Scale-Out (add more nodes) or Scale-Up (add more resources to existing nodes) depends on your specific performance and operational goals. While StarRocks is built for high elasticity through scaling out, there is a technical trade-off involving network overhead and cache efficiency. 1. Scale-Out: The Primary Recommendation Scale-out is generally the default recommendation for Shared Data architecture because: * Elasticity: CN nodes are stateless. You can add or remove them in seconds without the data migration/rebalancing overhead required in Shared-Nothing clusters. * High Availability & Concurrency: More nodes provide better isolation and higher throughput for concurrent query workloads. * MPP Parallelism: StarRocks' Massively Parallel Processing (MPP) framework is designed to split queries into fragments that run across all available nodes, aiming for near-linear performance gains as you add compute. 2. When to Consider Scaling-Up Despite the benefits of scaling out, continuously adding small nodes can lead to diminishing returns. You should consider Scaling-Up existing CNs in the following scenarios: * Reducing Network Overhead (Shuffle Cost): In complex queries involving Shuffles or Broadcast Joins, data must be transmitted over the network between nodes. As the number of nodes ($N$) increases, the potential network connections between them grow ($N^2$). Using fewer, "fatter" nodes (e.g., 32 cores instead of 8) reduces the total number of RPC calls and data serialization/deserialization cycles required for the same amount of compute power. * Improving Data Cache Efficiency: Each CN maintains a local disk cache for "hot" data. * If you have many small nodes, your hot data is fragmented across many small caches. * Fewer, larger nodes allow for larger, more consolidated local caches, which can improve cache hit rates for localized workloads. * Query Memory Limits: Some memory-intensive queries (like large hash joins) may fail if a single node doesn't have enough RAM. Scaling up a node's memory is often the only way to support these specific "heavy" queries. 3. Recommended Sizing & Limits * Node Specification: StarRocks generally recommends at least 16 CPU cores and 64 GB RAM per compute node for production environments. Scaling up to 32 or 64 cores per node is common before continuing to scale out horizontally. * Cluster Size: StarRocks can theoretically scale to 500+ nodes, but most production environments find a "sweet spot" by using mid-to-high-tier instances (e.g., 32-64 vCPUs) rather than hundreds of tiny instances to keep network complexity manageable. Summary * Scale-Out when you need more concurrency or want to take advantage of cost-effective elasticity (e.g., using Spot instances). * Scale-Up when your complex joins are bottlenecked by network shuffle time, or when individual queries require more memory/cache than your current node size provides. If you find that query latency is increasing despite adding nodes, check the Exchange Operator metrics in your Query Profile. High

NetworkTime

or

SerializeChunkTime

relative to CPU time is a strong signal that scaling up might be more efficient than further scaling out-efficient than scaling out. References • introduction/Architecture.md • deployment/plan_cluster.md • using_starrocks/caching/query_cache.md • data_source/data_cache.md