@Rocky We are planning to run StarRocks on AWS and would like your recommendation on the deployment architecture. In our test environment we deployed FE + BE all-in-one on a single EC2 instance. For production, we are now moving to EKS + EC2 worker nodes with the StarRocks Operator for EKS. Our workload: data in Iceberg format on S3 is aggregated into materialized views, and queries are run against those MVs. We expect around 300 concurrent client connections, with concurrent access as the normal pattern. Workload details: • MV refresh frequency: once per hour • Total MV data size: approximately 10 GB • Latency requirement: P95 under 2 seconds Given this, we are unsure where the MVs should be stored. Previously they were placed on EBS. Should we switch to a shared-data (FE + CN) architecture and store them on S3, or should we keep them on node-local storage on the pods? Which of these is the better choice architecturally? If you recommend the shared-data architecture, we would also like guidance on the appropriate local disk cache configuration for the CN nodes at this data size and latency target.

For your production deployment on EKS, a shared-data (FE + CN) architecture is the recommended choice architecturally. This architecture aligns with cloud-native principles by separating storage (S3) from compute (CN), making it easier to scale your 300 concurrent client connections independently of your data size. Architecture Recommendation: Shared-Data (FE + CN) In this setup, your Materialized Views (MVs) will be stored as StarRocks native tables on Amazon S3, rather than being bound to local EBS volumes on specific pods. * Elasticity: Since the 10 GB of MV data resides on S3, you can scale the number of CN pods up or down instantly in response to concurrency spikes without needing to rebalance or shuffle data between nodes. * Cost & Reliability: S3 provides higher durability and lower cost compared to EBS for long-term storage. * Performance: With Data Cache enabled, StarRocks caches hot data from S3 onto the local storage of the CN nodes. For a 10 GB dataset, the entire MV will likely reside in the local cache after the first access or a manual warmup, providing performance comparable to local EBS storage. Guidance for Local Disk Cache Configuration Since your total MV data size is small (10 GB) and your latency target is sub-2 seconds, the cache configuration is straightforward. 1. Disk Selection To meet a P95 < 2s requirement with 300 concurrent connections, the speed of the cache disk is critical. * Recommendation: Use EC2 instance types with local NVMe SSDs (e.g.,

c5d

,

m5d

, or

r5d

series). Local NVMe provides the lowest latency and highest IOPS for cache hits. * Alternative: If using standard instances, use EBS gp3 volumes. Ensure you provision enough IOPS to handle the concurrent read load. 2. Cache Size Even though your data is 10 GB, you should over-provision the cache to account for metadata, multiple versions of data during refreshes (hourly), and potential growth. * Recommended Cache Size: 50 GB - 100 GB per CN node. This ensures the entire 10 GB working set is "pinned" in the fast local tier. 3. CN Configuration (

cn.conf

) Ensure the following parameters are set in your CN configuration (or via the StarRocks Operator's

config

section):

properties
# Enable the Data Cache (enabled by default in v3.2.3+)
starlet_use_star_cache = true

# Path where the cache will be stored (point this to your NVMe/SSD mount)
storage_root_path = /opt/starrocks/storage

# Set cache size: use 80% of the allocated disk for caching
starlet_star_cache_disk_size_percent = 80

4. Proactive Warmup Because your MV refreshes hourly, the "new" data starts on S3. To ensure the first queries after a refresh hit the P95 target, you can proactively warm up the cache:

sql
-- Manually trigger a cache warmup for the MV
CACHE SELECT * FROM your_mv_name;

Concurrency Considerations 300 concurrent connections is a standard workload for StarRocks. To optimize for this: * FE Scaling: Ensure you have at least 3 FE nodes for high availability and to handle connection overhead. * Variable Tuning: Set

max_user_connections

high enough (e.g.,

ALTER USER 'your_user' SET PROPERTIES ("max_user_connections" = "1000");

) and consider setting

query_queue_concurrency_limit

if you want to prevent resource exhaustion during extreme peaks. References • faq/Others.md • knowledge/usage/best-practice.md • faq/Sql_faq.md • best_practices/audit_log_resource_group.md • deployment/environment_configurations.md • administration/management/resource_management/Query_management.md

@Rocky Are there any performance benchmarking test results? I would like to compare the results between internal disk and object storage + cache

Yes, StarRocks has conducted official benchmarks comparing the Shared-nothing architecture (local disk) and the Shared-data architecture (Object Storage + Data Cache). The general conclusion is that once the Data Cache is warmed up, the shared-data architecture provides performance that is comparable to, and in some cases faster than, the traditional shared-nothing architecture. Benchmark Results: SSB 1TB Dataset In a test using StarRocks v3.3.0 on Alibaba Cloud (1 FE + 5 CN/BE nodes), the total execution time for 13 SSB queries was measured across different configurations: Architecture | Total Latency (ms) | Performance Comparison Shared-nothing (Local Disk) | 100,406 ms | Baseline Shared-data (S3 + Data Cache) | 83,444 ms | ~1.2x Faster than local disk Shared-data (S3 - No Cache) | 238,412 ms | ~2.4x Slower than local disk Key Takeaway: The shared-data architecture with Data Cache enabled actually outperformed the local-disk-only architecture. This is attributed to architectural optimizations in the shared-data engine, such as Parallel Scan and I/O Merge, which are highly optimized for object storage workflows. ────────── Benchmark Results: TPC-DS 1TB Dataset A separate comparison using the TPC-DS 1TB dataset showed near-parity between the two architectures for hot data: * Shared-nothing (Local Disk): 423 seconds * Shared-data (S3 + Cache Hit): 428 seconds * Shared-data (S3 - Cold Data): 668 seconds Key Takeaway: In this complex join-heavy workload, the performance difference between local disk and object storage + cache was less than 2%. Even in the "worst-case" scenario (all queries hitting cold data on S3), performance was still within a reasonable range (~1.5x slower) due to prefetching. ────────── Why Shared-Data matches Local Disk performance You might wonder how reading from S3 + Cache can be faster than reading from a local disk. 1. Cache Efficiency: The Data Cache in StarRocks (since v3.2.3) uses a block-based mechanism (MB-level) rather than file-based. This allows it to load only the specific portions of a column needed for a query. 2. I/O Parallelism: Shared-data clusters are designed to handle high-latency storage by using massive I/O parallelism. When this logic is applied to a local NVMe-based cache, it often results in higher throughput than traditional local disk reads. 3. Local NVMe Advantage: Since your EKS deployment will likely use Local NVMe SSDs for the cache, the "Cache Hit" scenario is physically reading from the same (or faster) hardware as a local-disk BE would. Recommendation for your Latency Target (P95 < 2s) Given your 10 GB dataset and your latency requirement, the Shared-data + Cache approach is highly recommended. * Initial Query: The first time an MV is queried after its hourly refresh, it may take 3–5 seconds to pull data from S3. * Subsequent Queries: Once cached on your CN nodes' local SSDs, you can expect sub-second response times for a 10 GB dataset, easily meeting your 2-second P95 target. * Warmup Strategy: You can eliminate the "first query penalty" by running

CACHE SELECT * FROM mv_name

immediately after the hourly refresh completes. References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • knowledge/usage/load/shared_data_batch_ingestion.md References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • knowledge/usage/load/shared_data_batch_ingestion.md

@Rocky Based on the information you provided, I assume that cn with data cache is suitable for my use case. Can you provide the recommeded ec2 instance types and classes that matches for me? My plans for the fisrst cut is 3 FE, 3CN.

For your production workload on EKS with 300 concurrent connections and a requirement for sub-2 second P95 latency, the following EC2 instance types are recommended. The key for the CN (Compute Nodes) is to use the "d" series instances, which include local NVMe SSDs. These are required for the StarRocks Data Cache to provide the performance you need for your S3-backed Materialized Views. 1. CN Nodes (Compute & Cache) Recommendation:

m6id.4xlarge

* Specs: 16 vCPU, 64 GB RAM, 1 x 950 GB NVMe SSD. * Why: * CPU: 300 concurrent connections (distributed as ~100 per CN) require significant vCPU to handle query planning execution and context switching. 16 vCPUs is the recommended production baseline for high-concurrency BE/CN nodes. * RAM: 64 GB allows for sufficient overhead for query execution and metadata management. * Storage: The local NVMe SSD is the most critical part. It will store the 10 GB Data Cache for your MVs. Local NVMe latency is orders of magnitude lower than EBS, which is essential for hitting your 2s P95 target. * Alternative: If your queries are computationally light, you could use

c6id.4xlarge

(16 vCPU, 32 GB RAM), but

m6id

is safer for production memory overhead. 2. FE Nodes (Management & Planning) Recommendation:

m6i.2xlarge

* Specs: 8 vCPU, 32 GB RAM. * Why: * High Concurrency: 300 concurrent users will put pressure on the FE for connection management and query parsing. 8 vCPUs is the standard recommendation for production FEs. * Metadata: StarRocks stores metadata in memory. While your 10 GB MV is small, 32 GB of RAM provides plenty of room for growth as you add more MVs or tables. * Disk: FE metadata logs can be stored on standard EBS (gp3) volumes, as they are not in the query data path. ────────── Comparison Summary Node Type | Recommended EC2 Instance | Role in your Architecture FE (x3) |

m6i.2xlarge

| Manages 300 connections and query planning. CN (x3) |

m6id.4xlarge

| Executes queries and hosts the Data Cache on local NVMe. EKS Deployment Notes * NVMe Mounting: When using the StarRocks Operator on EKS, ensure you use a Local Static Provisioner or a similar method to mount the EC2 instance's local NVMe SSD to the CN pods at the path specified in

storage_root_path

. * Networking: Use the Amazon VPC CNI and ensure your CN nodes are in the same placement group or availability zones as your FEs to minimize inter-node latency. * Warmup: Since you refresh hourly, your first query after a refresh will fetch data from S3. To maintain your P95 < 2s, automate a

CACHE SELECT

or a dummy query against the MV immediately after the refresh finishes. References • deployment/plan_cluster.md • deployment/environment_configurations.md • faq/Deploy_faq.md • administration/management/BE_configuration.md • deployment/deploy_manually.md • introduction/Architecture.md • quick_start/shared-data.md • deployment/plan_cluster.md • deployment/environment_configurations.md • faq/Deploy_faq.md • administration/management/BE_configuration.md • deployment/deploy_manually.md • introduction/Architecture.md