http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> - we are currently in process setti...
# questions-and-troubleshooting
v
@Rocky - we are currently in process setting up the starrocks cluster in aws ec2. we have done basic testing and got comfertable performance from the m8id.4xlarge type of instances. I want to know if we want better performance which type instance could be better option. we are using shared nothing architecture. our workload contains SAAS type of workload. we will be using the primarykey table where queries pattern will be high throughput queries mostly focues n filtering +joins ( ~60-70%) ..very few queries may be heavy and long running in 10's secs (~5%). What type instances will be best for our use case?
r
For your SAAS workload on AWS using a shared-nothing architecture and Primary Key (PK) tables, performance is typically constrained by three factors: CPU throughput (for high-frequency joins), Memory capacity/bandwidth (for PK indexes and join hash tables), and Disk I/O (for persistent indexes and data scans). Since you are already using 
m8id.4xlarge
(16 vCPU, 64 GiB RAM, local NVMe), you are on a high-performance modern generation. To get better performance for your specific mix of high-throughput filtering/joins (~70%) and occasional heavy queries, here are the recommended instance types: 1. Memory-Optimized: 
r8id
or 
r7id
(Highly Recommended) Primary Key tables in StarRocks are memory-intensive. Even with the Persistent Index enabled, StarRocks caches a portion of the index in memory. * Why it’s better: The 
r
family provides a 1:8 vCPU-to-RAM ratio (e.g., 
r8id.4xlarge
has 128 GiB RAM vs. your current 64 GiB). * The Benefit: More memory allows for larger Join Hash Tables and more room for the PK index, which directly reduces "index miss" latency and speeds up the 60-70% of queries involving joins. The 
d
suffix ensures you keep the local NVMe storage required for shared-nothing performance. 2. Storage-Optimized: 
i4i
Series If your testing shows that disk I/O or "Compaction Score" (common in high-throughput PK table workloads) is the bottleneck, the 
i4i
family is the gold standard for StarRocks BE nodes. * Why it’s better: These instances use AWS Nitro SSDs, which provide lower latency and higher IOPS than the general-purpose NVMe found in 
m
or 
r
types. * The Benefit: It accelerates the Persistent Index (if 
enable_persistent_index=true
) and handles heavy background compaction of PK tables more efficiently, ensuring that data freshness doesn't impact query latency. 3. Compute-Optimized: 
c8id
or 
c7id
(Use with caution) * Why it’s better: If your bottlenecks are strictly CPU-bound (e.g., complex filtering on very large datasets) and your PK index is relatively small. * The Benefit: Provides the highest per-core performance. * Risk: The 1:2 vCPU-to-RAM ratio is often too low for StarRocks PK tables, potentially leading to OOM (Out of Memory) issues during heavy joins or high-concurrency loads. Performance Tuning Tips for your Use Case: * Persistent Index: Ensure you are using 
enable_persistent_index = true
. Since you have local NVMe (the 
d
suffix), this will offload the PK index from RAM to disk while maintaining sub-millisecond lookups. * Colocated Joins: For your "Filtering + Joins" workload, if you often join on the same keys (e.g., 
tenant_id
), use Colocation Groups. This keeps related data on the same BE node, eliminating the "Shuffle" overhead and significantly improving throughput for SAAS patterns. * Compaction Tuning: For high-throughput SAAS writes, you may need to increase the compaction threads in `be.conf`:
Copy code
properties
    update_compaction_num_threads_per_disk = 2 (or more depending on vCPU count)
Summary Recommendation: If you want better performance for Joins, move to 
r8id
 (Memory-Optimized). If the performance bottleneck is data ingestion/compaction or disk-based index lookups, move to 
i4i
 (Storage-Optimized). References • knowledge/usage/best-practice.md • best_practices/query_tuning/query_hint.md • best_practices/overview.md • best_practices/query_tuning/query_planning.md • best_practices/query_tuning/schema_tuning.md
v
@Rocky, as this is first time setup which we are yet to start we have tested only with m series as it was recommended balanced sweet spot and use cases are also new what should be recommendation. Should we first start with m series and see bottlenecks with production data / query pattern and based on bottlenecks phased change the instance type?
r
Yes, starting with the M series (
m8id.4xlarge
) to establish a baseline is a sound strategy, especially since you have already found the performance "comfortable." However, because you are using a shared-nothing architecture with local NVMe storage (
d
suffix), a "phased change" later is not a simple click-of-a-button in AWS; it requires a data rebalancing process. Here is the recommended path for your first setup: 1. The Migration Caveat (Shared-Nothing + Local NVMe) Because 
m8id
instances use Instance Store (local NVMe), you cannot simply stop the instance and change the type to 
r8id
. If you stop a 
d
-type instance, the data on the local disk is lost. * The Upgrade Path: To change instance types later, you will need to add new BE nodes of the new type (e.g., 
r8id
) to the cluster, wait for StarRocks to automatically rebalance the data (tablet migration), and then decommission the old 
m8id
nodes. * Impact: This consumes network bandwidth and IOPS during the transition. If your dataset is multi-terabyte, this could take hours or days. 2. Why "M" is a good start, but "R" is likely the destination Your workload (60-70% filtering + joins on PK tables) is heavily dependent on two things that consume RAM: * Primary Key Index: Even with the 
persistent_index
enabled, StarRocks performs better with more RAM for index caching. * Join Hash Tables: High-throughput joins create many hash tables in memory. The 
m8id.4xlarge
provides 4GB of RAM per vCPU (64GB total). The 
r8id.4xlarge
provides 8GB of RAM per vCPU (128GB total). * Recommendation: If you find during your initial production phase that BE nodes are hitting 
mem_limit
(check 
<http://be.INFO|be.INFO>
logs or Grafana for "OOM" or "Memory exceed limit" errors), that is your signal to move to the R series. 3. Monitoring for Bottlenecks During your initial run on the 
m8id
series, watch these specific metrics to decide if you need to switch: * Memory Usage (High Priority): If your memory usage is consistently >70% or you see query cancellations due to memory limits, move to 
r8id
. * Compaction Score: If the "Compaction Score" remains high and write latency increases, your local NVMe might be the bottleneck. In this case, consider the I4i series, which has even higher-performing storage. * CPU Idle: If CPU is consistently <10% idle but memory is plenty, you might need more nodes (scale out) rather than bigger nodes (scale up). Final Recommendation for Your Phase 1 1. Proceed with 
m8id.4xlarge
 for your initial production launch. It is a modern, high-performance instance that meets the StarRocks baseline recommendation (16 vCPU / 64GB RAM). 2. Enable Persistent Index: On your PK tables, ensure 
enable_persistent_index = true
is set. This is the most effective way to keep your memory footprint low on the M series. 3. Plan for Scale-Out: In StarRocks shared-nothing architecture, it is often better to add more "medium" nodes (like your current 4xlarge) rather than fewer "massive" nodes. This provides better data distribution and parallel processing power for your 70% join workload. If your budget allows for a ~15-20% higher cost, starting directly with 
r8id.4xlarge
provides more "headroom" for those 5% heavy queries and prevents a complex migration later. References • deployment/plan_cluster.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md • deployment/deployment_prerequisites.md References • best_practices/primarykey_table.md
Starting with the M series (
m8id.4xlarge
) is a sensible "middle ground" for a baseline, but there are specific architectural and operational reasons why you might want to consider the R series (
r8id.4xlarge
) immediately or be very cautious about a "phased" approach later. Here is a breakdown of how to decide: 1. The "Phased Change" Challenge (Shared-Nothing + Local NVMe) In a shared-nothing architecture using the 
d
series (like 
m8id
or 
r8id
), your data is stored on Local Instance Store (NVMe). * The Problem: Unlike EBS-backed instances, you cannot simply stop an 
m8id
, change the instance type to 
r8id
, and start it again. Stopping a local-NVMe instance results in total data loss on that drive. * The Migration Path: To change instance types later, you must: 1. Add new 
r8id
nodes to the cluster. 2. Decommission the old 
m8id
nodes one by one. 3. Wait for StarRocks to perform a full data rebalance (shuffling terabytes of data across the network) to move data from the old nodes to the new ones. * Decision: If your initial data volume is small (e.g., < 1TB), a phased change is easy. If you expect to grow to 10TB+ quickly, the "phased change" becomes a significant operational project that consumes network bandwidth and IOPS. 2. Why the R Series is often the "Real" Sweet Spot for SAAS While the M series is balanced, SAAS workloads on StarRocks Primary Key tables almost always hit Memory bottlenecks before CPU bottlenecks for two reasons: * Primary Key Index: Even with the 
persistent_index
enabled (which offloads the index to local NVMe), StarRocks still requires memory for the "Index Cache" and "L0 Segment" metadata. More RAM directly speeds up the "Filtering" part of your query pattern. * Join Throughput: You mentioned 60-70% of your queries involve joins. StarRocks uses Hash Joins, which build hash tables in memory. If you have many concurrent queries (high throughput), the 128GB RAM in an 
r8id.4xlarge
provides significantly more "headroom" to prevent OOM (Out of Memory) errors compared to the 64GB in the 
m8id
. 3. Comparison for your Workload Feature | M8id (Current) | R8id (Recommended) | Why it matters for you vCPU:RAM Ratio | 1:4 (64GB) | 1:8 (128GB) | Better for high-concurrency joins & PK index. Price | Base | ~15-20% higher | Small cost increase for 2x memory. SAAS Fit | Good for testing | Best for production | Avoids scaling pains as tenant count grows. Recommendation 1. If budget is tight and data is small: Stick with 
m8id.4xlarge
. It is a great modern instance. Just ensure you monitor 
mem_limit
and "Compaction Score" closely during your initial production run. 2. If you expect rapid growth or high concurrency: Start with 
r8id.4xlarge
 now. The cost difference is usually outweighed by the performance stability and the fact that you won't have to perform a complex data migration 3 months from now. 3. Critical Setting: For either choice, since you have local NVMe, ensure you set: * 
enable_persistent_index = true
(Mandatory for PK tables on local NVMe). * 
persistent_index_type = LOCAL
(Standard for shared-nothing). Summary: Your "phased approach" is safer if you use EBS volumes, but since you are using the high-performance local NVMe path, it is better to "over-provision" slightly on memory (R series) from day one to avoid the data rebalancing overhead of moving data later. References • administration/data_migration_tool.md • faq/operation_maintenance_faq.md • deployment/upgrade.md • introduction/Architecture.md
v
@Rocky, what about I instances?
r
Sorry, I encountered an error while processing your request: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}
Sorry, I encountered an error while processing your request: 503 UNAVAILABLE. {'error': {'code': 503, 'message': 'This model is currently experiencing high demand. Spikes in demand are usually temporary. Please try again later.', 'status': 'UNAVAILABLE'}}
3 Views
Open in Slack
PreviousNext
Powered by