@Rocky Does StarRocks support deployment on ARM EC2 instances

Yes, StarRocks supports deployment on ARM-based EC2 instances (such as AWS Graviton). Starting from StarRocks v3.3.0, the system has been significantly optimized for the ARM architecture. Performance tests on AWS Graviton instances have shown ARM-based clusters to be up to 39% faster than x86 in certain benchmarks like Clickbench. Key Deployment Details: * Availability: While the standard binary packages (tarballs) on the official website are typically for x86_64, StarRocks provides ARM-compatible Docker images. * Manual Deployment on ARM: If you are not using Docker for the final deployment, you should pull the StarRocks Docker image on an ARM machine and extract the binaries.

bash
    # Example for v3.3.3 on Ubuntu
    docker pull starrocks/artifacts-ubuntu:3.3.3
    docker run --rm starrocks/artifacts-ubuntu:3.3.3 tar -cf - -C /release . | tar -xvf -

* Version Recommendation: For the best performance and compatibility on ARM, it is recommended to use v3.3.0 or later, as these versions include specific instruction-level optimizations for ARM. Support Summary: Feature | Status AWS Graviton (ARM) | Fully Supported & Optimized Minimum Version | Recommended v3.3+ for performance; v3.1.14+/v3.2.10+ for standard ARM Docker images Performance | Optimized for ARM instruction sets; benchmarks show parity or better than x86 in several workloads References • faq/Deploy_faq.md • integrations/authenticate_to_aws_resources.md • deployment/environment_configurations.md • release_notes/release-3.3.md

@Rocky So using docker on ARM machine is better option than x86_64 on raw EC2

Whether "better" means performance or ease of use, ARM (AWS Graviton) with StarRocks v3.3+ is generally superior to x86_64 in terms of price-to-performance. However, there is a nuance in how you deploy it: you typically use Docker to obtain the binaries, but you can still run them "raw" on the EC2 host. 1. Performance: ARM vs. x86_64 Starting with StarRocks v3.3.0, the engine includes instruction-level optimizations specifically for ARM. Internal benchmarks on AWS Graviton show significant leads over x86: * Clickbench: ~39% faster * TPC-DS: ~35% faster * SSB: ~11% faster Since AWS Graviton instances are typically cheaper than equivalent Intel/AMD instances, the cost-efficiency (price/performance) is substantially better on ARM. 2. Deployment: Docker vs. Raw EC2 There is a common misunderstanding that you must run inside a container to use ARM. In reality: * Distribution: StarRocks currently distributes ARM-compatible versions via Docker Images (

starrocks/artifacts-ubuntu

). Official

.tar.gz

downloads on the website are often x86-only. * Running "Raw": You can pull the Docker image, extract the binaries, and then run them directly on your raw EC2 instance (manual deployment). This avoids any (minimal) Docker networking overhead and fits into traditional systemd-based management. To get raw ARM binaries:

bash
# Pull the artifacts image for your version
docker pull starrocks/artifacts-ubuntu:3.3.3

# Extract the binaries to the current directory
docker run --rm starrocks/artifacts-ubuntu:3.3.3 tar -cf - -C /release . | tar -xvf -

Recommendation * Use ARM (v3.3+): If performance and cost are your primary drivers, choose AWS Graviton (ARM) instances. * Deployment Choice: * Use Docker/Kubernetes if your infrastructure is already containerized; the performance overhead is negligible on Linux. * Use Raw EC2 (Manual Deployment) if you prefer managing processes via

systemd

or want to squeeze out every bit of networking performance. You simply use the Docker method above to download the files first. Crucial Note: Ensure you use v3.3.0 or later to benefit from the ARM-specific performance optimizations. Older versions will run but won't show the same performance gains. References • deployment/prepare_deployment_files.md • release_notes/release-3.3.md • faq/Deploy_faq.md References • deployment/prepare_deployment_files.md • developers/build-starrocks/Build_in_docker.md • developers/mac-compile-run-test.md

@Rocky How do I verify hybrid row column storage working?

To verify that hybrid row-column storage is working in StarRocks, you can check the table properties and the query execution plan. 1. Verify Table Configuration First, ensure the table was created correctly with hybrid storage enabled. Hybrid storage is only supported for Primary Key tables. * Check Table Properties: Run

SHOW CREATE TABLE <table_name>;

and look for

"store_type" = "column_with_row"

in the

PROPERTIES

section. * Prerequisite Config: The FE configuration

enable_experimental_rowstore

must be set to

true

for this feature to be available:

sql
    ADMIN SET FRONTEND CONFIG ("enable_experimental_rowstore" = "true");

2. Verify Query Execution (The "Short Circuit") The primary benefit of hybrid storage is the "Short Circuit Scan," which allows StarRocks to bypass the columnar engine and fetch the entire row directly from the row store for point queries. 1. Enable Short Circuiting: Ensure the session variable is active:

sql
    SET enable_short_circuit = true;

2. Run EXPLAIN: Perform a point query (using all primary key columns in the

WHERE

clause with

=

or

IN

) and check the plan:

sql
    EXPLAIN SELECT * FROM users WHERE id = 1;

3. Look for the Flag: In the output, locate the

OlapScanNode

. If hybrid storage is being utilized correctly, you will see:

Short Circuit Scan: true

3. Key Indicators * Performance: You should notice significantly lower latency and higher concurrency for point lookups compared to standard columnar tables. * Storage: Hybrid tables will consume more disk space (often 2x or more) because data is stored in both columnar and row formats (the row data is stored in a hidden

__row

column). References • table_design/hybrid_table.md • administration/management/FE_configuration.md • sql-reference/sql-statements/prepared_statement.md References • table_design/hybrid_table.md • administration/management/FE_configuration.md

@Rocky My update rowset commit apply duration dropped by 2/3rds to on average 1 second after using row storage. Transaction publish latency also dropped to about 2s. Aside from data freshness improving by that margin, what other benefits are there? CPU BE busy util is still approximately the same. This costs more than double disk usage. If i dont intend on using point queries, are there any other benefits

The drop you're seeing in commit/apply duration and transaction publish latency is directly related to how StarRocks handles Primary Key (PK) table updates. While the "headline" feature of hybrid storage is point queries, it has a profound impact on the write path of Primary Key tables that explains your observed performance gains. 1. Faster "Row Assembly" during the Apply Phase In a standard columnar PK table, an update is a "Delete + Insert." When you perform a partial update (updating only some columns), StarRocks must perform Row Assembly: it reads the "old" values of the columns you didn't update from disk to reconstruct a full new row to write. * Without Row Store: StarRocks has to open and scan multiple columnar files (potentially dozens or hundreds) to pull the values for the missing columns. This is IO-intensive and slow. * With Row Store: StarRocks can fetch the entire old row from a single location (the

__row

column) in one contiguous read. This drastically reduces IOPS and the "Apply" time, which is why your publish latency dropped so significantly. 2. Efficiency in Partial Updates If your table has many columns (a "wide table"), the row store makes partial updates much more stable. Instead of the CPU and Disk IO scaling with the number of columns in the table, it scales mostly with the row size. This provides: * Reduced IO contention: Fewer file handles and disk seeks during the write-apply phase. * Faster Version Publishing: Since the "Apply" step (where the PK index is updated and delete vectors are generated) is faster, the data becomes "Visible" (published) much sooner. 3. Support for Server-Side Prepared Statements Hybrid storage is required for Server-Side Prepared Statements. * If you plan to use

PREPARE

and

EXECUTE

for your queries, StarRocks uses the row store to achieve sub-millisecond response times. Without the row store, prepared statements don't get the "short-circuit" optimization and are processed through the standard (slower) OLAP engine. 4. High-Concurrency Query Scaling Even if you don't do "point queries" (looking up a single ID), the row store helps with small-range scans (fetching a few rows). Because it uses less CPU to reconstruct these rows compared to the columnar engine (which has to stitch columns back together), your BEs can handle a much higher number of concurrent small queries before hitting a bottleneck. Is it worth the 2x Disk Cost? If you absolutely do not intend to use point queries or prepared statements, and your previous ingestion latency (3s vs 1s) was acceptable: * The benefit is stability: You have bought "headroom." As your table grows or your update frequency increases, the columnar-only Apply phase would eventually become a major bottleneck (high

Write_Amp

). The row store keeps this cost constant. * The cost is storage: You are trading disk space for write-path stability and data freshness. Summary of benefits beyond point queries: 1. Lower Write Amplification: Faster row reconstruction for partial updates. 2. Stable Ingestion: Publish latency remains low even as the number of columns in your table increases. 3. Prepared Statements: Required for high-performance server-side prepared statements. 4. Concurrency: Higher throughput for "fetch by ID" or "fetch small batch" style workloads. References • table_design/hybrid_table.md • table_design/table_types/primary_key_table.md • sql-reference/sql-statements/prepared_statement.md References • table_design/table_types/primary_key_table.md • best_practices/primarykey_table.md

@Rocky If I drop my table due to bugs I cannot solve and it was decided the best course of action was to restore them, what is the fastest way? My backups are currently in s3 (shared nothing cluster), backed by partition every hour. Do I have to manually restore by partition?

If you have just dropped the table and realized it was a mistake or need to undo it, you might not even need the S3 backups. 1. The Instant Way:

RECOVER

Before attempting to pull data from S3, check if the table is still in the StarRocks "Trash." When you drop a table, StarRocks keeps the metadata and data for a period (default is 1 day, controlled by

catalogtrashexpire_second

). * Command:

RECOVER TABLE table_name;

* Benefit: This is instantaneous and requires no data movement. ────────── 2. The Fastest Way via S3:

RESTORE

If the table is gone from the trash and you must use S3 backups, here is how to optimize the process: A. Do you have to restore partition by partition? * If the partitions are in ONE snapshot: No. You can restore the entire table in one command:

sql
    RESTORE SNAPSHOT snapshot_name
    FROM your_repo
    ON (TABLE your_table);

* If you have separate snapshots for each partition: Yes, you must trigger a

RESTORE

for each snapshot. However, StarRocks has a limitation: only one

RESTORE

job can run per database at a time. B. How to speed up the

RESTORE

To maximize speed when dealing with many snapshots: 1. Reduce Replication: Use

replication_num = 1

during the restore to avoid the overhead of the cluster copying data between nodes while it's being downloaded from S3.

sql
    RESTORE SNAPSHOT snapshot_name FROM your_repo
    ON (TABLE your_table)
    PROPERTIES ("replication_num" = "1", "backup_timestamp" = "...");

_After the restore is complete, run

ALTER TABLE your_table SET ("replication_num" = "3");

to let the cluster replicate data in the background._ 2. Parallelize via Temporary Databases: Since you can only run one restore per DB, you can restore different snapshots into temporary databases simultaneously. *

RESTORE ... DATABASE temp_db1 ...

*

RESTORE ... DATABASE temp_db2 ...

_ Once finished, you can use

INSERT INTO destination_db.table SELECT _ FROM temp_db1.table

or swap them. *3. Alternative: `INSERT INTO ... SELECT * FROM files()`* If your hourly "backups" are actually raw files (like Parquet or CSV) stored in S3 rather than StarRocks snapshots, the fastest way is often using the

files()

table function:

sql
INSERT INTO your_table
SELECT * FROM files(
    "path" = "<s3://your-bucket/backups/partition_date=2023-10-01/*>",
    "format" = "parquet",
    ...
);

This is often faster for bulk loading than the

RESTORE

command because it leverages the full compute power of all BE nodes in parallel without the metadata overhead of the snapshot system. Summary Recommendation 1. Try

RECOVER TABLE

first. 2. If using

RESTORE

, use

replication_num = 1

to save time. 3. If you have dozens of separate snapshots, script the restore to run sequentially, or parallelize across multiple temporary databases. References • sql-reference/sql-statements/backup_restore/RECOVER.md • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/RESTORE.md • sql-reference/sql-functions/table-functions/files.md References • sql-reference/sql-statements/backup_restore/RECOVER.md • sql-reference/sql-statements/backup_restore/RESTORE.md • administration/management/Backup_and_restore.md References

@Rocky What is the ideal AWS machine type for a shared nothing cluster? I have 12 BE nodes at r6in.8xlarge currently. Inserts around 60k QPS updates about 30k QPS, both batched appropriately such that stream loads occur roughly every 20-30 seconds. I am already maxing out EBS at 1000 750 throughput and 10k IOPS. Disk IO util is very high

For a shared-nothing cluster with high-frequency updates (30k QPS) and high disk I/O utilization, the EBS-only architecture is your primary bottleneck. While

r6in

instances have excellent network bandwidth, they are limited by the maximum throughput and IOPS a single EC2 instance can push to EBS. Your 1000 MB/s limit is likely the Amazon EC2 instance-level throughput limit for that size, meaning adding more IOPS to your EBS volume won't help. The Ideal Machine Type: AWS

i4i

Family For StarRocks BE nodes in high-I/O scenarios (especially Primary Key tables with Persistent Index and Row Store), the

i4i

(Intel) or

i3en

families are the gold standard. * Instance Recommendation:

i4i.8xlarge

(to match your current 32 vCPU / 256 GiB RAM). * Why it's better: * Local NVMe SSDs: You get

AWS Nitro SSDs

directly attached to the physical host. * I/O Performance: A single

i4i.8xlarge

provides roughly 3,750 MB/s read throughput and 2,500 MB/s write throughput with hundreds of thousands of IOPS. This is 3–4x your current EBS cap. * Lower Latency: Local NVMe latency is measured in microseconds (μs) vs. milliseconds (ms) for EBS. This drastically speeds up the Persistent Index lookups and Row Store reads during the "Apply" phase of your updates. Comparison Table Feature |

r6in.8xlarge

(Current) |

i4i.8xlarge

(Recommended) Storage Type | EBS Only (Remote) | Local NVMe SSD (Direct) Disk Throughput | ~1,000 MB/s (Max Instance) | ~3,750 MB/s Disk IOPS | 10k–16k (Typical) | Hundreds of thousands Memory | 256 GiB | 256 GiB Best For | High Network/Memory | High I/O / High Concurrency Updates Strategic Advice for your Bottleneck 1. Move to Local Storage: For a shared-nothing cluster, using EBS is essentially "shared-data performance with shared-nothing complexity." By switching to

i4i

instances, you eliminate the EBS network-storage bottleneck entirely. 2. Persistent Index on NVMe: Since you are using Primary Key tables, ensure the

Persistent Index

is enabled. On local NVMe, the random I/O required for the index to check for duplicates will be significantly faster than on EBS. 3. RAID 0 for Local Disks: If the instance provides multiple local NVMe drives (like

i3en

), use RAID 0 to strip them for maximum throughput. 4. Note on Data Safety: Because local NVMe is ephemeral (data is lost if the instance is terminated/fails), ensure you are running with 3 replicas (

replication_num = 3

). StarRocks will handle the background recovery if a node fails. Summary: Your

r6in

nodes are CPU/RAM rich but "I/O starved" for your write load. Moving to

i4i.8xlarge

will likely drop your Disk IO util from 90%+ to under 30% and significantly improve your transaction commit/apply stability. References • deployment/deployment_prerequisites.md • deployment/plan_cluster.md • best_practices/primarykey_table.md References • deployment/deployment_prerequisites.md • deployment/plan_cluster.md • best_practices/primarykey_table.md • introduction/Architecture.md • table_design/table_types/primary_key_table.md • table) • administration/management/monitoring/Monitor_and_Alert.md References • best_practices/primarykey_table.md

@Rocky What if I using local NVMe SSD is not an option? I dont want data to be ephemeral. If node goes down, it needs to be able to survive. Plus, the i4i machine caps me at 3750 GiB of NVM space, only way to get more is to get more ec2 machines.

If local NVMe is not an option due to its ephemeral nature and capacity limits, you can still significantly improve your performance by addressing the EBS instance-level vs. volume-level bottlenecks. Your

r6in.8xlarge

instances are actually capable of much more than you are currently getting. According to AWS specs, an

r6in.8xlarge

has a maximum EBS-optimized throughput of 3,125 MB/s and 100,000 IOPS. You are currently capped at ~1,000 MB/s and 10k IOPS because of your EBS volume configuration, not the machine itself. 1. Upgrade to

io2 Block Express

Standard

gp3

volumes max out at 1,000 MB/s throughput. If you need more from a single volume, switch to

io2 Block Express

. * Performance: A single

io2

volume can provide up to 4,000 MB/s and 256,000 IOPS. * StarRocks Benefit: This will immediately remove the I/O wait you are seeing without requiring more EC2 nodes. Since

io2

is persistent, your data survives instance stops/starts. 2. Multi-EBS Volume Striping (Shared-Nothing) StarRocks BEs can handle multiple storage paths. Instead of one large EBS volume, you can attach 4x

gp3

volumes (each provisioned at 750 MB/s and 25k IOPS) to each BE. * How to do it: In

be.conf

, set

storage_root_path = /data1;/data2;/data3;/data4

. * Result: StarRocks will strip data across these volumes, allowing you to hit the 3,125 MB/s limit of your

r6in

instance. This is often more cost-effective than a single large

io2

volume. 3. Move to Shared-Data Architecture (StarRocks 3.x) If your primary concern is that data "needs to be able to survive" and you don't want storage to be tied to EC2 count, the Shared-Data (Storage-Compute Separation) architecture is the architectural solution. * How it works: All persistent data is stored in S3 (non-ephemeral, unlimited). Your BEs (called CNs in this mode) use EBS or local SSDs strictly as a cache. * Durability: If a node goes down, zero data is lost. A new node can spin up, connect to S3, and start serving queries immediately. * Performance: For the 30k QPS update workload you mentioned, StarRocks 3.2+ includes "Batch Publish" optimizations specifically for S3 to keep latency low. * Index: You can use

persistent_index_type = CLOUD_NATIVE

, which stores the Primary Key index on S3, removing the local disk capacity constraint for your indexes. 4. BE Configuration for High-Throughput Storage When you move to higher-performing storage (like

io2

or striped

gp3

), ensure the BE is configured to utilize it: *

flush_thread_num_per_store

: Increase this (e.g., to 8 or 16) to allow more concurrent flush threads to hit your disks. *

update_stop_and_wait_threshold

: If you see "too many versions" errors during your 30k QPS updates, you may need to tune this to allow the compaction threads more breathing room on your new faster storage. Recommendation * Short term: Stick with your current instances but attach multiple

gp3

volumes per node or a single

io2 Block Express

volume to reach the 3,125 MB/s instance cap. * Long term: Migrate to the Shared-Data architecture. This decouples your storage from your EC2 count entirely, provides the best "non-ephemeral" guarantee (S3), and lets you use the local EBS volumes purely as a performance accelerator (fast) cache. References • deployment/shared_data/s3.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md References • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md

@Rocky If disk IO is not my issue, currently well below 50% IO util, EBS IOPS and throughput not being exceeded, what is the ideal machine

If Disk I/O is not your bottleneck (utilization <50% and EBS limits not reached), then for a workload of 60k inserts/30k updates per second, your performance is almost certainly CPU-bound. In the StarRocks Primary Key engine, every update triggers an "Apply" phase where the system must calculate delete vectors, manage the persistent index, and handle background compaction. These are all CPU-intensive operations. 1. The Ideal Machine: AWS

m7i

or

c7i

Family Your current

r6in.8xlarge

has a 1:8 CPU-to-Memory ratio (32 vCPUs to 256 GiB RAM). This is generally overkill for StarRocks unless you are using a fully in-memory index. For your specific high-update workload, you should transition to a family with a better CPU-to-RAM balance and newer architecture: * Recommended Instance:

m7i.8xlarge

or

m7i.12xlarge

. * Why: The

m7i

uses Sapphire Rapids (latest Intel) which offers better single-threaded performance and AVX-512 instruction support that StarRocks' vectorized engine leverages heavily. * Ratio: It moves you to a 1:4 ratio, which is the StarRocks "Golden Ratio." You get more compute power per dollar spent compared to the

r

(memory-optimized) series. * Alternative for pure throughput:

c7i.12xlarge

. * Ratio: 1:2 ratio. If your memory monitoring shows you are using less than 64-96 GiB of RAM per node, moving to the

c

(compute-optimized) series will give you the most "Apply" and "Compaction" power for your budget. 2. Why CPU matters more than RAM here * Update "Apply" Phase: StarRocks uses

transaction_apply_worker_count

to process updates. If your CPU is saturated, these workers queue up, causing "version too many" errors or ingestion lag. * Compaction: High-frequency updates generate many small versions. StarRocks must constantly merge these (compaction). This is a purely CPU-driven process in a shared-nothing cluster once the data is read from the Page Cache. 3. Tuning for High QPS Updates Since you have 12 nodes and I/O is fine, ensure you aren't hitting software concurrency limits before swapping hardware. Check these

be.conf

settings: *

transaction_apply_worker_count

: Increase this (default is often 8 or 16). For an 8xlarge instance, try setting this to 24 or 32 to allow more concurrent update processing. *

transaction_publish_version_worker_count

: Increase this to speed up the final commit phase of your stream loads. *

paimon_worker_count

/

compaction_threads

: Ensure background compaction can keep up with the 30k QPS update stream. If "Cumulative Compaction" is falling behind, your read performance will eventually degrade even if I/O is low. Summary Recommendation 1. Switch to

m7i

or

m7i.flex

: You will get newer, faster CPU cores. The

r6in

is great for network/RAM, but you are paying a "memory tax" that isn't helping your update throughput. 2. Verify Memory Usage: Run

SHOW BACKENDS

and check

MemUsed

. If you are consistently using <100GB, move to

m7i

(128GB RAM) immediately. If you are using <60GB, move to

c7i

(64GB RAM) to maximize CPU core count. 3. Scale Out vs. Up: If you move to a 12xlarge (48 cores) and still see CPU bottlenecks, it is better to scale out to more nodes (e.g., 18 nodes of 8xlarge) rather than scaling up to a single massive node. StarRocks' MPP architecture thrives on more nodes for update-heavy workloads. References • deployment/plan_cluster.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md References • deployment/plan_cluster.md • best_practices/primarykey_table.md

@Rocky ISnt the problem that m6i has much lower EBS throughput and IOPS? Also reason why it's cheaper

You are correct that moving from the

r6in

series to a standard

m6i

or even

m7i

is a downgrade in terms of EBS throughput headroom. The

r6in

is an outlier—the "n" signifies it is specifically optimized for high network and EBS performance. A

r6in.8xlarge

can push up to 3,125 MB/s to EBS, whereas a standard

m7i.8xlarge

or

r7i.8xlarge

is capped at 1,250 MB/s. The Paradox: If I/O is Not the Issue If your disk I/O utilization is low (<50%) and you are no longer hitting the EBS throughput limits, your 30k QPS update workload is likely CPU-bound. In the StarRocks Primary Key engine, the "Apply" phase (calculating delete vectors and merging versions) is extremely compute-intensive. The ideal machine for you depends on whether you want to prioritize CPU power or keep the EBS headroom just in case. Option 1: Prioritize CPU Performance (The

m7i

Family) If you are confident that you don't need >1,250 MB/s of disk throughput, the

m7i

(4th Gen Intel Sapphire Rapids) is the ideal choice. * Why: Even though it has a lower EBS cap than the

r6in

, it has a much newer CPU architecture. Clock-for-clock, Sapphire Rapids is significantly faster at the vectorized operations StarRocks uses for high-speed updates. * Throughput Check: Since you mentioned you were maxing out at 1,000 MB/s, the

m7i.8xlarge

(1,250 MB/s) still gives you more than you are currently using, without the price premium of the "n" series. * Recommendation:

m7i.12xlarge

. This gives you 48 vCPUs (up from 32) and 1,875 MB/s of EBS throughput, which is a significant upgrade to your CPU while still providing more disk throughput than your current 1,000 MB/s bottleneck. Option 2: Keep the High EBS Headroom (The

r6in

Up-scale) If you want to keep the massive 3,125 MB/s EBS pipe of the

r6in

but solve the CPU bottleneck: * Recommendation:

r6in.12xlarge

. * Why: You move from 32 to 48 vCPUs, and your EBS throughput cap increases to 4,687 MB/s. This is the safest move if you think your updates might eventually spike back into high I/O. Comparing the Options Instance Type | vCPU | RAM | Max EBS Throughput | Architecture

r6in.8xlarge

(Current) | 32 | 256 GiB | 3,125 MB/s | 3rd Gen Intel

m7i.8xlarge

| 32 | 128 GiB | 1,250 MB/s | 4th Gen Intel (Faster)

m7i.12xlarge

| 48 | 192 GiB | 1,875 MB/s | 4th Gen Intel (Faster)

r6in.12xlarge

| 48 | 384 GiB | 4,687 MB/s | 3rd Gen Intel Final Recommendation for High Updates If you are hitting CPU limits on your 30k QPS updates (check your BE CPU usage), move to the

m7i

generation. You lose "theoretical" EBS headroom that you aren't using, but you gain newer, faster CPU cores that process the "Apply" and "Compaction" tasks much more efficiently. Pro Tip: If your memory usage is low (e.g.,

< 100GB

), the

m

family (1:4 ratio) is the most cost-effective. If you need the 256GB RAM you currently have, stick with the

r

family but move to the

r7i

(Sapphire Rapids) for the CPU boost. References • deployment/plan_cluster.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md References • deployment/plan_cluster.md • best_practices/primarykey_table.md • administration/management/BE_configuration.md • table_design/table_types/primary_key_table.md • administration/management/monitoring/Monitor_and_Alert.md References

@Rocky If I don't need the memory optimized machines because each BE only using around 60-80 GB of RAM currently, but want to keep the high EBS performance, what machines should I look at

To clarify the performance and cost trade-offs, you are correct that the

n

(Network Optimized) variant you are currently using (

r6in

) has significantly higher EBS and network ceilings than the standard

m6i

or

r6i

families. However, there is a nuance in how AWS prices and specs these. 1. EBS Throughput Comparison (8xlarge) Instance Type | EBS Bandwidth (Mbps) | EBS Throughput (MB/s) | Max IOPS (16KB)

r6in.8xlarge

(Current) | 20,000 | 2,500 | 80,000

m6i.8xlarge

| 10,000 | 1,250 | 40,000

m7i.8xlarge

| 10,000 | 1,250 | 40,000 You are right: the

m6i

has exactly half the EBS throughput of your

r6in

. The "n" suffix on your current machine is specifically designed for high-throughput storage and networking. The reason

m6i

is cheaper is primarily the RAM-to-vCPU ratio (4:1 vs 8:1) and the lower network/EBS overhead. 2. If Disk I/O is NOT the issue (CPU-Bound) If your I/O utilization is now under 50% (perhaps due to better batching or the Persistent Index finally warming up), then your bottleneck has shifted to CPU instruction efficiency. In this case, the

m7i

(7th Gen) is actually superior to your

r6in

(6th Gen) despite having lower EBS "ceilings": * Newer Architecture:

m7i

uses Intel Sapphire Rapids. In StarRocks' vectorized engine, the architectural improvements in the 7th gen often result in 15-25% better throughput per core for the same clock speed. * The "EBS Bottleneck" Trap: Even though

m7i

"caps" at 1,250 MB/s, if your actual workload is only pushing 500-700 MB/s (which is typical for a 50% util on a 1000MB/s setup), the `r6in`'s 2,500 MB/s ceiling is "unused runway." You are paying for a high ceiling you aren't hitting. 3. The Ideal "Middle Ground":

m7i

or

c7i

If you want to maximize performance for a 60k/30k QPS update workload without the "ephemeral" risk of NVMe: * For pure performance/price:

m7i.8xlarge

. You lose the extreme EBS ceiling of the

n

series, but you gain much faster CPU cores. Since you are currently at <50% I/O util, the 1,250 MB/s limit of the

m7i

is still a comfortable buffer. * If you want the best of both worlds:

r7iz

(High frequency). These have the fastest clock speeds (up to 3.9 GHz) and high EBS throughput. They are expensive, but for high-concurrency Primary Key updates, single-core speed is king. Why was your I/O high before? A common reason for high I/O util on Primary Key tables that then "disappears" is the Persistent Index. * When a node starts or a new large load occurs, StarRocks may need to read the index from disk. This spikes I/O. * Once the index is cached or the "Apply" phase catches up, I/O drops and CPU becomes the limit. Final Recommendation If you are CPU-bound and your I/O is stable: 1. Move to

m7i.8xlarge

: It’s the most cost-effective way to get more "Apply" power for your updates. 2. Scale Out: Instead of 12x

r6in.8xlarge

, consider 16x

m7i.8xlarge

. This gives you more CPU cores to handle the 30k QPS update "Apply" tasks while staying well within the EBS limits of the M series. 3. Check

update_stop_and_wait_threshold

: If you see performance degradation during peaks, ensure this is set high enough so your ingest doesn't "pause" to wait for compaction. References • deployment/plan_cluster.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md References • deployment/plan_cluster.md • administration/management/BE_configuration.md • best_practices/primarykey_table.md