@Rocky Which storageclass I should use for FE nodes? IO2 or gp3?

For StarRocks FE (Frontend) nodes, gp3 is the recommended choice over io2. Why gp3 is better for FE nodes: * Workload Requirements: FE nodes primarily store metadata (using BDBJE). This metadata is relatively small and, while it requires consistency, it is not nearly as I/O intensive as the data scanning performed by BE (Backend) nodes. * Performance: StarRocks documentation typically states that 100 GB of HDD-level storage is sufficient for FE metadata in most scenarios. According to community guidance, 5,000 IOPS is plenty for low-latency queries on FE. A standard

gp3

volume starts at 3,000 baseline IOPS and can be easily provisioned up to 16,000 IOPS at a much lower cost than

io2

. * Cost Efficiency:

io2

is designed for mission-critical, ultra-high IOPS (up to 64,000+) workloads. Using

io2

for FE nodes would result in significantly higher costs without providing a noticeable performance benefit for metadata operations. Recommendation: * Storage Type:

gp3

* Size: 100 GB to 200 GB is standard for production FE metadata. * IOPS/Throughput: The baseline

gp3

settings (3,000 IOPS / 125 MB/s) are usually sufficient. If you have a very large cluster with high metadata churn (e.g., thousands of partitions/tables), you might consider bumping the

gp3

IOPS to 5,000. Example Kubernetes StorageClass (gp3): If you are defining this for a Kubernetes cluster on AWS (EKS), your StorageClass should look similar to this:

yaml
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
  name: starrocks-fe-gp3
provisioner: <http://ebs.csi.aws.com|ebs.csi.aws.com>
parameters:
  type: gp3
  iops: "3000"        # Baseline is 3000, increase if needed
  throughput: "125"  # Baseline is 125 MiB/s
allowVolumeExpansion: true
reclaimPolicy: Retain

────────── In your StarRocksCluster CRD (Operator) starRocksFeSpec: storageSpec: storageClassName: "starrocks-fe-gp3" storageSize: "200Gi" ``` ## References - [administration/management/FE_configuration.md](https://docs.starrocks.io/docs/administration/management/FE_configuration)

@Rocky How about FE logs how much volume shall I assign for prod based cluster?

For a production-level StarRocks cluster, it is a best practice to separate the FE logs from the FE metadata using different volumes. This prevents the FE from crashing or becoming "Read Only" if the log volume fills up. Recommended Volume Size for FE Logs For most production clusters, a volume of 100 GB to 200 GB is the standard recommendation. The sizing depends largely on your query volume because the Audit Log (

fe.audit.log

) is typically the largest contributor to disk usage. Here is the breakdown based on default StarRocks retention settings: 1. System Logs (

fe.log

): * Default: Keeps up to 10 rotated files, each up to 1 GB. * Estimated Space: ~10 GB. 2. Audit Logs (

fe.audit.log

): * Default: Keeps up to 90 rotated files, each up to 1 GB (retention is often set to 30 days by default in newer versions). * Estimated Space: Up to 90 GB. 3. Other Logs (Big Query, Profiles, etc.): * Estimated Space: ~5–10 GB. Best Practices for FE Log Management * StorageClass: Use gp3. Logs are sequential writes; you do not need the expensive IOPS of

io2

. * Compression: Enable log compression to significantly reduce disk usage. * Set

sys_log_enable_compress = true

and

audit_log_enable_compress = true

in your

fe.conf

. * Retention Policy: If your compliance requirements allow, you can reduce

audit_log_roll_num

(e.g., to 30) or

audit_log_delete_age

(e.g., to 7d) to save space. * Separate PVCs (Kubernetes): If using the StarRocks Operator, define a separate

mountPath

for

/opt/starrocks/fe/log

and

/opt/starrocks/fe/meta

. Summary Recommendation * Metadata Volume: 100 GB - 200 GB (

gp3

). * Log Volume: 200 GB (

gp3

) — this provides a comfortable buffer for audit logs and prevents manual cleanup tasks. References • administration/management/logs.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/management/logs.md • faq/operation_maintenance_faq.md

@Rocky Both Metadata and Logs would be the PVCs right?

Yes, for a production StarRocks cluster on Kubernetes (using the StarRocks Operator or Helm Chart), both the Metadata and Logs are typically provisioned as separate PVCs. Separating them is a critical best practice to ensure that if the log volume fills up (e.g., due to a burst in audit logs), it does not corrupt or block the FE's metadata writes, which would crash the cluster. How it works in Kubernetes When you define the

storageSpec

in your

StarRocksCluster

CRD or Helm

values.yaml

, the system automatically creates two distinct Persistent Volume Claims for each FE pod: 1. Metadata PVC (

fe-meta-...

): * Mount Path:

/opt/starrocks/fe/meta

* Usage: Stores BDBJE metadata (the "brain" of the cluster). 2. Log PVC (

fe-log-...

): * Mount Path:

/opt/starrocks/fe/log

* Usage: Stores

fe.log

,

fe.audit.log

, and

fe.warn.log

. Example Configuration (Operator/Helm) If you use the Helm chart, a single storage configuration often handles both, but the chart internally splits them into two PVCs per pod:

yaml
starrocksFESpec:
  storageSpec:
    name: fe-storage        # Prefix for the PVC names
    storageClassName: gp3   # Recommended SC
    storageSize: 200Gi      # Note: Often this size is applied to BOTH logs and meta

Important Sizing Note Because some deployment methods might apply the same

storageSize

to both volumes, ensure you choose a size that satisfies the Log requirement (which is usually larger). * Metadata: Rarely exceeds 50GB even in massive clusters. * Logs: Can easily reach 100GB+ depending on your audit log retention. * Recommendation: If your configuration doesn't allow for separate sizes, setting it to 200GB is a safe production baseline for both. Summary * Yes, they are separate PVCs. * Why: To prevent "Disk Full" errors in logs from breaking the metadata. * StorageClass: Use

gp3

for both. References • quick_start/helm.md • administration/management/FE_configuration.md • administration/management/logs.md References • quick_start/helm.md • deployment/shared_data/hdfs.md • deployment/shared_data/s3.md • deployment/shared_data/minio.md

@Rocky throughput as 125 would be enough?

For a StarRocks FE (Frontend) node, a throughput of 125 MB/s (the baseline for AWS

gp3

) is more than enough for almost all production scenarios. Why 125 MB/s is sufficient: 1. Metadata (BDBJE): FE metadata updates (edit logs) are frequent but very small in size. They are latency-sensitive rather than throughput-sensitive. The total volume of metadata written per second is typically in the kilobyte or low megabyte range, even in large clusters. 2. Logging: The largest consumer of disk throughput on an FE is the

fe.audit.log

. Even with high query concurrency (e.g., 2,000 queries per second), a text-based audit log will rarely exceed 10–20 MB/s. 125 MB/s provides a massive buffer. 3. Image Syncing: When a new FE joins or follows a leader, it downloads a metadata "image." These files are usually a few hundred MBs to a few GBs. A 125 MB/s throughput allows these images to transfer in seconds. *What actually matters for FE: IOPS* Instead of throughput (MB/s), you should focus on IOPS (Input/Output Operations Per Second). * Recommendation: Aim for 3,000 to 5,000 IOPS. * Why: StarRocks uses BDBJE for metadata consensus. If disk latency is high (low IOPS), you may see

Latch timeout

or

LogbufferPool_FullLatch

errors in your logs, which can cause FE instability or leader elections. * gp3 Advantage: Since

gp3

allows you to provision IOPS (up to 3,000 baseline) and Throughput (125 MB/s baseline) independently, you are perfectly fine with the baseline 125 MB/s. You only need to increase the IOPS if you have a very large metadata set (thousands of tablets/shards) or extremely high query volumes. Summary * Throughput: 125 MB/s is plenty. * IOPS: 3,000 (gp3 baseline) is usually enough; consider 5,000 for very large clusters. * StorageClass: Stick with gp3. Upgrading to

io2

or increasing throughput beyond 125 MB/s for an FE node is generally an unnecessary expense. References • administration/Meta_recovery.md • administration/management/FE_configuration.md • administration/management/logs.md References • administration/management/FE_configuration.md • administration/management/logs.md • administration/management/monitoring/metrics.md

@Rocky If I am not using NVME based instances then for CN node what should we use in production? What do you think about this? apiVersion: "storage.k8s.io/v1" kind: "StorageClass" metadata: name: "starrocks-cn-node" provisioner: "ebs.csi.aws.com" reclaimPolicy: "Delete" volumeBindingMode: "WaitForFirstConsumer" allowVolumeExpansion: true parameters: type: "io2" encrypted: "true" csi.storage.k8s.io/fstype: "ext4" allowAutoIOPSPerGBIncrease: "true" blockExpress: "true" iops: "4000"

For a production StarRocks CN (Compute Node), the storage is primarily used for the Data Cache (caching data from S3/HDFS). While the

io2 Block Express

configuration you shared is technically "high-end," it is likely not the most cost-effective or performant choice for this specific role. Here is an evaluation of your

StorageClass

and what is recommended for production CN nodes: 1. Evaluation of your

io2

Configuration * The Problem with 4,000 IOPS: For a CN node, 4,000 IOPS is quite low. StarRocks Data Cache handles many small random reads (blocks are typically 1MB). A single complex query or multiple concurrent users can easily saturate 4,000 IOPS, leading to "I/O wait" and slow query performance. * Cost vs. Benefit:

io2

is designed for extreme durability (99.999%) and sub-millisecond latency. However, CN cache is stateless. If a CN disk fails, StarRocks simply fetches data from remote storage (S3) again. Paying the high premium for

io2

durability on a cache layer is usually unnecessary. * Block Express: This only works with specific Nitro-based instances (e.g., c6g, r6g, m6g). If your instance type doesn't support it, you won't see the benefits. 2. The Recommended Approach: AWS

gp3

For StarRocks CN nodes,

gp3

is the industry standard for production EBS-backed clusters. It allows you to provision high IOPS and Throughput independently at a much lower cost. Recommended Production Targets for CN Cache: * IOPS: 10,000 to 16,000. * Throughput: 500 MB/s to 1,000 MB/s. * Size: 500 GB to 2 TB (depending on your "hot" data size). 3. Suggested

StorageClass

for CN If you want a high-performance production setup without NVMe, use a

gp3

class tuned for throughput and IOPS:

yaml
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
  name: starrocks-cn-gp3
provisioner: <http://ebs.csi.aws.com|ebs.csi.aws.com>
parameters:
  type: gp3
  iops: "10000"           # Higher than your 4k io2 for better cache performance
  throughput: "500"      # Crucial for scanning large data segments
  encrypted: "true"
  <http://csi.storage.k8s.io/fstype|csi.storage.k8s.io/fstype>: xfs  # XFS is generally preferred for high-concurrency DBs
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

4. Why XFS over EXT4? While your config uses

ext4

, XFS is highly recommended for StarRocks storage. XFS handles concurrent I/O and large files more efficiently, which is exactly how StarRocks manages its cache blocks and segment files. Summary Checklist for Production CN 1. Prefer `gp3`: It is ~60-80% cheaper than

io2

for the same performance. 2. Scale IOPS/Throughput: Don't settle for the baseline 3,000 IOPS. For production, aim for at least 10,000 IOPS and 500 MB/s per CN node. 3. Local NVMe is still King: If you ever switch instance types, the

i3en

or

r6id

families (with local NVMe) will significantly outperform any EBS volume for the Data Cache because of the direct physical connection. References • data_source/data_cache.md • administration/management/BE_configuration.md • deployment/shared_data/s3.md References • deployment/shared_data/s3.md • administration/management/BE_configuration.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md References • using_starrocks/caching/block_cache.md

@Rocky Why is IO2 not preferred over gp3?

The primary reason

io2

(and

io2 Block Express

) is not preferred over

gp3

for StarRocks is that it offers "extreme" features—specifically ultra-high durability and sub-millisecond latency—that StarRocks nodes (especially CN nodes) simply don't need, at a cost that is often 5x to 10x higher. Here is the detailed breakdown of why

gp3

is the better choice for production StarRocks: 1. Cost-to-Performance Efficiency * The Price Gap: For an identical performance profile (e.g., 10,000 IOPS and 500 MB/s throughput), a

gp3

volume costs roughly $150–$200/month, while an

io2

volume can easily exceed $1,000/month. * Wasted Performance: Most database workloads (including StarRocks) rarely hit the sustained sub-millisecond latency that

io2

guarantees.

gp3

provides consistent single-digit millisecond latency, which is more than sufficient for StarRocks' columnar scanning and block-based caching. 2. The Role of the CN Node (Cache vs. Source of Truth) * Stateless Cache: In a StarRocks Shared-Data architecture, the CN node's local disk is used for the Data Cache. The "source of truth" for your data is AWS S3. * Durability Overkill:

io2

provides 99.999% durability. However, if a CN node's cache disk fails, StarRocks just fetches the data from S3 again. You are paying a massive premium for "five-nines" durability on a disk that is essentially a temporary mirror of S3.

gp3

(99.8-99.9%) is perfectly safe for this role. 3. Flexibility of

gp3

* Independent Scaling: Unlike the older

gp2

or

io1

,

gp3

allows you to provision IOPS and Throughput independently of the storage size. _ _Scenario:* You need a small 200GB disk but high 10,000 IOPS for cache performance. * With

gp3

, you just pay for those specific numbers. * With

io2

, you are forced into a very high price tier to reach those performance levels. 4. Throughput is the Bottleneck, not Latency StarRocks is an OLAP engine. It excels at large-scale scans. For these scans, Throughput (MB/s) is usually more important than random I/O latency. *

gp3

supports up to 1,000 MB/s (and recently up to 2,000 MB/s in some regions). * This is the same baseline as

io2

. Unless you are using

io2 Block Express

(which requires specific high-end Nitro instances), you don't actually get more throughput by switching to

io2

. When WOULD you use

io2

? You should only consider

io2

if: 1. Extreme Scale: You need more than 16,000 IOPS per single volume (though most StarRocks users would just add more CN nodes or use multiple

gp3

disks instead). 2. Compliance: You have a strict corporate mandate for 99.999% durability on all attached volumes, regardless of the data's role. Summary Comparison Feature | ***gp3** (Recommended)* | ***io2** (Overkill)* Price | Base $0.08/GB | Base $0.125/GB + High IOPS fees Latency | 1-5ms (Excellent for OLAP) | <1ms (Unnecessary for OLAP) Durability | 99.9% | 99.999% Best For | Production Data Cache / FE Meta | Mission-critical OLTP (Postgres/Oracle) Recommendation: Stick with

gp3

. Use the money you save to provision higher IOPS (10k+) and higher Throughput (500MB/s+) on that

gp3

volume, which will give you a much better performance boost for StarRocks than simply switching to the

io2

type. References • data_source/data_cache.md • introduction/Architecture.md • deployment/plan_cluster.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • using_starrocks/caching/block_cache.md

@Rocky For FE node we should use retainPOlicy as Delete or retain??

For a production StarRocks FE (Frontend) node, you should use a

reclaimPolicy

of

Retain

. Why

Retain

is the Production Standard: 1. Metadata Protection: The FE node stores the "brain" of your cluster (the

bdbje

metadata). If this data is lost, your entire cluster's schema, user permissions, and table structures are gone. 2. Accidental Deletion: If you use

Delete

, and someone accidentally deletes the Kubernetes Namespace, the Helm release, or the

PersistentVolumeClaim

(PVC), the underlying cloud storage (EBS) is instantly wiped. With

Retain

, the EBS volume remains in your AWS account even if the Kubernetes resource is removed, allowing for manual recovery. 3. Disaster Recovery: If a node or a cluster fails and needs to be rebuilt, having the physical disks "retained" allows you to re-attach the existing metadata to a new FE pod without having to restore from a backup (which might be hours or days old). When to use `Delete`: * Test/Dev Environments: If you are frequently creating and destroying clusters and don't want to manually clean up orphaned EBS volumes (and avoid extra costs),

Delete

is fine for non-production use. Recommendation Update your FE

StorageClass

to include the `reclaimPolicy`:

yaml
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
kind: StorageClass
metadata:
  name: starrocks-fe-storage
provisioner: <http://ebs.csi.aws.com|ebs.csi.aws.com>
reclaimPolicy: Retain  # <--- Change from Delete to Retain
volumeBindingMode: WaitForFirstConsumer
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"

Pro Tip: If you have already deployed your cluster with

Delete

, you can actually patch the existing PersistentVolumes (PV) directly without downtime to change them to `Retain`:

kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

References • administration/Meta_recovery.md • faq/Deploy_faq.md • quick_start/helm.md References • quick_start/helm.md • administration/Meta_recovery.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • administration/management/BE_configuration.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • deployment/plan_cluster.md • introduction/Architecture.md • faq/operation_maintenance_faq.md • administration/management/logs.md • administration/management/monitoring/metrics.md • data_source/data_cache_troubleshooting.md • feature_support/data_lake_analytics.md • quick_start/helm.md • faq/Deploy_faq.md

@Rocky How about for BE node? github.com/awslabs/data-on-eks/blob/…/storageclass.yaml one is using Delete

For a BE (Backend) node, the recommended

reclaimPolicy

depends entirely on whether you are running a Shared-Nothing or Shared-Data architecture. 1. Shared-Nothing (Classic BE) * Recommended Policy:

Retain

* Why: In this mode, the BE disks store the actual data replicas. If a disk is deleted, you lose a piece of your data. While StarRocks will automatically repair the lost replica by copying data from other nodes, this process is expensive in terms of CPU and Network. * Production Safety: Using

Retain

prevents a situation where a accidental

kubectl delete pvc

or a Helm uninstall wipes out your data. 2. Shared-Data (CN/BE as Cache) * Recommended Policy:

Delete

(usually acceptable) * Why: In Shared-Data mode, the disks on the BE/CN nodes are used for the Data Cache. The "Source of Truth" is S3 or HDFS. If a cache disk is deleted, no data is lost; your queries just run slower for a few minutes while the cache is refilled from remote storage. * Efficiency: Using

Delete

makes scaling down much cleaner. When you remove a CN node, the EBS volume is automatically deleted, so you don't end up with "orphaned" volumes costing you money in AWS. Why does the AWS Labs example use

Delete

? The AWS Data on EKS blueprint is designed for automated infrastructure. 1. Cleanup: These blueprints are often used for demos, testing, or ephemeral clusters. If they used

Retain

, every time a user ran

terraform destroy

or deleted the cluster, the EBS volumes would stay in the AWS account, leading to unexpected costs. 2. Shared-Data Focus: Most modern StarRocks deployments on AWS/EKS favor the Shared-Data architecture (S3-backed), where BE/CN nodes are essentially stateless compute workers. In that context,

Delete

is the correct cloud-native approach. Summary Checklist Role | Recommended Policy | Reason FE (Frontend) |

Retain

| Metadata. Loss of these disks is catastrophic without backups. BE (Shared-Nothing) |

Retain

| Data Replicas. Prevents unnecessary data re-replication and loss. CN / BE (Shared-Data) |

Delete

| Cache Only. Safe to lose, easier to scale and manage costs. Rocky's Advice: If you are building a production cluster where you might ever need to perform manual forensics or emergency recovery on a node's local state, use

Retain

. You can always delete the volumes manually once you are sure they aren't needed. References • administration/Meta_recovery.md • deployment/shared_data/s3.md • faq/Deploy_faq.md • introduction/Architecture.md References • administration/management/BE_configuration.md • faq/Others.md