Hello all, I am a newbie of Starrocks and tryin to do a benchmark with ~60 Tb of data among starrocks eks shared model vs databricks. I have below configuration and my question is how is these configurations considering the benchmark data size. Thank you in advance for the answers!

eks_node_instance_types = ["r7g.8xlarge"]
eks_node_ami_type       = "AL2023_ARM_64_STANDARD"
eks_node_desired_size   = 1
eks_node_min_size       = 1
eks_node_max_size       = 12

# Total Resources: 192 vCPU, 1.5TB RAM

# StarRocks FE Configuration
starrocks_fe_replicas       = 1 # Single FE for minimal cost (no HA)
starrocks_fe_storage_size   = "200Gi"
starrocks_fe_cpu_request    = "4"
starrocks_fe_memory_request = "16Gi"
starrocks_fe_cpu_limit      = "8"
starrocks_fe_memory_limit   = "32Gi"

# StarRocks CN Configuration (Primary for shared data mode)
starrocks_cn_replicas       = 4       # Start with 4 CNs (minimal baseline)
starrocks_cn_storage_size   = "500Gi"
starrocks_cn_cpu_request    = "16"
starrocks_cn_memory_request = "120Gi"
starrocks_cn_cpu_limit      = "30"
starrocks_cn_memory_limit   = "240Gi"

# Autoscaling Configuration
enable_cluster_autoscaler                           = true
cluster_autoscaler_version                          = "1.30.1"
cluster_autoscaler_scale_down_delay                 = "5m"
cluster_autoscaler_scale_down_unneeded_time         = "5m"
cluster_autoscaler_scale_down_utilization_threshold = "0.5"

enable_starrocks_hpa                   = true
starrocks_cn_max_replicas              = 24 # Can scale to 24 CNs during heavy queries
starrocks_cn_target_cpu_utilization    = 70
starrocks_cn_target_memory_utilization = 80

Hey Murat, welcome to the community! Your setup looks reasonable as a starting point for a the POC. However, without more details on your query and Databricks side config, it's hard to give specific advice. But overall looks good, you can always tune further during the POC. And one thing I'd suggest is probably considering increase the Data Cache size to get better performance.

What type of disk volume are you using? Sometimes, when ingestion becomes heavy, the throughput of the disk volume can become a bottleneck. If you are using a network volume like EFS, you can either choose a higher-throughput performance mode or configure multiple disk volumes to distribute the load.

Thanks a lot for the feedbacks. @ahn heesuk I am using EBS gp2. Do you recommend me to use EFS over EBS? also what do you think about the cluster family? is r7g a proper choice to do this benchmark or need a cluster which has better network bandwidth

@Murat Yildirim If you want high performance from a disk I/O perspective, it is generally recommended to use instance types that provide NVMe storage. In my case, however, I am not using NVMe. Instead, for cost efficiency, I am using EBS gp3 volumes and tuning only the IOPS and throughput. As far as I know, the recommended specification for a StarRocks CN node is typically around 16 CPU and 64 GB of memory. You could either start with a specification around that range, or begin with a smaller instance type and gradually scale up until you reach a configuration that provides satisfactory performance.