@Rocky does this config flush_thread_num_per_store also applies to CN node in a shared-data deployment?

In a StarRocks shared-data (compute-storage separation) deployment, the configuration

flush_thread_num_per_store

is generally not the parameter you should use for CN nodes. Instead, StarRocks provides a specific parameter for shared-data mode:

lake_flush_thread_num_per_store

* Description: This parameter controls the number of threads used for flushing MemTables in each store (cache directory) specifically for the shared-data architecture (CN nodes). * Default:

0

. * When set to

0

, the system automatically uses twice the number of CPU cores as the thread count. _ When set to a negative value (e.g.,

-2

), the system uses the absolute value multiplied by the CPU core count (e.g.,

2 _ cores

). * Applicability: This is the effective parameter for CN nodes when they are running in a shared-data cluster to manage data flushing to object storage (via the local cache). Why the distinction? *

flush_thread_num_per_store

: This is designed for the shared-nothing architecture, where Backend (BE) nodes manage local tablets on physical disks. *

lake_flush_thread_num_per_store

: Introduced specifically for the "Lake" (shared-data) engine to optimize how compute nodes (CN) flush data to remote storage. Recommendation: If you need to tune the flush concurrency on your CN nodes in a shared-data deployment, you should modify

lake_flush_thread_num_per_store

in your

cn.conf

or via the

UPDATE information_schema.be_configs

command if your version supports dynamic modification for this parameter. References • administration/management/BE_configuration.md • deployment/shared_data/minio.md • deployment/shared_data/azure.md • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • quick_start/shared-data.md • deployment/shared_data/gcs.md

@Rocky this is related to this tarrocks-cn-0 -- curl -s http://localhost:8040/metrics | grep -E "^starrocks_be_(mem|datacache|load)" | sort starrocks_be_datacache_disk_quota_bytes 10737418240 starrocks_be_datacache_disk_used_bytes 2792095744 starrocks_be_datacache_mem_bytes 629350476 starrocks_be_datacache_mem_quota_bytes 1043677052 starrocks_be_datacache_mem_used_bytes 629350476 starrocks_be_load_bytes 986615479 starrocks_be_load_channel_active_threads 0 starrocks_be_load_channel_add_chunks_duration_us 463203315 starrocks_be_load_channel_add_chunks_eos_total 89726 starrocks_be_load_channel_add_chunks_total 89726 starrocks_be_load_channel_add_chunks_wait_memtable_duration_us 0 starrocks_be_load_channel_add_chunks_wait_replica_duration_us 0 starrocks_be_load_channel_add_chunks_wait_writer_duration_us 413310858 starrocks_be_load_channel_count 0 starrocks_be_load_channel_execute_time_ns_total 13806495573 starrocks_be_load_channel_executed_tasks_total 89730 starrocks_be_load_channel_pending_time_ns_total 8690168650 starrocks_be_load_channel_queue_count 0 starrocks_be_load_channel_running_threads 2 starrocks_be_load_channel_threadpool_size 2 starrocks_be_load_mem_bytes 0 starrocks_be_load_rows 1213660 starrocks_be_load_rpc_threadpool_size 10 starrocks_be_load_spill_block_merge_active_threads 0 starrocks_be_load_spill_block_merge_execute_time_ns_total 0 starrocks_be_load_spill_block_merge_executed_tasks_total 0 starrocks_be_load_spill_block_merge_pending_time_ns_total 0 starrocks_be_load_spill_block_merge_queue_count 0 starrocks_be_load_spill_block_merge_running_threads 1 starrocks_be_load_spill_block_merge_threadpool_size 1 starrocks_be_load_spill_local_blocks_read_total 0 starrocks_be_load_spill_local_blocks_write_total 0 starrocks_be_load_spill_local_bytes_read_total 0 starrocks_be_load_spill_local_bytes_write_total 0 starrocks_be_load_spill_remote_blocks_read_total 0 starrocks_be_load_spill_remote_blocks_write_total 0 starrocks_be_load_spill_remote_bytes_read_total 0 starrocks_be_load_spill_remote_bytes_write_total 0 starrocks_be_memory_pool_bytes_total 0 starrocks_be_memtable_finalize_duration_us 7558489 starrocks_be_memtable_finalize_task_total 18832 starrocks_be_memtable_flush_active_threads 0 starrocks_be_memtable_flush_disk_bytes_total 417744994 starrocks_be_memtable_flush_duration_us 95963540 starrocks_be_memtable_flush_execute_time_ns_total 0 starrocks_be_memtable_flush_executed_tasks_total 0 starrocks_be_memtable_flush_io_time_us 21430769 starrocks_be_memtable_flush_memory_bytes_total 1064829037 starrocks_be_memtable_flush_pending_time_ns_total 0 starrocks_be_memtable_flush_queue_count 0 starrocks_be_memtable_flush_running_threads 2 starrocks_be_memtable_flush_threadpool_size 2 starrocks_be_memtable_flush_total 18832

Hello! I'm Rocky. Based on the metrics you shared, here is an analysis of your StarRocks Backend (BE) performance, focusing on Data Cache and Loading: 1. Data Loading Analysis (Potential Bottleneck) The most notable metric in your list is the

starrocks_be_load_channel_add_chunks_wait_writer_duration_us

. * Metric Value: 413,310,858 us (~413 seconds). * Total Duration: 463,203,315 us (~463 seconds). * Observation: The time spent waiting for the "Async Delta Writer" accounts for approximately 89% of the total

add_chunk

duration. * Insight: This indicates a downstream bottleneck in the loading pipeline. Your system is spending most of its time waiting for the writers to process or commit data rather than actually receiving/processing the chunks. Possible causes for high `wait_writer`: * Heavy Compaction: If the BE is busy with background compaction tasks, writing performance for new loads can be delayed. * High Concurrency: Many simultaneous load tasks might be exhausting the writer thread pool. * Disk Latency: While I/O time isn't the only factor, check if your disks are under heavy load. 2. MemTable Flush Performance * Flush Duration: 95.9s total for 18,832 tasks (~5.1ms avg per flush). * I/O Time: 21.4s total, which is only about 22% of the total flush execution time. * Compression: You flushed ~1GB of memory data into ~417MB of disk data, indicating a healthy compression ratio of about 2.5x. * Recommendation: Since I/O is only 22% of the flush time, the bottleneck in flushing is likely not the disk hardware itself, but potentially CPU overhead for compression or thread scheduling. 3. Data Cache (Block Cache) Status * Disk Cache: You are using ~2.8GB out of a 10.7GB quota (26% usage). * Memory Cache: You are using ~0.6GB out of a 1GB quota (60% usage). * Status: The cache is currently healthy and well within its configured limits. It has plenty of room to grow before eviction starts (which typically triggers at 80% disk usage). Summary & Suggestions: * Investigate Load Bottlenecks: Since the

wait_writer

duration is high, you should check the BE config

push_worker_count_high_priority

or

flush_thread_num_per_store

if you suspect the threads are saturated. * Monitor Profiles: To get a deeper look, run

ANALYZE PROFILE

on a slow load job. This will confirm if the

WaitWriterTime

is indeed the primary culprit for specific transactions. * Check Compaction: Use

SHOW PROC '/compactions';

to see if there is a massive backlog of compaction tasks competing for resources with your loads. Do you have a specific performance issue or error message you're trying to solve with these metrics? I'm here to help! References • administration/management/monitoring/metrics.md • loading/loading_introduction/troubleshooting_loading.md • administration/management/resource_management/Memory_management.md

@Rocky im deploying to eks and my problem is that the memory for the starrocks nodes are already 80% even it is deployed using shared-data deployment. i also have 20+ kafka connectors loading data from kafka to starrocks

High memory usage (around 80%) on Compute Nodes (CN) in a shared-data EKS deployment is actually quite common because of how StarRocks manages its memory. However, when you have 20+ Kafka Routine Load jobs running, you need to ensure the memory is being used efficiently and not leading to OOM (Out of Memory) kills. Here is a breakdown of why this is happening and how to optimize it: 1. Understanding StarRocks Memory Behavior By default, StarRocks is designed to utilize as much memory as possible for caching and performance. *

mem_limit

: In a containerized environment like EKS, if you haven't explicitly set

mem_limit

in your

cn.conf

, StarRocks defaults to using 90% of the pod's memory limit. Seeing 80% usage often means the node is just operating within its designated healthy range. * Memory Tracker: StarRocks uses its own internal memory tracker. You can check what is actually consuming the memory by running this SQL command:

sql
    SHOW BACKEND OPTIMIZE; -- For shared-nothing
    -- Or check the metrics endpoint of the CN node:
    -- http://<cn_ip>:<be_http_port>/mem_tracker

2. Routine Load Memory Consumption With 20+ Kafka connectors, the cumulative memory used for data buffering can be significant. Each Routine Load task involves: *

write_buffer_size

: The default is 100MB. This is the amount of data buffered in memory per tablet before being flushed to storage. * Concurrent Tasks: If each of your 20 jobs has a high

desired_concurrent_number

, you could have many tasks running simultaneously. _ _Calculation:* 20 jobs × 3 tasks/job × 100MB buffer = ~6GB of memory just for write buffers. How to optimize Routine Load memory: * Limit Total Load Memory: Adjust

load_process_max_memory_limit_percent

in

cn.conf

. The default is 60%. If you want to leave more room for queries, you can lower this (e.g., to 30%). * Reduce Buffer Size: If your data volume per partition is small, you can reduce

write_buffer_size

(e.g., to 32MB or 64MB) in

cn.conf

to reduce the per-task footprint. 3. Shared-Data "Data Cache" In shared-data mode, CN nodes use a Data Cache to speed up queries by keeping remote data (from S3/EKS) on local disk. * While the data is on disk, the metadata for this cache resides in memory. * If you have a very large disk cache but small RAM, the metadata can consume a noticeable percentage of memory. 4. Recommended EKS/CN Configurations Since you are on EKS, ensure your Kubernetes resource limits and StarRocks internal limits are synced: 1. Set Explicit Memory Limits: In your

cn.conf

(via ConfigMap), explicitly set the memory limit to match your Kubernetes

limits.memory

(minus a small buffer for the OS).

properties
    mem_limit = 90% # Or a specific value like 14GB for a 16GB pod

2. Monitor via Grafana: Check the

Capacity

vs

Usage

metrics. If the memory usage is a flat line at 80-90% and you aren't seeing "Process Memory Exhausted" errors in the

<http://cn.INFO|cn.INFO>

logs, the system is likely stable. 3. Adjust Routine Load Concurrency:

sql
    -- Check how many tasks are actually running
    SHOW ROUTINE LOAD;

If you have too many concurrent tasks, reduce

desired_concurrent_number

in your

CREATE ROUTINE LOAD

statements. Summary Checklist: * Check

/mem_tracker

on the CN node to see if

load

or

storage

(metadata) is the top consumer. * Lower

load_process_max_memory_limit_percent

to 30-40% if loading is crowding out the node. * Ensure your CN pods have enough memory requested/limited in the Helm chart or Operator YAML (typically at least 16GB+ is recommended for production Kafka ingestion). References • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md

@Rocky is it possible to be also affected of this setup for creating a table like this? PRIMARY KEY(

id

) DISTRIBUTED BY HASH(

id

) BUCKETS 10 PROPERTIES ( "replication_num" = "1" );

Hello! I'm Rocky. Looking at your table configuration, there are a few key areas where this setup might "affect" your cluster's performance or reliability. Here is a breakdown of the potential impacts of your current

CREATE TABLE

properties: *1.

replication_num" = "1"

(High Risk)* * Data Loss Risk: Setting replication to 1 means there is only one copy of your data. If the Backend (BE) node storing that data goes down or the disk fails, the data becomes unavailable, and you may suffer permanent data loss. * Recommendation: For production environments, it is strongly recommended to use

"replication_num" = "3"

to ensure high availability and fault tolerance. *2.

BUCKETS 10

(Static vs. Automatic)* * Scaling Issues: By manually setting 10 buckets, you are fixing the parallelism for this table. * If your data grows to hundreds of GBs or TBs, 10 buckets will be too few, leading to very large tablets (ideally, tablets should be kept between 1GB and 10GB). * If you have a large cluster (e.g., 20 BE nodes), some nodes will be idle during queries because there aren't enough buckets to distribute across the whole cluster. * Recommendation: If you are on StarRocks 2.5.7 or 3.0+, you can omit the

BUCKETS

keyword entirely to use Automatic Bucketing. StarRocks will then automatically tune the number of buckets based on the data volume. *3. Primary Key Table Memory Usage* * Index Cost: Since this is a

PRIMARY KEY

table, StarRocks maintains a Primary Key Index to ensure uniqueness. * Memory vs. Disk: By default, in recent versions, the

persistent_index

is enabled (

"enable_persistent_index" = "true"

). If it is not enabled, the entire index must reside in memory, which can lead to high RAM consumption if your

id

column has billions of rows. * Recommendation: Ensure you are using the Persistent Index if you expect a high row count to save memory. *4. Distribution by `id`* * The Good: Using

id

as a hash distribution key is generally excellent because it usually has high cardinality, ensuring data is spread evenly across buckets (preventing "data skew"). * The Caveat: If most of your queries filter by a different column (like

user_id

or

timestamp

) and never by

id

, StarRocks will have to scan all 10 buckets for every query. Suggested Improved Setup: If this is for a production-like environment, consider this version:

sql
PRIMARY KEY(`id`)
DISTRIBUTED BY HASH(`id`) -- Let StarRocks handle bucket count automatically
PROPERTIES (
    "replication_num" = "3",
    "enable_persistent_index" = "true"
);

Does this match the specific "effect" or issue you were concerned about? Let me know if you're seeing a specific error or performance bottleneck! References • table_design/data_distribution/Data_distribution.md • best_practices/bucketing.md • best_practices/primarykey_table.md • table_design/table_types/primary_key_table.md