@Rocky How can I control segment size that stored to s3 storage when I use shared-data mode?

In a StarRocks shared-data (cloud-native) cluster, data is stored as segment files in object storage like S3. You can control the size of these files and how they are organized using the following configurations: 1. Control Segment File Size (CN Configuration) The primary parameter for controlling the maximum size of a single segment file is

max_segment_file_size

. This is a configuration for the Compute Nodes (CNs): * Parameter:

max_segment_file_size

* Default:

268435456

(256 MB) * Description: This specifies the maximum size of a single

.segment

file. When a flush or compaction produces data exceeding this limit, it is split into multiple segment files. * How to set: Add or modify this in your

cn.conf

(or

be.conf

if using BEs in shared-data mode).

properties
    max_segment_file_size = 1073741824  # Example: Set to 1GB

2. Control Memtable Flush (CN Configuration) The size of the initial rowset segments written to S3 during data ingestion is influenced by the memtable size. If the memtable is small, many small files will be generated. * Parameter:

write_buffer_size

* Default:

104857600

(100 MB) * Description: This defines the size of the memory buffer before it is flushed to storage. Increasing this can help produce larger initial segment files and reduce the number of small files on S3. 3. Compaction (FE & CN Configuration) In shared-data mode, compaction is the process that merges smaller files into larger ones to optimize query performance and S3 storage costs. * lake_compaction_max_tasks (FE): Controls how many concurrent compaction tasks the FE initiates. * max_cumulative_compaction_num_singleton_deltas (CN): Controls how many segments are merged in a single cumulative compaction task. Increasing this (e.g., to 100 or higher) allows for more aggressive merging into larger files. * lake_pk_compaction_max_input_rowsets (CN): For Primary Key tables, this limits the number of rowsets merged per task. 4. Tablet/Bucket Sizing (Table Level) The total size of data in S3 is partitioned into tablets based on your

DISTRIBUTED BY

clause. * Best Practice: Aim for a tablet size between 1GB and 10GB (compressed). * If your tablets are too small (e.g., only a few MBs), you will end up with a huge number of small files in S3, which increases AWS S3 API costs and metadata overhead. You can adjust this by changing the bucket count (

BUCKETS

) in your table definition. 5. Storage Page Size (Table Property) While this doesn't change the overall file size, it changes the internal granularity of the data blocks within the segment. * Property:

"storage_page_size" = "16384"

(16KB default) * This can be set during

CREATE TABLE

to optimize for specific read patterns. Recommendation: If you are seeing too many small files in S3, start by reducing the number of buckets (to increase tablet size) and ensuring compaction is running effectively. If you need larger individual files, increase

max_segment_file_size

and

write_buffer_size

in your CN configuration. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md

@Rocky write_buffer_size is uncompressed data size. so can real file size that stored in s3 be more smaller?

Yes, your understanding is correct. The

write_buffer_size

represents the uncompressed size of the data held in the MemTable in memory. Consequently, the actual file size stored in S3 is typically much smaller than this buffer size for several reasons: 1. Compression and Encoding When the data in the MemTable reaches the

write_buffer_size

(default 100 MB), StarRocks flushes it to storage. During this flush process: * Encoding: Data is converted into a columnar format and encoded (e.g., Dictionary encoding, Run-length encoding, Bitshuffle). * Compression: The encoded data is then compressed using algorithms like LZ4 (default), ZSTD, or Snappy. * Result: Depending on the data's cardinality and type, 100 MB of uncompressed memory data often shrinks to 10–30 MB (or even less) on S3. 2. Relationship with

max_segment_file_size

It is important to distinguish between the flush trigger and the file size limit: * `write_buffer_size`: Controls when memory is emptied. It usually results in a relatively small "initial" segment file. * `max_segment_file_size`: Controls the maximum allowed size of an individual

.segment

file on S3 (default 256 MB). * If a single flush or a compaction process produces a data block that (after compression) exceeds this limit, StarRocks will split it into multiple segment files. * Because flushes from the MemTable are usually smaller than 256 MB after compression, you will often see many small files in S3 immediately after ingestion. 3. How to achieve larger files on S3 If your goal is to prevent having many small files on S3 (which increases costs and slows queries): 1. Increase `write_buffer_size`: You can increase this (e.g., to 200MB or 500MB) to allow more data to accumulate in memory before a flush, creating larger initial files. 2. Rely on Compaction: This is the most important mechanism. StarRocks periodically merges these small flushed segments into larger ones in the background. You can tune

max_cumulative_compaction_num_singleton_deltas

to merge more segments at once. 3. Adjust Bucket Count: Ensure your table's bucket count isn't too high. If data is spread across too many buckets (tablets), each bucket will only receive a tiny amount of data per flush, leading to small files regardless of your buffer settings. Summary: Yes, the real file size on S3 will be smaller than the

write_buffer_size

due to compression. To control the final size of files on S3, you should focus on compaction settings and *tablet/or increasing the

write_buffer_size

and `max_segment_file_size`* in your CN configuration. References • administration/management/resource_management/Memory_management.md • administration/management/compaction.md • administration/management/BE_configuration.md • knowledge/admin_and_operation/shared_data_compaction.md