http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> How can I ingest the data to bucket...
# questions-and-troubleshooting
m
@Rocky How can I ingest the data to bucket storage in shared data efficiently with very less bucket operations using starrocks? What configs/optimizations to consider to reduce bucket operations without much compromising performance ?
r
To ingest data efficiently into StarRocks shared-data (compute-storage decoupled) mode while minimizing bucket operations (API calls like 
PUT
, 
GET
, 
LIST
), you should focus on batching, segment sizing, and compaction management. Here are the key optimizations and configurations to consider: 1. Enable Batch Transaction Commits In shared-data mode, every ingestion commit involves metadata operations in the object store. If you have high-concurrency ingestion (e.g., many small Stream Loads), this can lead to a high volume of metadata-related bucket operations. * Optimization: Enable the batch publish version feature. This allows the Frontend (FE) to batch multiple transaction commits together, significantly reducing the overhead on the object store. * Configuration (FE):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
2. Increase Segment File Size By default, StarRocks writes data in segments. If segments are too small, you will trigger a high number of 
PUT
operations during ingestion and more 
LIST/GET
operations during queries. * Optimization: Increase the maximum segment size to ensure fewer, larger files are written to the bucket. * Configuration (CN/BE):
Copy code
sql
    -- Check your current be_configs via:
    -- UPDATE information_schema.be_configs SET value = "1073741824" WHERE name = "max_segment_file_size";
(Standard recommendation is 256MB to 1GB depending on your data volume and memory). 3. Optimize Compaction for Shared-Data Compaction is the process of merging small files into larger ones. While it reduces the file count (good for long-term efficiency), aggressive compaction can also cause a spike in 
GET
(reading old files) and 
PUT
(writing merged files) operations. * Optimization: Use a balanced compaction strategy. Reducing the number of segments merged in a single task can make compaction more incremental and less resource-intensive. * FE Configurations: • `lake_compaction_max_tasks`: Controls how many compaction tasks run concurrently (default is adaptive). * CN/BE Configurations: • `max_cumulative_compaction_num_singleton_deltas`: Set this to a lower value (e.g., 
100
) to accelerate compaction tasks and keep file counts under control without causing massive I/O spikes.
Copy code
sql
    UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
4. Efficient Loading Methods * Stream Load / Pipe: If you are using Stream Load, try to batch your data on the application side before sending it to StarRocks. Aim for 100MB+ per load. * Asynchronous Data Ingestion: Shared-data mode supports asynchronous ingestion to object storage. This allows CNs to return a "success" to the client while the data is still being finalized in the bucket, improving throughput. 5. Data Cache (To reduce GET operations) While this doesn't reduce ingestion bucket operations, it significantly reduces 
GET
operations during the "read-after-write" phase or during compaction. * Optimization: Ensure your CN nodes have sufficient local NVMe/SSD disk space for the 
starrocks_cache
to avoid repeated bucket reads. Summary Checklist Feature | Parameter | Recommended Value Transaction Batching | 
lake_enable_batch_publish_version
| 
true
Segment Size | 
max_segment_file_size
| 
1GB (1073741824)
Compaction Batch | 
max_cumulative_compaction_num_singleton_deltas
| 
100
Concurrency | 
lake_compaction_max_tasks
| 
-1
(Adaptive) Note: Always monitor your bucket operation costs and latency via the object storage provider's console (e.g., CloudWatch for S3) after applying these changes to find the "sweet spot" for your specific workload. References • knowledge/admin_and_operation/shared_data_compaction.md
m
@Rocky If I use kafka sink connector of starrocks, along with provided optimizations, what else can i use?
r
When using the StarRocks Kafka Sink Connector, you have a variety of complementary features and optimizations available to enhance your data pipeline. Since the connector integrates with the Kafka Connect framework and uses Stream Load under the hood, you can leverage the following: 1. Kafka Connect Framework Features * Converters: You can use standard Kafka Connect converters for Avro, Protobuf, and JSON (with or without schemas). This allows you to handle complex data formats that are not supported by Routine Load. * SMTs (Single Message Transforms): Use built-in or custom SMTs to transform, filter, or rename fields in your messages before they reach StarRocks. * Debezium Integration: Pair the sink connector with Debezium source connectors for a robust CDC (Change Data Capture) pipeline from databases like MySQL, PostgreSQL, or Oracle. 2. StarRocks Table & Loading Optimizations * Primary Key Tables: For CDC scenarios, loading into a Primary Key table allows you to perform real-time updates and deletes. * Partial Updates: You can enable partial updates via 
sink.properties.partial_update
. * Row Mode: Best for real-time updates with many columns but small batches. * Column Mode: Optimized for batch updates where only a few columns (e.g., 10%) are being updated across many rows. * Stream Load Parameters: Any parameter supported by StarRocks Stream Load can be passed through the 
sink.properties.*
prefix (e.g., 
sink.properties.columns
, 
sink.properties.where
, 
sink.properties.jsonpaths
). 3. Latency and Throughput Tuning You can fine-tune the "Flush Policy" to balance latency and resource consumption: * 
bufferflush.maxbytes
: Controls the maximum size of data buffered in memory before flushing to StarRocks. * 
bufferflush.intervalms
: Controls the maximum time interval between flushes. * 
<http://offset.flush.interval.ms|offset.flush.interval.ms>
: Adjusts how often Kafka Connect commits offsets, which also triggers a flush in the connector. 4. Post-Ingestion Processing * Asynchronous Materialized Views: Once data is loaded into StarRocks, use Async MVs to perform complex joins, aggregations, or multi-stage transformations in a cost-efficient way. * Generated Columns: Use these to pre-calculate values or extract nested JSON fields during the loading process to speed up downstream queries. 5. Monitoring * JMX Metrics: Since it's a Kafka Connect plugin, you can monitor standard JMX metrics for task status, error counts, and processing lag. * StarRocks FE/BE Metrics: Monitor 
Stream Load
metrics on the StarRocks side to see ingestion rates and transaction success/failure rates. If you need even more complex stream processing (like windowed aggregations or joining two Kafka streams before ingestion), you might consider using the StarRocks Flink Connector as an alternative to the Kafka Sink Connector. References • loading/Kafka-connector-starrocks.md • loading/Flink-connector-starrocks.md
m
@Rocky on top of this to reduce bucket operations, what else can I do from starrocks sink connector? https://starrocks.slack.com/archives/C02FACZSNJV/p1771090608262809?thread_ts=1771090583.753839&amp;cid=C02FACZSNJV
r
To reduce bucket operations (specifically S3/GCS/Azure Blob operations like PUT, LIST, and DELETE) when using a StarRocks Sink Connector (such as Flink or Kafka), the primary goal is to maximize the amount of data per load and minimize the frequency of commits. Here are the specific configurations you can tune beyond basic buffering: 1. Increase Buffering Thresholds The connectors buffer data in memory and trigger a Stream Load when a threshold is met. To reduce the number of loads (and thus bucket operations): * Flink Connector: * `sink.buffer-flush.max-bytes`: Increase this (e.g., to 
512MB
or 
1GB
) to ensure each load is substantial. * `sink.buffer-flush.interval-ms`: Increase the flush interval (e.g., to 
60000
or 
300000
ms) to prevent frequent small loads during low-traffic periods. * Kafka Connector: * `bufferflush.maxbytes`: Set to a higher value. * `bufferflush.intervalms`: Increase the time between flushes. 2. Tune Connector Parallelism High parallelism in your Flink or Kafka Connect job splits the data into many smaller streams. If you have 20 parallel sink tasks, each task buffers data independently. * Action: Reduce the parallelism of the sink operator to the minimum required to handle your throughput. Fewer parallel tasks mean more data is aggregated per task before a flush is triggered. 3. Adjust Checkpoint Intervals (Exactly-once) If you are using exactly-once semantics in Flink: * The connector flushes data to StarRocks every time a Flink checkpoint is triggered. * The 
buffer-flush
settings are ignored in this mode. * Action: Increase the Flink checkpoint interval (e.g., from 1 minute to 5 or 10 minutes). This is the most effective way to reduce bucket operations for exactly-once sinks. 4. Optimize Table Schema (Bucketing) The number of files created on your cloud storage is proportional to the number of Tablets (buckets) in your StarRocks table. * Action: If your table has too many buckets (e.g., 
DISTRIBUTED BY HASH(...) BUCKETS 100
), each Stream Load will attempt to write to many tablets, creating many small files. * Reduce the bucket count to a reasonable number based on your data volume (aim for ~1GB to 10GB per tablet). 5. Shared-Data Mode Specifics If you are running StarRocks in Shared-data mode (where data lives directly on S3/cloud storage): * Compaction: StarRocks performs compaction to merge small files. You can tune the FE/BE configs 
storage_flood_stage_usage_percent
and compaction threads, but usually, the connector-side buffering is the most impactful lever. * Sink Properties: You can pass 
sink.properties.compression
(e.g., 
lz4
) to reduce the size of the data being uploaded, which can slightly improve the efficiency of bucket PUT operations. 6. Kafka Connector: Offset Flush Interval In the StarRocks Kafka Connector, flushes can also be triggered by the Kafka Connect framework itself. * Action: Increase 
<http://offset.flush.interval.ms|offset.flush.interval.ms>
in your Kafka Connect worker configuration (default is often 60000ms). Summary Recommendation: Start by increasing 
sink.buffer-flush.max-bytes
to 256MB+ and decreasing your sink parallelism. If using Flink exactly-once, your biggest lever is increasing the checkpoint interval. References • loading/Flink-connector-starrocks.md • loading/Spark-connector-starrocks.md • loading/Kafka-connector-starrocks.md
m
r
StarRocks has introduced several key architectural optimizations specifically designed to "escape the small file trap" and reduce the overhead of bucket-level operations, especially during high-concurrency or bulk ingestion. Based on the blog post you mentioned and recent StarRocks releases, here are the core built-in optimizations: 1. Merge Commit (v3.4+) This is a major optimization for high-concurrency, small-batch loading (like real-time Stream Loads). * How it works: StarRocks can now automatically merge multiple concurrent Stream Load requests into a single transaction and a single data version. * Why it helps: In previous versions, every single load created a new rowset (file) for every tablet involved. If you had 100 concurrent loads and 50 tablets, you'd create 5,000 "bucket operations" and 5,000 small files. Merge Commit batches these into one version, drastically reducing the number of files, I/O operations, and metadata pressure on the FE (Leader). * Configuration:
Copy code
sql
    -- Available in FE config or table properties in v3.4+
    "merge_commit_interval_ms" = "1000" -- Batches loads every 1 second
2. Random Bucketing with Auto-split (
bucket_size
) Traditional Hash Bucketing requires you to pick a fixed number of buckets upfront, which often leads to "over-bucketing" (too many small tablets) for small tables/partitions. * How it works: Since v3.2, if you use Random Bucketing and set the 
bucket_size
property (e.g., 1GB or 4GB), StarRocks starts with a minimal number of tablets. As data grows, it dynamically splits tablets. * Why it helps: It prevents the "Small File Trap" at the source by ensuring you don't have 100 empty or near-empty tablets for a new partition. It ensures that data is only distributed across more buckets when the volume actually justifies it. 3. Automatic Bucketing * How it works: Introduced in v2.5.7, StarRocks can automatically determine the 
BUCKETS
count based on cluster resources (number of BE nodes) and the data volume in previous partitions. * Why it helps: It removes the guesswork that leads to users setting static bucket counts that are either too high (creating too many small files) or too low (limiting parallelism). 4. Shared Data: Transaction Commit Batching For StarRocks' shared-data (compute-storage separation) architecture: * Optimization: 
lake_enable_batch_publish_version
. * Effect: This batches the "publish version" tasks to object storage. Instead of every tablet updating its metadata independently, StarRocks batches these commits, which is critical for maintaining performance when ingesting into tables with many tablets on high-latency object storage (like S3). 5. Tablet Internal Parallelism * How it works: By setting 
enable_tablet_internal_parallel = true
, StarRocks can use multiple threads to scan a single large tablet. * Why it helps: This allows you to follow the "fewer but larger tablets" best practice (e.g., aiming for 10GB+ per tablet) to reduce metadata overhead and "bucket operations," without sacrificing query performance, because the system no longer relies solely on the number of tablets for parallelism. Summary of Benefits: By combining Merge Commit with Automatic/Dynamic Bucketing, StarRocks reduces the "write amplification" at the bucket level. This results in: * Lower CPU/Memory usage on BEs (less compaction work). * Reduced IOPS on storage (fewer, larger files). * Lower Metadata pressure on the FE, allowing the cluster to scale to millions of tablets more easily. References • faq/resource_isolation_faq.md • knowledge/usage/load/shared_data_batch_ingestion.md • best_practices/bucketing.md
m
@Rocky offset.flush.interval - setting to 1mins and if buffer flush interval set to 5mins, flushing to starrocks happening every 1min with sink connector. even though buffer condition not met, does it flush data to starrocks? also on flushing data does it not store in cn disks and aggregate more data and flush to bucket with large files?
r
Hi there! I'm Rocky. I can help answer your questions about the StarRocks Kafka Sink Connector and how it handles data flushing and storage. 1. Does it flush even if the buffer condition is not met? Yes, it does. The StarRocks Kafka Sink Connector has multiple trigger conditions for flushing data. A flush is triggered whenever any of the following occurs: * The data volume reaches 
bufferflush.maxbytes
. * The time interval reaches 
bufferflush.intervalms
(which you set to 5 minutes). * The Kafka Connect framework triggers an offset commit, which is controlled by the 
<http://offset.flush.interval.ms|offset.flush.interval.ms>
setting (which you set to 1 minute). Because Kafka Connect must ensure that all buffered data is safely written to the destination before it can commit the offsets back to Kafka, the Sink Connector is forced to flush data every time 
<http://offset.flush.interval.ms|offset.flush.interval.ms>
expires. In your case, the 1-minute offset interval will "override" the 5-minute buffer interval. 2. Does it store on CN disks and aggregate more data before flushing to the bucket? How the data is handled once it leaves the connector depends on your StarRocks architecture: * In Shared-Data (Cloud Native) Mode: When the connector flushes, it initiates a Stream Load to a CN (Compute Node). The CN receives the data, processes it into segments, and then writes those segments to the remote object storage (your bucket). * Small Files Issue: If you flush very frequently (e.g., every minute) with small amounts of data, it will result in many small files in your bucket, which can impact query performance and metadata management. * Optimization: To mitigate this, StarRocks supports a feature called Merge Commit. This allows the server to buffer multiple small Stream Load requests into a single transaction and write them as a larger, more optimized file to the bucket. You can enable this by adding 
"sink.properties.merge_commit" = "true"
in your connector configuration (available in newer versions). * CN Disk Usage: The CN uses its local disk primarily for caching (to speed up subsequent reads) and as a temporary landing zone for data being written before it is uploaded to the bucket. It does not "hold" data on local disk indefinitely to aggregate it across multiple separate Load jobs unless you are using the Merge Commit feature on the server side. Summary Recommendation If your goal is to reduce the number of small files and aggregate more data: 1. Increase `offset.flush.interval.ms`: Align it closer to your desired 5-minute window if your application can tolerate the higher latency for offset commits. 2. Enable Merge Commit: Use 
sink.properties.merge_commit
in your Sink Connector config to let StarRocks handle the aggregation of small writes into larger files on the server side. Let me know if you need help with the specific configuration syntax! References • loading/Kafka-connector-starrocks.md • loading/Flink-connector-starrocks.md
m
@Rocky how can I optimize the compaction, give configs to use if I'm ok with latency of 5mins of data, how frequently should I do compaction and give respective configs that I can use in prod to have less bucket operations as well as decent performance
r
To optimize compaction in StarRocks, especially when you are okay with a data latency of around 5 minutes and want to minimize the number of background operations (write amplification), the best approach is to leverage the Size-tiered Compaction Strategy and tune the trigger thresholds. Here are the recommended configurations and strategies for a production environment: 1. Enable and Optimize Size-tiered Compaction Size-tiered compaction is more efficient than the traditional cumulative policy because it reduces the number of times data is rewritten. It is enabled by default in recent versions. * 
size_tiered_level_multiple
: (Default: 5) * Recommendation: Set to 
10
. * Why: This increases the size difference required between levels. Increasing this value reduces "bucket operations" (I/O) because rowsets are merged less frequently, though it may slightly increase the number of files present at any given time. * 
size_tiered_min_level_size
: (Default: 128 KB) * Recommendation: Set to 
64MB
or 
128MB
(e.g., 
134217728
bytes). * Why: Rowsets smaller than this value trigger compaction immediately. If you are okay with 5 minutes of latency, increasing this prevents the system from constantly merging tiny files as they arrive. 2. Frequency Tuning (The "5-Minute" Latency Goal) StarRocks compaction is background-driven. You don't call it on a schedule, but you can control how "lazy" or "aggressive" it is. * 
update_compaction_per_tablet_min_interval_seconds
: (Primary Key tables) * Recommendation: Set to 
300
(5 minutes). * Why: This explicitly prevents the same tablet from being compacted more than once every 5 minutes, directly matching your latency tolerance. * 
lake_compaction_score_selector_min_score
: (Shared-data/Cloud-native clusters) * Recommendation: Set to 
30.0
(Default is 10.0). * Why: The "score" roughly represents the number of unmerged versions. Increasing this threshold ensures the FE only triggers a compaction task when a significant amount of data has accumulated. 3. Production Resource & Performance Configs To ensure compaction performs well without starving your queries of resources, use these BE/CN settings: * 
compact_threads
: * Recommendation: 25% of your total CPU cores. * Command: 
UPDATE information_schema.be_configs SET value = <num_cores_divided_by_4> WHERE name = "compact_threads";
* 
max_cumulative_compaction_num_singleton_deltas
: * Recommendation: 
100
. * Why: This limits the number of segments merged in one go, preventing individual compaction tasks from consuming too much memory or taking too long. * 
lake_compaction_max_tasks
: (FE Config) * Recommendation: 
-1
(Adaptive). * Why: This allows the FE to automatically scale the number of concurrent tasks based on the number of backends in your cluster. Summary of SQL commands for Production: On FE (Global):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_score_selector_min_score" = "30.0");
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
On BE/CN (Apply to nodes):
Copy code
sql
-- Increase the size threshold for small file merging
UPDATE information_schema.be_configs SET value = "134217728" WHERE name = "size_tiered_min_level_size";
-- Reduce rewrite frequency
UPDATE information_schema.be_configs SET value = "10" WHERE name = "size_tiered_level_multiple";
-- Limit task size to avoid OOM
UPDATE information_schema.be_configs SET value = "100" WHERE name = "max_cumulative_compaction_num_singleton_deltas";
-- Match your 5-min latency preference for PK tables
UPDATE information_schema.be_configs SET value = "300" WHERE name = "update_compaction_per_tablet_min_interval_seconds";
Pro-tip: You can monitor the effectiveness of these changes by checking the Compaction Score in your Grafana dashboard or by running 
SHOW PROC '/compactions';
. If the score stays consistently high (e.g., over 100), you may need to make the thresholds slightly more aggressive. References • administration/management/BE_configuration.md • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/monitoring/metrics.md
m
@Rocky any way to reduce the list operations, fslib_list_dir I could these are continuously increasing, how can I reduce this metadata list operations?
r
To reduce the number of metadata list operations (tracked as 
fslib_list_dir
in metrics), you should focus on tuning the Metadata Cache settings of your External Catalogs (Hive, Iceberg, etc.). In StarRocks, these operations typically occur during query planning when the system needs to list files in a directory to determine which data to read, or during background metadata refreshes. Here are the primary ways to reduce these operations: 1. Tune Catalog Refresh Intervals By default, StarRocks asynchronously refreshes the file list for external tables. If your data doesn't change every minute, you can increase the refresh interval in your Catalog properties: * 
remote_file_cache_refresh_interval_sec
: Controls how often StarRocks re-lists the files in the storage system (e.g., S3/HDFS). The default is often 60 seconds. * 
metastore_cache_refresh_interval_sec
: Controls how often the partition/table metadata from Hive Metastore/Glue is refreshed. Example adjustment:
Copy code
sql
ALTER CATALOG hive_catalog SET (
    "remote_file_cache_refresh_interval_sec" = "3600", -- Increase to 1 hour
    "metastore_cache_refresh_interval_sec" = "3600"
);
2. Adjust Background Refresh Frequency StarRocks performs background polling to perceive metadata changes for frequently accessed catalogs. You can slow this down via FE configuration: * 
background_refresh_metadata_interval_millis
: The interval for polling the metastore. Default is 
600000
(10 minutes). Increase this to reduce background activity. * 
background_refresh_metadata_time_secs_since_last_access_secs
: How long a catalog remains "active" for background refreshing. Reducing this will stop refreshing unused catalogs sooner. To modify (FE dynamic config):
Copy code
sql
ADMIN SET FRONTEND CONFIG ("background_refresh_metadata_interval_millis" = "3600000"); -- 1 hour
3. Disable Recursive Listing If your table structure is flat (e.g., Hive partitions without deep sub-folders), ensure 
enable_recursive_listing
is not unnecessarily set to 
true
. Recursive listing forces StarRocks to crawl every sub-directory, significantly increasing 
list_dir
calls. * Set 
"enable_recursive_listing" = "false"
in your Catalog properties if your file structure allows it. 4. Increase Metadata TTL To prevent StarRocks from evicting metadata and having to re-list directories upon the next query, increase the Time-To-Live (TTL): * 
remote_file_cache_ttl_sec
: Increase this (default is 36 hours) to keep the file list in memory longer. * 
metastore_cache_ttl_sec
: Increase this (default is 24 hours) for partition/table metadata. 5. Check for Manual Refresh Spikes If you have automated scripts or Airflow jobs running 
REFRESH EXTERNAL TABLE
, these will bypass the cache and force a new list operation. Ensure these are only triggered when data has actually changed. Summary Recommendation: If your data updates are infrequent (e.g., daily or every few hours), set 
remote_file_cache_refresh_interval_sec
to a much higher value (like 
3600
or higher) to immediately see a drop in 
fslib_list_dir
metrics. References • data_source/catalog/hive_catalog.md • data_source/catalog/iceberg/iceberg_catalog.md
m
@Rocky while writing data to starrocks shared data, is there any way I can reduce the list operations 
fslib_list_dir?
r
To reduce the number of 
fslib_list_dir
(directory listing) operations during data writes in a StarRocks shared-data (compute-node) architecture, you can apply several architectural and configuration-level optimizations. These operations are typically triggered when the system needs to discover tablet metadata or manage data versions in object storage (S3/OSS/GCS/HDFS). 1. Optimize Table Schema (The Most Effective Way) The number of list operations is directly proportional to the number of tablets and partitions. * Reduce Bucket Count: In shared-data mode, you don't need a high number of buckets for performance as you might in shared-nothing mode. Reducing the number of buckets per partition significantly reduces the number of "directories" StarRocks has to list. Aim for larger tablets (e.g., 1GB to 10GB per tablet). * Consolidate Partitions: Avoid creating excessive tiny partitions. Each partition and tablet combination requires its own metadata management in the storage backend. 2. Enable List Caching (BE/CN Configuration) StarRocks provides internal caching mechanisms for the StarOS/Starlet layer (the shared-data backend) to reduce redundant object storage calls. * 
starlet_list_cache_timeout_ms
: You can add/modify this in your 
cn.conf
or 
be.conf
. It controls how long directory listing results are cached in memory. Increasing this value can reduce the frequency of 
fslib_list_dir
. _ _Example:* 
starlet_list_cache_timeout_ms = 3600000
(caches for 1 hour). * 
enable_file_metacache
: Ensure this is enabled (usually 
true
by default in recent versions). It caches file-level metadata which helps avoid re-listing and re-fetching file attributes. 3. Ingestion Tuning If you are seeing these operations during high-frequency small loads: * Batch Your Loads: Instead of many small Stream Loads, batch data into larger transactions. Each load commit in shared-data mode involves metadata updates that may trigger listings to ensure version consistency. * Optimize Compaction: High 
fslib_list_dir
frequency is often related to background compaction checking for stale versions. Monitor your compaction scores (
show proc '/dbs/<db_id>/<table_id>/partitions'
) and adjust 
cumulative_compaction_num_threads_per_disk
or 
base_compaction_num_threads_per_disk
if necessary, though reducing tablet count (Point 1) is usually the better fix here. 4. FE Metadata Caching * 
lake_enable_list_cache
: In some StarRocks versions (v3.x+), you can enable this FE configuration to allow the Frontend to cache listing results for shared-data tablets, reducing the RPC calls to the object storage during planning and commit phases. Summary of Recommended Configs (CN/BE):
Copy code
properties
# Increase listing cache timeout (Unit: ms)
starlet_list_cache_timeout_ms = 600000

# Ensure file metadata cache is active
enable_file_metacache = true
By increasing the cache TTL and reducing the total number of tablets, you should see a significant drop in 
fslib_list_dir
operations. References • deployment/shared_data/feature-support-shared-data.md • data_source/feature-support-data-lake-analytics.md • using_starrocks/caching/block_cache.md • administration/management/FE_configuration.md • data_source/data_cache.md
m
@Rocky what can I do to reduce bucket list/write operations when ingesting data through stream load http api. to have very less bucket operations and directly upload big files with compacted. suggest configs to add
r
To reduce bucket list and write operations (such as S3 
PutObject
, 
ListBucket
, etc.) when using Stream Load in a shared-data (cloud-native) architecture, the goal is to batch data as much as possible before it hits the object storage and reduce the frequency of metadata commits. Here are the suggested configurations and best practices to achieve this: 1. Increase Batch Size at the Source (Primary Recommendation) The most effective way to reduce bucket operations is to increase the amount of data sent in a single Stream Load request. Instead of many small files, send fewer, larger files. * Target Size: Aim for 100 MB to 1 GB per Stream Load. * Header Configuration: If you send large files, increase the timeout to prevent premature disconnection.
Copy code
bash
    curl --location-trusted -u <user>:<password> \
        -H "timeout: 3600" \
        -H "column_separator:," \
        -T large_file.csv \
        http://<fe_host>:<fe_http_port>/api/<db>/<table>/_stream_load
* System Config: Ensure the BE can handle large bodies (default is usually high enough, but check 
streaming_load_max_mb
). 2. Enable Transaction Batching (FE Config) Introduced in StarRocks v3.2.3, this configuration allows the Frontend (FE) to batch multiple transaction commits together, significantly reducing the metadata overhead and S3 operations associated with "publishing" a new version. * Config: 
lake_enable_batch_publish_version
* Setting:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
3. Reduce the Number of Buckets (Table Design) In shared-data mode, every bucket (tablet) generates its own set of files and metadata in the object store. If you have 100 buckets and perform a Stream Load, you might potentially write 100 small files. * Recommendation: Use a smaller bucket count for tables with frequent ingestion. Aim for 1 bucket per 1–5 GB of data per partition. * Action: When creating or altering the table, set 
DISTRIBUTED BY HASH(...) BUCKETS <smaller_number>
. 4. Adjust Segment Size (BE Config) StarRocks flushes data into segment files. If you want larger files directly in the object store, you can increase the segment size limit. * Config: 
max_segment_size
* Default: 256 MB * Action: Increase this if you want larger individual files in S3 (e.g., to 512MB or 1GB).
Copy code
sql
    -- On BE/CN nodes
    update information_schema.be_configs set value = "1073741824" where name = "max_segment_size";
5. Tune Ingestion Threads (BE Config) To handle the higher throughput of larger batch uploads efficiently, you can increase the threads dedicated to flushing and writing: * `flush_thread_num_per_store`: Increase to 32 (default varies). * `number_tablet_writer_threads`: Increase to 32.
Copy code
sql
    update information_schema.be_configs set value = "32" where name = "flush_thread_num_per_store";
    update information_schema.be_configs set value = "32" where name = "number_tablet_writer_threads";
6. Background Compaction In shared-data mode, StarRocks handles compaction automatically to merge small files. While you want to minimize their creation, ensure compaction is healthy so that any remaining small files are merged quickly. * Check the status using: 
ADMIN SHOW TABLET STATUS FROM <table>;
Summary of Configs to Add: Level | Parameter | Value | Description FE | 
lake_enable_batch_publish_version
| 
true
| Batches metadata commits. BE | 
max_segment_size
| 
1073741824
| 1GB segment files. BE | 
flush_thread_num_per_store
| 
32
| Increases write concurrency. Table | 
BUCKETS
| Small count | Reduces the number of tablets/files. Load | 
timeout
| 
3600
| Supports longer, larger uploads. References • deployment/shared_data/s3.md
2 Views
Open in Slack
PreviousNext
Powered by