@Rocky sugguest tunning config for starrocks deploy share data on high latency s3 compatible

When deploying StarRocks in shared-data mode on high-latency S3-compatible storage, tuning is focused on reducing the impact of round-trip times (RTT) and maximizing throughput via parallelism and caching. 1. Data Cache Tuning (Critical) Since remote I/O is slow, the local Data Cache is your most important performance lever. * Enable and Expand Cache: *

starlet_use_star_cache = true

(BE/CN config): Enables the local data cache. * `starlet_star_cache_disk_size_percent = 80-90`: Use a high percentage of your local SSD for caching hot data. * Control Data Eviction: * `datacache.partition_duration`: (Table Property) Set this to the time range of your "hot" data (e.g.,

7d

). This prevents cold data from evicting hot data from the cache. * Asynchronous Population: * `SET GLOBAL enable_datacache_async_populate_mode = true;`: This allows queries to return immediately from remote storage while background threads populate the local cache, hiding latency for subsequent queries. 2. Network & SDK Timeouts High-latency storage often suffers from jitter. Adjusting timeouts prevents premature task failures. * BE/CN Configurations: * `object_storage_connect_timeout_ms = 10000`: Increase from default (usually SDK default) to handle slow handshake. * `object_storage_request_timeout_ms = 60000`: Increase for large object reads/writes that might take longer over high-latency links. 3. Parallelism and Worker Threads To saturate a high-latency link, you need more concurrent I/O operations. * Table Creation: *

create_tablet_worker_count = 16

(or higher): High latency drastically slows down the creation of many tablets (as each requires a metadata write). Increasing this speeds up

CREATE TABLE

. * Compaction & Loading: * `lake_compaction_stream_buffer_size_bytes = 4194304`: (4MB) Increase this to improve compaction throughput by reading larger chunks from S3. * `upload_buffer_size = 8388608`: (8MB) Increasing the buffer size for uploads helps maintain throughput when RTT is high. 4. Storage Design (Avoid Small Files) Every file operation on S3 (PUT/GET/LIST) incurs a latency penalty. * Avoid Over-Bucketing: Each bucket creates at least one segment file. On high-latency S3, aim for larger segments (1GB–5GB per tablet). * Rule of thumb:

Total Partition Size / (1 to 5 GB)

= Number of Buckets. * Batch Loads: Avoid high-frequency, small-batch

ROUTINE LOAD

or

STREAM LOAD

. Larger batches reduce the number of small files and the overhead of transaction commits to object storage. 5. Monitoring To verify if latency is your bottleneck, monitor these specific metrics in the Starlet Dashboard: * `fslib_read_io_latency`: Shows the actual S3 read RTT. * `fslib_cache_hit_ratio`: If this is low, your query performance will be bound by S3 latency. * `CompressedBytesReadRemote`: High values indicate you are fetching too much from S3 rather than the local cache. For persistent issues, you may also consider using Data Cache Warmup (

CACHE SELECT

) to pre-load hot data before users start querying. References • administration/management/BE_configuration.md

@Rocky sugguest tunning config for starrocks deploy share data on high latency s3 compatible. increase threadpool size to get s3, does it good?

Yes, increasing the thread pool size for S3 access is a recommended and effective strategy for shared-data deployments on high-latency S3-compatible storage. High latency (high Round Trip Time) significantly limits the throughput of a single thread. By increasing the number of concurrent threads, you can "hide" this latency and saturate the available network bandwidth. 1. Recommended Thread Pool & Concurrency Settings To improve performance on high-latency S3, focus on these

be.conf

(or

cn.conf

) parameters: *

scanner_thread_pool_thread_num

: * Default: 48. * Tuning: Increase to 64 or 128 if you observe low CPU usage but slow query performance on remote reads. This pool handles the scan tasks that fetch data from S3. *

create_tablet_worker_count

: * Default: 3. * Tuning: Increase to 16 or 32. In shared-data mode, every new tablet requires writing a metadata file to S3. High latency makes this extremely slow during

CREATE TABLE

statements unless you increase parallelism. *

number_tablet_writer_threads

&

flush_thread_num_per_store

: * Tuning: Set both to 32 for better ingestion throughput. This allows more concurrent uploads to S3 during data loading. 2. Crucial Cache Tuning (The Latency "Shield") Increasing threads helps with throughput, but the best way to handle high latency is to avoid S3 altogether via the Data Cache. *

enable_datacache_async_populate_mode = true

: (Global Variable) This is critical. When a query misses the cache, StarRocks will fetch from S3 and return the data to the user immediately, while a background thread populates the cache. This prevents the query from being strictly blocked by the cache-fill process. *

starlet_use_star_cache = true

: Ensure this is enabled (it is the default in v3.2.3+). *

starlet_star_cache_disk_size_percent = 90

: Dedicate as much local SSD as possible to the cache to minimize S3 round trips. 3. Network & SDK Resilience High-latency storage often has higher jitter. Adjust these to prevent timeouts: *

object_storage_request_timeout_ms

: Increase to 60000 (60s) or higher for large object operations. *

starlet_fslib_s3client_connect_timeout_ms

: Increase to 5000 (5s) if you see handshake errors in logs. *

starlet_fslib_s3client_nonread_max_retries

: Set to 10 to handle transient network hiccups common in high-latency environments. 4. Storage Optimization * Reduce Bucket Count: Since every S3 operation (PUT/GET) has a latency penalty, aim for larger files. Target 1GB to 5GB per tablet. Avoid "over-bucketing," which creates thousands of small files that exacerbate latency issues. * Batch Ingestion: Use larger batch sizes for

STREAM LOAD

or

ROUTINE LOAD

to reduce the frequency of S3 commits. Monitoring Tip Check the Starlet Dashboard or metrics: • If

fslib_read_io_latency

is high, your tuning must focus on concurrency (more threads) and caching (higher hit ratio). • If

fslib_cache_hit_ratio

is low, focus on

datacache.partition_duration

settings to keep hot data local data stays cached data hot data` table properties to keep hot data local data "hot." keep hot data local SSD storage SSD storage. • Keep "hot" data in the cache. References • faq/shared_data_faq.md • administration/management/BE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • deployment/shared_data/s3.md