@Rocky how to configure caching on shared cluster in 3.5.19? and whats the difference between 3.5.16 and 3.5.19?

In StarRocks 3.5.19, configuring caching for a shared cluster (shared-data architecture) follows the unified Data Cache mechanism. Since version 3.4.0, StarRocks uses a unified cache instance for both internal cloud-native tables and external catalogs. 1. Caching Configuration (Shared Cluster) To configure the local disk cache on Compute Nodes (CNs) or Backend nodes (BEs), use the following parameters: Node Configurations (CN/BE

be.conf

) *

storage_root_path

: Specifies the directory where the cached data is stored. *

datacache_disk_size

: The absolute disk size (e.g.,

100G

) the cache can use. *

starlet_star_cache_disk_size_percent

: The percentage of disk capacity the cache can use (Default is

80%

). _ _Note: StarRocks uses the greater value between these two settings.* *

starlet_use_star_cache

: Must be set to

true

(Default in recent versions). Table-Level Properties You can control caching behavior when creating or altering a table: *

datacache.enable

: Set to

"true"

(default) to enable local disk caching for the table. *

datacache.partition_duration

: Defines the validity duration (TTL) of the hot data (e.g.,

"1 MONTH"

or

"7 DAY"

). If not specified, all data is eligible for caching. Cache Sharing (Scaling Optimization) In version 3.5, you can enable Cache Sharing to prevent performance degradation during cluster scaling. This allows nodes to fetch cache from peers if a local miss occurs: * FE Config:

enable_trace_historical_node = true

* System Variable:

SET GLOBAL enable_datacache_sharing = true;

────────── 2. Differences: 3.5.16 vs. 3.5.19 While both are patch releases in the 3.5 branch, 3.5.19 introduces several stability and observability improvements over 3.5.16. Feature / Area | StarRocks 3.5.16 Improvements | StarRocks 3.5.19 Improvements JSON Handling | Added constant folding for

regexp_replace

. |

parse_json

now respects

ALLOW_THROW_EXCEPTION

(errors vs. NULL). Observability | Added thread names to thread stack utility; clearer

WarehouseManager

logs. | Added histogram metrics for slow locks (

slow_lock_held_time_ms

, etc.). Locking | Standard metadata locking. | Optimized replica locks by moving from Database-level to Table-level in report/clone flows. Ingestion | Supports pausing Routine Load on non-retryable errors. |

FILES()

and Broker Load now honor

isAdjustedToUTC=false

for Parquet timestamps. Performance | Basic PostgreSQL external table improvements. | Optimized scans on

information_schema

views; pushed down predicate computation to BE. Summary Recommendation: If you are on 3.5.16, upgrading to 3.5.19 is recommended for the improved metadata lock efficiency (reducing contention in large clusters) and better diagnostic metrics for troubleshooting slow queries. References • release_notes/release-3.5.md

@Rocky so in both versions I should configure data cache via same variables?

Yes, you use the exact same variables to configure the Data Cache in both StarRocks 3.5.16 and 3.5.19. There have been no changes to the primary configuration parameters between these two patch versions. Shared Configuration (3.5.16 and 3.5.19) For both versions, you configure the cache on your Compute Nodes (CN) or Backends (BE) using these parameters in `be.conf`: * Path:

storage_root_path

(Cache data is stored in the

/datacache

sub-directory of this path). * Size:

datacache_disk_size

(absolute size, e.g.,

200G

) and

starlet_star_cache_disk_size_percent

(percentage, default

80

). StarRocks will use the larger of the two values. * Enablement:

starlet_use_star_cache

and

datacache_enable

(both default to

true

in the 3.5 series). Minor Differences & "Under the Hood" Changes While the variables are the same, 3.5.19 includes several bug fixes and stability improvements over 3.5.16 that affect how the cache and metadata behave: 1. Metadata Cache Refinement: Both versions benefit from the

CacheOptions

introduced in 3.5.9, which provides more stable metadata caching for cloud-native tables. 2. Locking Efficiency: 3.5.19 includes a significant optimization that moves certain locks from the Database-level to the Table-level. This reduces resource contention during metadata operations, which can indirectly improve cache consistency and performance in high-concurrency environments. 3. Observability: 3.5.19 introduces new histogram metrics for tracking "slow locks" (

slow_lock_held_time_ms

). This makes it easier to diagnose if cache or metadata operations are being delayed by locking issues compared to 3.5.16. 4. Fix for "Aggressive Disabling": A bug (fixed in 3.5.9, so present in both .16 and .19) where the cache would sometimes disable itself if it thought the disk was too small is resolved, ensuring the configuration you set is more reliably honored. Summary: You do not need to change your configuration approach when moving from 3.5.16 to 3.5.19. The latter simply offers better stability and better tools for monitoring the performance of that cache. References • release_notes/release-3.5.md

@Rocky datacache_disk_size if its 0 it means that after consuming cache memory it want use disk as a cache?

Not exactly. Setting

datacache_disk_size = 0

does not cause the system to automatically spill or "failover" to memory cache. In StarRocks, memory cache (Page Cache) and disk cache (Block Cache) are managed as two distinct layers. Here is how the behavior works in version 3.5.19 for a shared cluster: 1. The "Greater Of" Rule In a shared cluster, the actual disk cache size is determined by the larger of two variables: *

datacache_disk_size

(Absolute value, e.g.,

100G

) *

starlet_star_cache_disk_size_percent

(Percentage of disk, default is

80

) If you set

datacache_disk_size = 0

but leave the percentage at the default

80

, StarRocks will still use 80% of your disk for caching. To truly set the disk cache to zero, you would need to set both parameters to

0

. 2. Memory vs. Disk is Not Automatic * Memory Cache (

datacache_mem_size

): This is the Page Cache. It stores small, frequently accessed pages. If this is full, it evicts old pages; it does not "use disk instead" unless the data was already destined for the Block Cache. * Disk Cache (

datacache_disk_size

): This is the Block Cache. It stores larger data blocks from remote storage (S3/HDFS). If you successfully set the disk cache to 0, you are essentially disabling the local disk cache. The system will not compensate by using more memory for those large blocks; instead, every query will be forced to fetch data directly from the remote object storage (S3/HDFS), which will significantly slow down performance. 3. How to check what is actually happening You can verify the current active limits by running:

sql
SHOW BACKENDS\G
-- or
SHOW COMPUTE NODES\G

Look at the

DataCacheMetrics

field. It will show something like:

Status: Normal, DiskUsage: 44MB/1TB, MemUsage: 0B/0B

The second number in

DiskUsage

(e.g.,

1TB

) is the actual limit currently being applied by the system. Recommendation If you want to use only memory for caching (which is generally discouraged for shared clusters because object storage latency is high), you would need to: 1. Set

datacache_disk_size = 0

2. Set

starlet_star_cache_disk_size_percent = 0

3. Increase

datacache_mem_size

to your desired memory limit. Warning: Most shared-data workloads are too large for memory-only caching. It is highly recommended to keep a large disk cache to avoid "Remote I/O" bottlenecks." References • data_source/data_cache.md • using_starrocks/caching/block_cache.md References • data_source/data_cache.md • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md • knowledge/usage/best-practice.md