Hi all and @Rocky, is cache not at all shared across CNs or FE doesn't schedule queries on CN keeping caching in account in Data shared architecture? I found cache sharing only in the cluster scale up operation. Couldn't found in any official doc, any idea?

In StarRocks' shared-data architecture (Compute Nodes/CNs), query scheduling and caching work together to optimize performance, but they follow specific rules: 1. Does the FE consider the cache when scheduling? Yes. The Frontend (FE) uses consistent hashing to schedule query tasks to CNs in a shared-data cluster. * By mapping specific data ranges (tablets) to specific CNs consistently, StarRocks ensures that a query accessing the same data will likely be routed to the same node every time. * This maximizes the local cache hit rate because the data needed for the query is already residing on that CN's local disk. 2. Is the cache shared across CNs? By default, caches are local to each CN and not globally shared for every query. However, there is a specialized feature called Cache Sharing: * The Feature: Controlled by the session variable

enable_datacache_sharing

(introduced in v3.4/v3.5+). When enabled, a CN can fetch cached data from a peer CN via the network if it doesn't have the data locally. * The "Scaling" Constraint: As you suspected, this is primarily intended for cluster scaling operations. The variable

datacache_sharing_work_period

(default 600 seconds) defines the window after a scaling event during which nodes will attempt to pull cache from their peers. * The Goal: This prevents "performance jitter" after you add or remove nodes. Without it, new nodes would have "cold" caches and would need to fetch everything from slow object storage (S3/HDFS). With it, they can quickly "warm up" by pulling from the existing nodes that previously held that data hash. Summary * Normal Operation: FE uses consistent hashing to send queries to the CN that already has the data in its local cache. * Scale Up/Down: Because the hash ring changes, data ranges move to different nodes. During this time, the Cache Sharing feature allows nodes to share their cached blocks over the network to avoid hitting remote storage. For detailed monitoring, you can check the

DataCacheReadPeerBytes

and

DataCacheReadPeerTimer

metrics in your Query Profile to see if your cluster is currently utilizing peer-to-peer cache sharing.

@Rocky. where is this documented - Yes. The Frontend (FE) uses consistent hashing to schedule query tasks to CNs in a shared-data cluster.?

The mechanism where the Frontend (FE) uses consistent hashing to maintain cache affinity for Compute Nodes (CNs) is primarily documented across three key areas of the StarRocks documentation and technical blogs: 1. System Variables Documentation The specific term "consistent hashing" and its role in scheduling is explicitly mentioned in the System Variables page under the variable introduced in v4.0: *

computation_fragment_scheduling_policy

: When set to

elastic

, it is documented as using consistent hashing to assign buckets to worker nodes. This ensures stable mapping so that the same data is processed by the same CN, maximizing the local cache hit rate. *

enable_datacache_sharing

and

datacache_sharing_work_period

: These variables (introduced in v3.5.1) describe how nodes can "pull" cache from peers over the network to mitigate the performance impact of a hash ring change during scaling. 2. Data Cache Documentation The Data Cache page explains the "Working Principles" where it describes how StarRocks achieves performance comparable to shared-nothing architectures by keeping "hot data" in the local disk cache of CNs. * It notes that during elastic scaling, tablet migration or reassignment occurs. To prevent "cold starts," the Cache Sharing feature (peer-to-peer cache access) is used to bridge the gap while the local cache warms up on the new node assigned to that data range. 3. Technical Deep Dives (Internal Design) The most detailed explanation of the I/O model and how the FE schedules tasks to specific CNs for cache locality is found in the "Deep Dive into StarRocks I/O Model" series (often linked from the official blog or documentation). * These resources explain that while data is stored in remote object storage (S3/HDFS), the FE maintains a logical mapping of Tablets to CNs. * To ensure high performance, the scheduler tries to keep this mapping stable. If you query the same tablet twice, the FE will route the request to the same CN so it can serve the data from its local NVMe/SSD cache rather than fetching from remote storage again. Summary of Documentation Links: * Data Cache Working Principles * System Variables Reference (Search for

datacache_sharing

) * Shared-data Architecture Overview