@Rocky what happens if one CN gets overloaded because the queries for one tablet related to that CN is more

In StarRocks, the behavior and mitigation for an overloaded node (CN/BE) due to a "hot tablet" depends on whether you are using Shared-data (Cloud-native) or Shared-nothing architecture. 1. Shared-data Architecture (CN Nodes) In shared-data mode, data is stored in remote storage (S3/HDFS), and CN nodes are stateless compute workers that cache data locally. * Consistent Hashing: To ensure high cache hit rates, StarRocks uses consistent hashing to bind a tablet to a specific CN node. This means all queries for "Tablet A" are routed to the same CN. * The Overload Problem: If "Tablet A" is queried very frequently (a "hotspot"), the specific CN it is hashed to will become overloaded, leading to high CPU/memory usage and increased latency for those queries. * Mitigation Mechanisms: * Multiple Query Replicas: StarRocks can map a single tablet to multiple candidate CN nodes instead of just one. The FE scheduler then picks the node with the lowest current load. _ _Configuration:* Set the FE parameter

lake_service_query_replica_count

(default is usually 1 or 2). Increasing this helps spread the load of a hot tablet across more CNs. * Cache Sharing: If a query is routed to a different CN (e.g., during scaling), that CN can fetch cached data from the "historical" node over the network rather than the remote S3 storage. _ _Configuration:*

enable_datacache_sharing = true

. * Workload Management: You can use Resource Groups to limit the CPU/Memory a specific query or user can consume, preventing one hot tablet from crashing the entire CN. 2. Shared-nothing Architecture (BE Nodes) In this mode, BE nodes own the data replicas on their local disks. * Replica Load Balancing: Each tablet typically has 3 replicas. The FE monitors the load of each BE. If one BE is overloaded, the FE will prefer routing queries to the other two replicas on less-loaded BE nodes. * Tablet Migration: If a BE is consistently overloaded or its disk is too full, the Tablet Scheduler in the FE will automatically migrate tablets from the high-load BE to lower-load BEs in the background. * Adaptive Parallelism: StarRocks can adjust the number of scan threads based on node load to prevent complete exhaustion. 3. Root Cause Mitigation (Data Modeling) Regardless of the architecture, a "hot tablet" is often a symptom of Data Skew. * Bucketing Strategy: If your

DISTRIBUTED BY HASH(column)

uses a column with low cardinality (e.g.,

gender

or

status

), one tablet will hold significantly more data and traffic than others. * Solution: Use a high-cardinality column (like

user_id

or

order_id

) or a combination of columns as the bucketing key to ensure data and queries are distributed evenly across all nodes. * Random Bucketing: Since v3.1, you can use Random Bucketing if you don't need Colocate Joins, which ensures an even distribution of data regardless of key skew. Summary If a CN becomes overloaded because of a hot tablet, the system may slow down or the query might time out. To fix this, you should increase the query replica count for shared-data clusters or optimize your bucketing keys to prevent the hotspot from forming in the first place. References • faq/shared_data_faq.md • administration/management/resource_management/Replica.md • data_source/data_cache.md • quick_start/shared-data.md

@Rocky is cache sharing in data shared model only applicable in cluster scaling?

In the StarRocks shared-data (cloud-native) model, Cache Sharing is primarily designed and optimized for cluster scaling scenarios. While the technology technically allows nodes to fetch data from each other, its implementation is specifically tuned to handle the transitionary period when tablets are re-routed due to adding or removing nodes. Key Reasons it is Scaling-Specific: 1. The

work_period

Limitation: The system variable

datacache_sharing_work_period

(introduced in v3.5.1) defines a window (default 600 seconds) during which cache sharing is active. This timer is triggered only after a cluster scaling operation. Once this period expires, nodes will stop trying to fetch cache from their peers and will revert to reading from remote storage (S3/HDFS) on a local cache miss. 2. Routing Strategy: In a steady state, StarRocks uses consistent hashing to ensure that a specific tablet is always routed to the same CN node. This maximizes the local cache hit rate. Cache Sharing is only needed when that "owner" node changes (due to scaling), leaving the new owner with a cold cache while the old owner still has the data on disk. 3. Performance Trade-offs: Fetching cache over the network from another CN is faster than fetching from S3, but it consumes the source CN's network bandwidth and disk I/O. For this reason, StarRocks does not keep it enabled permanently for standard queries, as it could lead to unpredictable performance on the "source" nodes. How to handle "Hot Tablets" without Scaling? If you are experiencing an overloaded CN due to a "hot tablet" (high query volume on a specific tablet) and you are not scaling, Cache Sharing is not the standard solution. Instead, you should use: * Query Replication (

lake_service_query_replica_count

): This FE configuration allows you to map a single tablet to multiple candidate CN nodes (e.g., 2 or 3). The FE will then pick the node with the lowest load. _ _Note*: In this case, each candidate node will eventually build its own local cache from remote storage. They do not typically share the cache between each other unless a scaling event is active. * Resource Groups: Use Workload Management to limit the CPU/Memory a specific user or query can consume on a CN so one hot tablet doesn't starve other queries. * Optimization: Ensure your

DISTRIBUTED BY

key provides enough entropy so that data is spread across many tablets rather than concentrated in one. Summary While you could theoretically set

datacache_sharing_work_period

to a very large value to make it "permanent," this is not the recommended architectural pattern. Cache Sharing is a stability feature for elastic scaling, not a general-purpose load balancing mechanism for hot tablets. References • data_source/data_cache.md • sql-reference/System_variable.md • deployment/shared_data/feature-support-shared-data.md References • using_starrocks/caching/query_cache.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md

@Rocky how is it decided which CN a query should go to

In the StarRocks shared-data (cloud-native) architecture, the FE (Frontend) uses a Consistent Hashing algorithm to decide which CN (Compute Node) should process a specific query for a tablet. The decision process follows these steps: 1. The Hashing Mechanism To maximize local cache performance, StarRocks needs "stickiness"—ensuring that queries for the same data go to the same node where that data is likely already cached. * Input: The FE takes the Tablet ID as the input for the hash function. * Mapping: This hash value is mapped onto a virtual hash ring containing all active CN nodes. * Stickiness: Under normal conditions, this means "Tablet A" is always routed to "CN-1." This ensures "CN-1" builds and maintains a local cache for that specific tablet. 2. Handling Multiple Candidates (Query Replicas) By default, each tablet is mapped to only one "preferred" CN. However, if that CN becomes a hotspot, you can use the FE configuration

lake_service_query_replica_count

(available in newer versions): * Expansion: If this value is set to $N$ (e.g., 3), the FE will identify the top $N$ nodes on the hash ring for that tablet. * Selection: For any given query, the FE scheduler then selects one node from these $N$ candidates. It typically chooses the one with the lowest current load (based on heartbeat metrics like CPU and pending tasks). 3. Dynamic Rebalancing When the cluster scales (adding or removing CNs), the hash ring changes: * Re-routing: Some tablets will naturally hash to different nodes. * Cache Sharing: During this transition, the new "owner" CN can fetch the data from the old "owner" CN over the network (if

enable_datacache_sharing

is on) to avoid a complete cold-start from remote storage (S3/HDFS). * Automatic Balancing: The parameter

lake_enable_balance_tablets_between_workers

(v3.3.4+) ensures that the total number of tablets assigned across the cluster is balanced, preventing a single node from being the "owner" of a disproportionately large number of tablets. Summary of Routing Logic Scenario | Selection Logic Normal (Default) | Hash(TabletID) -> Primary CN (Highest cache hit rate) Hot Tablet | Hash(TabletID) -> $N$ candidate CNs -> FE picks least-loaded Node Failure | FE detects node is down -> Re-hashes to the next available CN on the ring Tip: If you see one CN consistently overloaded while others are idle, first check for data skew (where one tablet is much larger than others). If the data is even but the traffic is high, increase

lake_service_query_replica_count

to spread the query load across more nodes. References • quick_start/shared-data.md • administration/management/FE_configuration.md • introduction/Architecture.md • using_starrocks/Colocate_join.md References • quick_start/shared-data.md • administration/management/FE_configuration.md • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • sql-reference/System_variable.md

@Rocky difference in purpose of block and page cache?

In StarRocks, Page Cache and Block Cache serve as two distinct layers of the "Data Cache" system. The primary difference lies in where the data is stored (Memory vs. Disk) and the state of the data (Decompressed vs. Raw). Comparison Table Feature | Page Cache | Block Cache Storage Medium | In-Memory (RAM) | Local Disk (SSD/NVMe) Data State | Decompressed & Structured | Raw/Compressed (as-is from storage) Primary Goal | Reduce CPU cost & Latency (avoid decompression) | Reduce Remote I/O (avoid S3/HDFS latency) Unit Size | Variable (matches storage engine pages) | Fixed-size blocks (default 1 MB) Typical Usage | Native Tables & External Metadata | Shared-data tables & External Catalogs ────────── 1. Page Cache (In-Memory) Page Cache is designed for ultra-fast access to data that has already been retrieved and processed. * Purpose: It stores the decompressed data pages and index pages of StarRocks' internal storage engine. By keeping data in RAM, it eliminates the overhead of reading from disk and the CPU cost of decompressing data for every query. * Best for: Repetitive queries on the same "hot" data where millisecond-level latency is required. * Configuration: Controlled by

disable_storage_page_cache

and

storage_page_cache_limit

. 2. Block Cache (Disk-Based) Block Cache is a feature primarily used in Shared-data (Cloud-native) clusters and for External Catalogs (like Iceberg or Hive). * Purpose: It acts as a local mirror for data stored in remote object storage (like AWS S3 or HDFS). When a query needs data, StarRocks checks if the required "blocks" are on the local disk. If not, it fetches them from S3 and saves them locally. * Mechanism: It splits remote files into fixed-size blocks (e.g., 1MB). Even if a query only needs a few rows, it will cache the surrounding block to take advantage of spatial locality. * Value: It effectively turns your cloud-native (shared-data) cluster into a shared-nothing cluster for "hot" data, providing the performance of local NVMe storage while maintaining the cost benefits of S3. * Configuration: Controlled by

block_cache_enable

and

datacache_disk_size

. How they work together When a query runs in a cloud-native cluster: 1. Block Cache Check: The system first looks for the raw data on the local disk (Block Cache) to avoid a slow network trip to S3. 2. Page Cache Check: Simultaneously (or once the block is read and decompressed), the system checks if the structured data pages are already in Memory (Page Cache). 3. Result: If both hit, the query is incredibly fast. If only Block Cache hits, you save on network I/O but still pay the CPU cost of decompression. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/query_cache.md References • data_source/data_cache.md • using_starrocks/caching/block_cache.md