@Rocky if you had data landing in a raw table that was partitioned by raw_import_timestamp only to be used once to move to the next level of cleaned tables, should you still enable cache? shared data arch.

In a StarRocks shared-data architecture, whether you should enable the data cache for a transient raw table depends on the priority of your ETL performance versus the stability of your hot data cache. Short Answer Yes, you should usually keep it enabled if you want the subsequent ETL process (the "move to next level" query) to be fast. However, if your raw data volume is so large that it evicts your primary "clean" tables from the cache, you should disable it or use a short partition duration. ────────── Detailed Considerations 1. Performance of the "One-Time" Read In shared-data mode, when

datacache.enable

is

true

, data is written to both the local disk (cache) and remote object storage (S3/GCS) during the initial load. * With Cache Enabled: When your ETL process runs (

INSERT INTO clean_table SELECT FROM raw_table

), StarRocks reads the raw data directly from the local NVMe/SSD cache on the Compute Nodes. This is significantly faster than fetching from remote storage. * With Cache Disabled: The ETL process must fetch every block of the raw table from S3/GCS, which increases network overhead and query latency. 2. The "Cache Pollution" Risk The main downside is that your raw data will occupy cache space that could be used for your more frequently queried cleaned tables. * LRU Eviction: StarRocks manages the cache using an LRU (Least Recently Used) policy. Once the cache reaches its limit (default 80% of disk), it evicts the least recently used data. * Loading Behavior: Data loading always populates the cache if enabled, regardless of any duration settings. If you load a massive raw dataset, it will displace other data in the cache to make room for itself. 3. Recommended Configuration To balance performance and resource management for raw tables, use these properties:

sql
PROPERTIES (
    "datacache.enable" = "true",
    "datacache.partition_duration" = "1 DAY"
)

*

datacache.enable

: Keep this

true

to ensure the ETL query hits the local disk. *

datacache.partition_duration

: Set this to a short window (e.g.,

1 DAY

or

12 HOUR

). While this doesn't prevent the initial load from filling the cache, it ensures that if the raw data is ever queried again after it has been evicted, StarRocks will not waste resources re-caching it from S3. When to Disable Cache for Raw Tables You should set

"datacache.enable" = "false"

only if: 1. Cache Churn is a problem: Your raw data is so massive that it causes "cache thrashing," where your production tables are constantly being evicted and re-loaded, hurting user-facing query performance. 2. ETL Speed is not a priority: You don't mind the ETL/cleaning process taking longer because it is running in the background and you want to prioritize the cache entirely for "cleaned" or "gold" layer tables. Note on Dropping Data Dropping a table or partition in StarRocks *does not immediately delete the data from the local cache*; it simply marks those blocks as "evictable." They will be overwritten by new data over time according to the LRU logic. References • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky if

datacache.enable

is set to false, a large load of data won't impact the cache at all right?

Correct. If you set

"datacache.enable" = "false"

for a table, a large load into that table will not occupy space in the StarRocks Data Cache and, consequently, will not evict your existing "hot" data. Here is the breakdown of why: 1. Bypassing the Cache: When this property is

false

, StarRocks writes the data directly to the remote object storage (like S3) and skips the step of writing a copy to the local disk cache on the Compute Nodes. 2. No LRU Eviction: Since no new data blocks from the raw table are being added to the cache, the LRU (Least Recently Used) mechanism is never triggered for this specific load. Your existing cached data (e.g., from your "clean" tables) stays exactly where it is. 3. Direct-from-S3 Reads: When you eventually run your ETL query to move data from the raw table to the next level, StarRocks will read that raw data directly from remote storage. This will be slower than a local cache hit, but it ensures your cache remains "pure" for your production workloads. Recommendation for your scenario: If your raw table is very large (e.g., multiple TBs) and you only need it for a single batch cleanup once a day, setting

datacache.enable = false

is the safest way to protect your cache from "thrashing" (where the raw load wipes out all your useful cached data). Summary Table: Property Setting | Load Performance | ETL Performance | Impact on Cache

datacache.enable = true

| Fast (Parallel write) | Fast (Local disk) | High (May evict other data)

datacache.enable = false

| Standard | Slower (S3/GCS read) | Zero References • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • faq/shared_data_faq.md

@Rocky Is the CN memory cache flushed during a large memory intensive operation? If so is the old cached then re-read?

In the StarRocks shared-data architecture, whether a "memory cache" is flushed depends on which specific cache you are referring to. On a Compute Node (CN), there are three distinct layers of "cache" that interact during memory-intensive operations: 1. The Operating System (OS) Page Cache This is the most common reason for "cache flushing" during large operations. * Behavior: The OS uses free RAM to cache the data blocks of the Data Cache (which are stored on the local NVMe/SSD). When a large, memory-intensive query (like a massive JOIN or aggregation) needs RAM for its intermediate results, the Linux kernel will automatically reclaim (flush) this Page Cache to prioritize the query's memory needs. * Re-read: Yes. Once the query finishes and the data is needed again, it must be re-read from the local disk back into RAM. This is still fast (local NVMe), but slower than a pure RAM hit. 2. StarRocks Internal Memory Caches (Page Cache / Query Cache) StarRocks has internal caches that live directly in the process's memory heap, such as the

storage_page_cache

(caching decoded blocks) and the

query_cache

(caching result sets). * Behavior: These caches have fixed size limits (e.g.,

storage_page_cache_limit

). They do not "flush" entirely during a large operation. Instead, they follow an LRU (Least Recently Used) eviction policy. If the query requires memory that pushes the CN toward its

mem_limit

, StarRocks will aggressively evict the oldest entries from these caches to stay within safety bounds. * Re-read: Yes. Evicted entries must be re-computed or re-fetched from the Data Cache (disk) or remote storage. 3. Data Cache Metadata The metadata for the disk-based Data Cache is kept in CN memory to track which blocks are on disk. * Behavior: This is generally not flushed during queries because it is relatively small and essential for the CN to know what is cached on disk. If this were flushed, the CN would lose track of its local disk cache. ────────── Summary for your ETL/Raw Data Scenario If you are running a massive "move to cleaned" operation: 1. Memory Pressure: If the query is so large it hits the CN memory limit, the OS will dump the Page Cache for the raw data to make room. 2. Performance Impact: The "old" cached data is not lost from the disk, but it is "flushed" from RAM. The next query will experience a slight delay as it pulls that data from the local SSD back into RAM. 3. Stability: StarRocks' memory manager will try to cancel the query or evict LRU memory caches before the CN crashes (OOM). Recommendation: If you frequently run operations that are so large they "flush" your cache and hurt other queries, consider increasing the CN RAM or using Resource Groups to limit the memory a specific ETL query can consume. References • using_starrocks/caching/block_cache.md • administration/management/resource_management/Memory_management.md • using_starrocks/caching/query_cache.md • administration/management/resource_management/Resource_group.md References • using_starrocks/caching/block_cache.md • using_starrocks/caching/query_cache.md

@Rocky if i understand what you said before, INSERT queries do not cache, would a query that INSERTS but has a SELECT subquery cache or not?

Yes, an

INSERT INTO ... SELECT

query will trigger caching for both the data being read and the data being written, provided

datacache.enable

is set to

true

for those tables. To clarify the behavior in StarRocks shared-data architecture: 1. The

SELECT

(Read) part caches When the subquery reads from the source (raw) table: * If the data is already in the local cache, StarRocks reads it from disk (fast). * If the data is not in the cache, StarRocks fetches it from S3/remote storage and simultaneously writes it to the local cache for future use. 2. The

INSERT

(Write) part caches Contrary to the premise that "INSERT queries do not cache," in shared-data mode, all data loading (including

INSERT

,

Stream Load

, etc.) is "write-through" by default. * When StarRocks writes new data to the destination table, it writes the data to the backend object storage (S3) and to the local disk cache at the same time. * This ensures that the "fresh" data is immediately available in the cache for the next step in your pipeline. Summary of Impact If you run `INSERT INTO cleaned_table SELECT * FROM raw_table`: * Source Table: Its data is pulled into the cache (if it wasn't there). * Destination Table: Its new data is placed into the cache as it is created. Why this matters for your architecture: If you want to avoid "cache pollution" from your raw tables, you should specifically set

"datacache.enable" = "false"

on the raw table. This ensures the

SELECT

part reads from S3 without filling the cache with transient data. However, you should likely keep it

"true"

for the cleaned table so that your final "gold" data is ready and fast for end-user queries. References • deployment/shared_data/feature-support-shared-data.md • introduction/Architecture.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md References • introduction/Architecture.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache.md