http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> difference between block cache, dat...
# questions-and-troubleshooting
a
@Rocky difference between block cache, data cache and starlet cache and what are the best options to setup?
r
The terms Block Cache, Data Cache, and Starlet Cache in StarRocks all refer to the same fundamental mechanism: caching remote data (from object storage or HDFS) onto local disks to accelerate queries. However, they are used in different contexts or represent different stages of StarRocks' evolution. 1. Differences * Block Cache: This is the functional name of the technology introduced in v2.5. It works by splitting remote files (like Parquet or ORC) into small, fixed-size blocks (e.g., 1 MB) and caching only the requested blocks on local BE/CN disks. * Data Cache: This is the official, unified name for this feature from v3.1.7 and v3.2.3 onwards. It replaced the older "File Cache" (which cached entire files) and "Block Cache" terminology in the documentation. It is the user-facing name for the unified caching system used for both External Catalogs (Data Lake) and Cloud-native (shared-data) tables. * Starlet Cache: This refers to the internal implementation name or the underlying directory name. "Starlet" is the name of StarRocks' internal storage engine for shared-data architecture. You will often see 
starlet_cache
in directory paths or configuration items like 
starlet_star_cache_disk_size_percent
. In short: They are essentially the same thing. Data Cache is what you call it, Block Cache is how it works, and Starlet Cache is the internal component that manages it. ────────── 2. Best Options for Setup To get the best performance from the Data Cache, follow these recommendations: Hardware & Storage * Use NVMe/SSD: Data Cache is disk-based. Using high-performance local NVMe or SSDs is the single most important factor for reducing query latency. * Multiple Disks: If you have multiple local disks, configure them in 
storage_root_path
to balance I/O pressure. Key Configuration (BE/CN config) * Enable it: Ensure 
datacache.enable = true
is set (usually default in shared-data, but check for external tables). * Disk Size: Set 
starlet_star_cache_disk_size_percent
(default is 80%). This limits how much of your disk the cache can consume. * Eviction Policy: • Use SLRU (Segmented LRU) instead of standard LRU. SLRU is better at preventing "one-time scans" (temporary data) from evicting your frequently used hot data. • Config: 
block_cache_evict_policy = SLRU
. Table Properties * Partition Duration: For very large tables, you can use 
datacache.partition_duration
to only cache recent data (e.g., the last 30 days), preventing old historical data from polluting the cache. * Hot/Cold Data: In shared-data clusters, you can explicitly set whether a table uses the cache via the 
datacache.enable
property during 
CREATE TABLE
. Summary Recommendation For a standard production setup: 1. Deploy on nodes with local NVMe disks. 2. Set the cache disk limit to 80-90% of available disk space. 3. Use the SLRU eviction policy. 4. If using v3.4+, the cache is *unified*—it will automatically manage blocks for both your data lake queries and your internal cloud-native internal tables in-house tables. References • data_source/data_cache.md • using_starrocks/caching/block_cache.md
a
@Rocky If I run mat views in shared data cluster using Iceberg tables as source, what caches will be in use for mat views?
r
When you run Materialized Views (MVs) in a shared-data cluster with Iceberg tables as the source, you aren't just using one cache; you are using a four-layer caching architecture. Here is how each cache type works in that specific scenario: 1. Iceberg Metadata Cache (FE/BE) Before StarRocks even reads a row of data, it needs to know which files to read. * What it caches: Snapshot IDs, manifest lists, manifest files, and partition info. * Role in MVs: Crucial for partition-level change detection. StarRocks uses this metadata to determine if an Iceberg table has changed so it can decide whether to perform an incremental refresh or a full refresh of the MV. * Where it lives: FE (memory/disk) and BE (memory). 2. Data Cache (The "Block/Starlet" Cache on BE/CN) This is the most critical cache for performance in shared-data clusters. * During MV Refresh: When the MV is refreshing, it queries the Iceberg base table. The Data Cache stores the remote Parquet/ORC blocks from Iceberg onto the local SSDs of your CN/BE nodes. Subsequent refreshes will be much faster because they read from local disk instead of remote S3/HDFS. * During MV Query: Since an MV in a shared-data cluster is itself a "cloud-native" table, its own data (the pre-aggregated results) is also stored in remote storage (e.g., S3). When you query the MV, the Data Cache will cache the MV's own data blocks locally. 3. Query Cache (BE Memory) * What it caches: Intermediate computation/aggregation results at the tablet level. * Role in MVs: If your MV definition includes complex aggregations (e.g., 
SUM
, 
COUNT
), and you have enabled the Query Cache (
SET enable_query_cache = true;
), StarRocks can reuse intermediate results during the MV refresh process or when querying the MV itself. * Version Note: Query Cache support for shared-data clusters was fully aligned in v3.4. 4. MV Rewrite Cache (FE Memory) * What it caches: The logical plan mapping between the base Iceberg table and the Materialized View. * Role in MVs: When you query the base Iceberg table, the FE uses this cache to quickly decide if it can "transparently rewrite" your query to hit the MV instead. This ensures that the overhead of checking for MV applicability is near zero. Summary of what to enable: To ensure this setup runs optimally: 1. 
datacache.enable = true
: (Default) Ensures Iceberg data and MV data are cached on local SSDs. 2. 
enable_iceberg_metadata_cache = true
: (Default) Speeds up the "planning" phase of the MV refresh. 3. 
enable_query_cache = true
: Recommended if your MV or the queries hitting it involve heavy aggregations. 4. 
mv_rewrite_staleness_second
: Set this on the MV if you want to allow queries to hit the MV cache even if the base Iceberg table has changed slightly (tolerating some lag for "async" consistency). References • data_source/catalog/iceberg/iceberg_catalog.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/query_cache.md References • using_starrocks/caching/query_cache.md • release_notes/release-3.4.md • faq/Sql_faq.md • using_starrocks/async_mv/use_cases/data_lake_query_acceleration_with_materialized_views.md
a
@Rocky how to disable cache for Iceberg tables data for specific catalog?
r
To disable caching for Iceberg table data at the catalog level, you must distinguish between Data Cache (caching actual data blocks to disk) and Metadata Cache (caching snapshot and file info). 1. Disabling Data Cache (Block Cache) The Data Cache (Block Cache) is primarily controlled by system variables. Currently, StarRocks does not provide a dedicated 
CREATE CATALOG
property named 
datacache.enable
for external Iceberg catalogs (unlike internal cloud-native tables). To disable it for a specific catalog's queries, you have two options: * Option A: Session Variable (Per Query/Session) If you want to ensure a specific workload doesn't use the cache, set the session variable before running your queries:
Copy code
sql
    -- Disable reading from and writing to the Data Cache
    SET enable_scan_datacache = false;
    SELECT * FROM iceberg_catalog.db.table;
* Option B: Catalog-Level Session Default (v3.2.4+) In recent versions, StarRocks allows you to set session variables as catalog properties. This acts as a default for any query hitting that catalog:
Copy code
sql
    ALTER CATALOG iceberg_catalog SET ("enable_scan_datacache" = "false");
_Note: Verify your specific StarRocks version supports this override for 
enable_scan_datacache
._ ────────── 2. Disabling Metadata Cache If your goal is to ensure StarRocks always sees the latest changes from the Iceberg source without waiting for the cache to expire, you should disable or tune the Metadata Cache: * Disable Metadata Cache via Catalog Property:
Copy code
sql
    -- During creation
    CREATE EXTERNAL CATALOG iceberg_catalog
    PROPERTIES (
        "type" = "iceberg",
        "enable_iceberg_metadata_cache" = "false",
        ...
    );

    -- Or for an existing catalog
    ALTER CATALOG iceberg_catalog SET ("enable_iceberg_metadata_cache" = "false");
* Alternative: Set TTL to 0 If you want to keep the cache mechanism but force it to refresh every single time:
Copy code
sql
    ALTER CATALOG iceberg_catalog SET ("iceberg_meta_cache_ttl_sec" = "0");
Summary Recommendation * To stop local SSD wear/usage for Iceberg data: Use 
ALTER CATALOG ... SET ("enable_scan_datacache" = "false")
. * To ensure data freshness (seeing new Iceberg snapshots immediately): Use 
ALTER CATALOG ... SET ("enable_iceberg_metadata_cache" = "false")
. References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md • data_source/catalog/maxcompute_catalog.md • data_source/catalog/hive_catalog.md • data_source/data_cache.md
a
@Rocky can I disable page cache for iceberg catalog?
r
Yes, you can disable the Page Cache for an Iceberg catalog. In StarRocks, the Page Cache is an in-memory cache on the BE that stores decompressed data pages. While it is a global BE resource, you can control its usage at the session or catalog level using system variables. 1. Identify the Cache Type It is important to distinguish between the two: * Page Cache (Memory): Caches decompressed data in RAM. * Data Cache (Disk): Caches raw remote blocks on SSD/Disk (also known as Block Cache). 2. How to Disable Page Cache for a Specific Catalog Since StarRocks (v3.2.4+) allows you to set session variables as catalog properties, you can effectively disable the Page Cache for all queries hitting a specific Iceberg catalog:
Copy code
sql
-- Disable the use of Page Cache for this specific Iceberg catalog
ALTER CATALOG iceberg_catalog SET ("use_page_cache" = "false");
Or, if you are using a newer version (v3.3.9+ / v3.4.0+), you can use the more explicit skip flag:
Copy code
sql
ALTER CATALOG iceberg_catalog SET ("skip_page_cache" = "true");
3. Disabling via Session (For specific queries) If you don't want to change the catalog property permanently, you can disable it for your current session before running a query against the Iceberg table:
Copy code
sql
-- Option A: Do not use the page cache for this session
SET use_page_cache = false;

-- Option B: Instruct the scanner to bypass the page cache (v3.3.9+)
SET skip_page_cache = true;

SELECT * FROM iceberg_catalog.db.table LIMIT 10;
Why would you do this? * Prevent Cache Pollution: If you are performing a massive one-time scan (e.g., an ETL job or a full table refresh) of an Iceberg table, you may want to disable the Page Cache so that it doesn't evict "hot" data from your internal StarRocks tables. * Memory Management: If you are memory-constrained on your BE nodes and want to reserve RAM for query execution rather than caching external data. Summary of Variables Variable | Introduced | Effect 
use_page_cache
| v3.2.0 | Controls whether the query uses the BE memory page cache. 
skip_page_cache
| v3.3.9 | Explicitly marks scan ranges to bypass the memory page cache layer. 
enable_scan_datacache
| v2.5 | Controls the Disk-based Data Cache (not the memory Page Cache). References • sql-reference/System_variable.md • data_source/data_cache.md • administration/management/BE_configuration.md • data_source/catalog/iceberg/iceberg_catalog.md References • sql-reference/System_variable.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • data_source/data_cache.md
a
@Rocky I run version 4.0.0. I did
Copy code
SET use_page_cache = false;
SET enable_scan_datacache = false;
SET skip_page_cache = true;
SET skip_local_disk_cache = true;
SET populate_datacache_mode = "never";
but see not difference, latency is the same as with enabled cache and it is low
r
In StarRocks v4.0.0 (and versions 3.4+), the caching architecture is deeply layered. If you have disabled the Data Cache (disk) and Page Cache (memory) but are still seeing low latency, it is likely because the query is being satisfied by a higher-level cache or a transparent rewrite. Here are the four reasons why your latency remains low despite your settings: 1. The Query Cache (Result Cache) is still active The most common reason for "instant" identical performance is the Query Cache. It caches the intermediate or final results of aggregations in BE memory. Even if you disable the data scanners, this cache intercepts the query before it ever reaches the storage layer. * Check: Run 
SET enable_query_cache = false;
and try again. * Verification: Look for 
QueryCacheHit
or similar metrics in the query profile. 2. Iceberg Metadata Cache is still enabled For Iceberg tables, the "cold" latency is usually dominated by fetching the snapshot and manifest files from S3/HDFS. StarRocks caches this metadata separately from the data. Disabling the Data Cache does not disable the Metadata Cache. * Effect: StarRocks already knows exactly which files to read, skipping the slow "Planning" phase. * Check: 
SET enable_iceberg_metadata_cache = false;
(Note: In some versions, this is a Catalog-level property: 
ALTER CATALOG <name> SET ("enable_iceberg_metadata_cache" = "false");
). 3. Materialized View Rewrite Since you mentioned using MVs, StarRocks might be automatically rewriting your query to hit a Materialized View instead of the base Iceberg table. If that MV is small or its data is already optimized, the query will be fast. * Check: 
SET enable_materialized_view_rewrite = false;
to force the query to hit the raw Iceberg table. * Observation: Run 
EXPLAIN <your_query>
to see if it shows an 
OlapScanNode
(hitting the MV) or an 
IcebergScanNode
(hitting the base table). 4. OS-level Page Cache If your BE/CN nodes have significant free memory, the Linux kernel will cache recently accessed files (the local blocks used by the shared-data cache) in the OS Page Cache. Even if StarRocks tells the application layer to bypass its internal cache, the OS may still serve the file from RAM. * Note: This is harder to disable without root access (
echo 3 > /proc/sys/vm/drop_caches
), but it only applies if the data was recently read from the local disk. How to verify which cache is hitting To see the "true" cold performance, you should disable everything in one block and check the profile:
Copy code
sql
-- Disable all result-level and rewrite caches
SET enable_query_cache = false;
SET enable_materialized_view_rewrite = false;

-- Disable data-level caches (your current settings)
SET use_page_cache = false;
SET skip_page_cache = true;
SET enable_scan_datacache = false;
SET populate_datacache_mode = "never";

-- Disable metadata cache (critical for Iceberg)
SET enable_iceberg_metadata_cache = false;

-- Execute your query
SELECT ... ;
After running the query, run 
GET_QUERY_PROFILE(query_id);
and look at the 
IcebergScan
 or 
OlapScan
 section. Check the following metrics: * 
DataCacheReadBytes
: Should be 0 if data cache is disabled. * 
BytesReadRemote
: Should equal the total bytes read if you are truly hitting the "cold" storage (S3/HDFS). * 
IcebergMetadataRefreshTime
: High value here indicates a metadata cache miss.
Open in Slack
PreviousNext
Powered by