@Rocky can you tell me how enable_iceberg_metadata_cache , table_cache for catalog property config work?

In StarRocks, both

enable_iceberg_metadata_cache

and the concept of "Table Cache" are used to optimize performance when querying external data lakes. While

enable_iceberg_metadata_cache

is a specific catalog property, "Table Cache" is a component within the Iceberg metadata caching system. 1.

enable_iceberg_metadata_cache

This property is the master switch for caching metadata in an Iceberg Catalog. * What it does: When set to

true

(default), StarRocks caches Iceberg-specific metadata in memory (and optionally on disk) to avoid repeated, slow calls to remote storage (S3/HDFS). * What it caches: * Table Cache (Metadata pointers/Snapshot IDs). * Partition Name Cache. * Manifest Caches (Data and Delete file paths). * Usage: It is configured in the

PROPERTIES

when creating an Iceberg catalog:

sql
    CREATE EXTERNAL CATALOG iceberg_catalog
    PROPERTIES (
        "type" = "iceberg",
        "iceberg.catalog.type" = "hive",
        "enable_iceberg_metadata_cache" = "true"
    );

2. Table Cache (Metadata Pointer Cache) The Table Cache is a sub-component of the metadata cache. * Function: It specifically caches the metadata pointer (the JSON file containing the current Snapshot ID and the location of manifest lists). * Significance: This is the first thing StarRocks checks. If the Snapshot ID in the Table Cache is current, StarRocks can skip checking the remote metastore (like Hive Metastore or Glue) for data changes. * Refresh Interval: You can control how long the Table Cache is considered "fresh" before StarRocks re-checks the remote metastore using: *

iceberg_table_cache_refresh_interval_sec

(Default:

60

seconds). _ _Supported from v3.5.7 onwards.* 3. Equivalent for Hive (Metastore Cache) If you are using a Hive Catalog, the equivalent property is

enable_metastore_cache

. *

enable_metastore_cache

: Controls whether to cache Hive table and partition metadata. *

metastore_cache_refresh_interval_sec

: Controls the refresh interval for Hive metadata (Default:

60

seconds). Summary Table Feature | Property Name | Description Iceberg Cache Switch |

enable_iceberg_metadata_cache

| Enables caching for all Iceberg metadata (including Table Cache). Table Cache Refresh |

iceberg_table_cache_refresh_interval_sec

| How often to refresh the Snapshot ID/pointer for Iceberg tables. Hive Cache Switch |

enable_metastore_cache

| Enables caching for Hive table/partition metadata. Hive Cache Refresh |

metastore_cache_refresh_interval_sec

| How often to refresh Hive metadata from HMS/Glue. Note: For Iceberg, if you want to ensure the metadata stays up-to-date automatically, StarRocks also supports a periodic refresh strategy controlled by the FE configuration

enable_background_refresh_connector_metadata

(Default:

true

). References • sql-reference/System_variable.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • using_starrocks/caching/query_cache.md

@Rocky I want to test the performance impact of StarRocks' Iceberg metadata caching by toggling two properties: enable_iceberg_metadata_cache and enable_iceberg_table_cache. My plan is to run the same query in two configurations — (A) caches enabled and warm, (B) both caches disabled — and compare the cost. When I use EXPLAIN ANALYZE and ANALYZE PROFILE on these runs, which specific fields/metrics should I focus on to quantify the impact of the Iceberg metadata cache? Specifically: 1. At the FE planning side — what timings or counters in the profile reflect Iceberg catalog/table loading, snapshot fetching, manifest listing, and partition pruning? (e.g., Planner, Analyzer, GetPartitionsTime, GetRemoteFilesTime, scan range generation time, etc.) 2. At the scan/BE side — which IcebergScanNode / HdfsScanNode operator counters indicate cold vs. warm metadata path? (e.g., manifest file open count, data file cache hits, IO wait, ScanTime, etc.) 3. Are there any FE/BE-level metrics or log lines (outside the per-query profile) that I should also capture to corroborate — e.g., Caffeine cache stats, refreshTable log lines, or FE→FE refresh RPC counts? 4. What's the recommended way to ensure a true "cold cache" state between runs without restarting the FE — is there a REFRESH EXTERNAL TABLE / invalidatePartitionCache style command, or do I need to disable the property and reconnect? 5. What other variables should I control to make A/B comparable (e.g., page cache on the object store side, BE data file cache, query result cache, statistics cache)? The goal is to produce a clear before/after comparison showing how much of total query latency is attributable to the Iceberg metadata cache layer specifically.

To quantify the performance impact of Iceberg metadata caching in StarRocks, focus on the following metrics and control variables. Since v3.3.3, StarRocks has significantly enhanced its metadata profiling and distributed planning capabilities for Iceberg. 1. FE Planning Metrics (The "Heart" of Metadata Impact) To see these, ensure you set

SET enable_metadata_profile = true;

before running your query. In the

EXPLAIN ANALYZE

or

ANALYZE PROFILE

output, look under the Query Planner or Metadata sections: *

GetTableMetaTime

/

GetTableMeta

: The time spent fetching basic table schema and the current Snapshot ID from the Iceberg catalog (Glue/HMS). A warm Table Cache reduces this significantly. *

GetPartitionsTime

: Time spent on partition pruning. *

GetRemoteFilesTime

: This is where Manifest Caching shows its value. It covers the time spent loading, decompressing, and parsing manifest files to resolve the list of data files. * Cold Cache: This will be high as SR must download and parse Avro manifest files from S3/HDFS. * Warm Cache: This should drop to near-zero as SR reads deserialized metadata objects from FE memory. *

PlanTime

: The overall time for the FE to generate the physical plan. Metadata caching's primary goal is to minimize this. 2. BE Scan Node Metrics If you use Distributed Planning (

SET plan_mode = 'distributed';

), metadata parsing is offloaded to BEs. Look at the

ConnectorScanNode

or

IcebergScanNode

metrics: *

ScanTime

: While this includes data IO, a "metadata-cold" run with many manifests will show high overhead in the initial setup. *

IOTaskWaitTime

: If metadata is cold and being fetched/parsed on the fly, tasks may wait longer. *

ManifestFileReadTime

/

ManifestFileReadCount

: Specifically shows the cost of manifest interaction (if the profile version exposes these detailed Iceberg-specific counters). *

DataCacheReadBytes

/

DataCacheWriteBytes

: Note that these usually refer to the Data Block Cache (the actual

.parquet

data), not the metadata. 3. Verification & Metrics (Non-Profile) To see the "Global" state of the cache: * FE Logs (

fe.log

): Look for

refreshTable

or

getTable

entries. When

enable_iceberg_metadata_cache

is

true

, you will see background refresh tasks triggering. * FE Prometheus Metrics: Look for

iceberg_metadata_memory_cache_capacity

and usage. * FE Disk Cache: Check the directory

STARROCKS_HOME/caches/iceberg

on the FE. If disk caching is enabled (

enable_iceberg_metadata_disk_cache = true

), files here represent the persisted metadata. 4. Achieving a "True Cold Cache" State Simply toggling the property to

false

and then

true

does not necessarily purge the memory. To ensure a 100% cold state for A/B testing: 1. Manual Refresh: Execute

REFRESH EXTERNAL TABLE <catalog>.<db>.<table>;

. This forces StarRocks to re-synchronize with the metastore and update the cached snapshot. 2. Clear Disk Cache: If disk caching is enabled, you must delete the contents of

fe/caches/iceberg/

on the FE node. 3. Purge Memory: Since the FE metadata cache uses an internal Caffeine cache, there is no direct

FLUSH CACHE

SQL command for all metadata. The most reliable "cold" state is: * Stop FE. * Delete the

caches/iceberg

directory. * Start FE. * (Alternatively, use a brand new table/catalog name for the "Cold" test to bypass existing cache keys). 5. Control Variables for A/B Testing To isolate Metadata Cache impact from Data/Result Cache impact, disable the following during your test: * Query Result Cache:

SET enable_query_cache = false;

(Prevents entire query results from being cached). * Data Block Cache: Set

datacache_enable = false

in BE configs or bypass it to ensure data IO remains constant. * Page Cache:

SET enable_storage_page_cache = false;

.