@Rocky What should i modify for tunning off of iceberg metadata cache in catalog property for 4.1.1 version? previously it was enable_iceberg_metadata_cache = false on 4.0.*

In StarRocks version 4.1.1 (and starting from the v3.3.x/4.0 series), the primary catalog property to disable the Iceberg metadata cache remains

enable_iceberg_metadata_cache

. However, because StarRocks introduced a new periodic metadata refresh strategy and a two-level (memory + disk) caching mechanism in recent versions, there are a few additional parameters you may need to modify to fully "turn off" or bypass the cache depending on your goal (e.g., ensuring data freshness vs. reducing memory usage). 1. Catalog Property (Primary) To disable the metadata cache at the catalog level, set the following property during

CREATE EXTERNAL CATALOG

or via `ALTER CATALOG`:

sql
ALTER CATALOG <your_iceberg_catalog_name> SET (
    "enable_iceberg_metadata_cache" = "false"
);

* Note: In versions 3.3.0 and later, the default value for this property was changed back to

true

(it was briefly

false

by default for certain metastores in v3.2.x). 2. Disabling Background Refresh Starting from v3.3.3/4.0, StarRocks performs periodic background refreshes of Iceberg metadata to keep the cache warm. If you want to disable this background activity, you should modify the FE configuration: * FE Configuration:

enable_background_refresh_connector_metadata

(Default:

true

) * Set this to

false

via

ADMIN SET FRONTEND CONFIG ("enable_background_refresh_connector_metadata" = "false");

to stop StarRocks from polling the metastore (HMS/Glue) periodically. 3. Achieving Real-Time Metadata (TTL Strategy) If your goal is to ensure every query sees the latest snapshot from the metastore (equivalent to "no cache"), you can set the TTL to zero: * Property:

"iceberg_meta_cache_ttl_sec" = "0"

* Setting this to

0

forces StarRocks to fetch a new snapshot for each query. 4. Advanced: Disabling Disk Cache If you specifically want to disable the disk portion of the metadata cache (which might be what's causing stale results in some 4.x edge cases), use: * Property:

"enable_iceberg_metadata_disk_cache" = "false"

(Default is usually

false

, but verify if you've enabled it). Summary of properties for 4.1.x: Property | Recommended for "Off" | Description

enable_iceberg_metadata_cache

|

false

| Disables caching of table pointers and manifests in memory.

iceberg_meta_cache_ttl_sec

|

0

| Ensures fresh snapshots are pulled for every query.

enable_cache_list_names

|

false

| Disables the partition name list cache. Important for 4.1.x users: If you are disabling the cache due to stale data issues, you may also want to check the session variable

plan_mode

. In some scenarios, setting

SET plan_mode = 'local';

or

'distributed';

can impact how metadata is parsed and whether the cache is bypassed. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/deltalake_catalog.md

@Rocky can you tell me metric list of starrocks fe for usage some resources related with external catalog. ( for example, amount of meta full scan, query plan time and so on.)

For monitoring external catalog performance and resource usage on the StarRocks FE (v4.1.1), metrics are available via the Prometheus endpoint (usually

http://<FE_IP>:8030/metrics

) and the Query Profile. 1. Prometheus FE Metrics (Cluster-wide) These metrics help you track the overall health and "amount" of metadata operations across all queries. * Query Planning Latency: *

starrocks_fe_query_latency_ms

: A histogram of total query time. While it includes execution, significant spikes here often correlate with slow metadata fetching or cache misses during the planning phase. * External Metadata Cache (Iceberg/Hive): *

starrocks_fe_external_metadata_cache_hit_total

: Increments when metadata (like table/partition info) is retrieved from the FE's internal cache. *

starrocks_fe_external_metadata_cache_miss_total

: Increments when StarRocks must perform a "full scan" or fetch from the remote metastore (HMS, Glue). High values indicate the cache is disabled or frequently expiring. * Background Refresh Activity: *

starrocks_fe_background_refresh_metadata_count

: Tracks how many periodic metadata refresh tasks have been triggered (related to the

enable_background_refresh_connector_metadata

setting). * Iceberg Specific Write/Delete Operations: *

iceberg_write_duration_ms_total

: Time spent on Iceberg write operations. *

iceberg_delete_total

: Count of delete operations, with labels for

delete_type

(metadata vs. position). * Metastore Connection Health: *

starrocks_fe_hms_event_process_latency

: If using HMS events for metadata sync, this tracks the lag in perceiving changes. 2. Query Profile Metrics (Per-Query Detail) To see the exact "query plan time" and "meta scan" details for a specific query, check the Query Profile (

GET TYPE=PROFILE

). Look for these fields in the FE (FrontEnd) section of the profile: *

QueryPlanTime

: The total time FE spent generating the plan. *

MetadataFetchTime

/

TableFetchTime

: Time spent fetching table/partition definitions from the external source. *

IcebergMetadataScanTime

: Specifically for Iceberg, the time spent reading manifest files during planning. *

PartitionPruningTime

: Time spent filtering partitions, which is a common bottleneck for large Hive/Iceberg tables. 3. BE-side Data Cache Metrics Since metadata caching on FE is often paired with data caching on BE, you should also monitor: *

block_cache_hit_bytes

/

block_cache_miss_bytes

: Tells you if the data (Parquet/ORC) is being read from local disk or remote storage (S3/HDFS). *

datacache_meta_used_bytes

: Memory used by the BE to track cached data blocks. Summary Table for Tuning Usage Category | Key Metric (Prometheus) | Profile Field Meta Full Scans |

starrocks_fe_external_metadata_cache_miss_total

|

MetadataFetchTime

Query Plan Time |

starrocks_fe_query_latency_ms

(filtered) |

QueryPlanTime

Metastore Load |

starrocks_fe_background_refresh_metadata_count

|

TableFetchTime

Resource Usage |

iceberg_metadata_memory_cache_capacity

(Gauge) |

PrepareChunkSourceTime

Tip: If you see high

QueryPlanTime

and frequent

cache_miss

metrics, ensure

enable_iceberg_metadata_cache

is

true

and check if your

iceberg_meta_cache_ttl_sec

is set high enough to be useful. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md