@Rocky We operate many StarRocks clusters sharing one Iceberg external catalog backed by Hive Metastore. We know two ways to keep the Iceberg metadata cache fresh without ETL-driven REFRESH EXTERNAL TABLE: setting enable_iceberg_table_cache=false, and shortening background_refresh_metadata_interval_millis. Are there any other built-in mechanisms in StarRocks for this? Please list them with version numbers and config keys.

Besides the two methods you mentioned (

enable_iceberg_table_cache

and

background_refresh_metadata_interval_millis

), StarRocks provides several other built-in mechanisms to maintain Iceberg metadata freshness, especially for clusters sharing a Hive Metastore (HMS). Here are the additional mechanisms, categorized by their implementation level: 1. HMS Event Listener (Automatic Event-Driven Refresh) Instead of polling all tables, StarRocks can listen to the Hive Metastore event log. When a table is updated (e.g., a new Iceberg snapshot is committed and the metadata location is updated in HMS), StarRocks perceives the

ALTER_TABLE

event and automatically refreshes its cache. * FE Configuration Keys: * `enable_metastore_event_listener`: Set to

true

to enable. (Default:

false

) * `hms_events_polling_interval_ms`: The interval to poll HMS for new events. (Default:

500

ms) * `hms_events_batch_size`: The number of events to process in one batch. (Default:

500

) * Version: Supported since v2.4.0 for Hive catalogs; applies to Iceberg catalogs backed by HMS as they trigger standard HMS table update events. 2. Fine-Grained Table Cache Refresh (Catalog Property) Introduced in recent versions, this property allows you to control how long the FE considers the cached Iceberg table metadata (like the current snapshot ID) to be "fresh" before checking the metastore again during a query plan. * Catalog Property:

iceberg_table_cache_refresh_interval_sec

* Default:

60

(seconds) * Description: If a query occurs within this interval since the last refresh, StarRocks uses the cache. If exceeded, it checks the metastore for a newer snapshot ID. * Version: Supported from v3.5.7 onwards. 3. Metadata Refresh Task Expiration While the background refresh (polling) keeps tables fresh, you can control how long StarRocks continues to refresh a table after its last access. This prevents the cluster from wasting resources on idle tables while keeping active ones fresh. * FE Configuration Key:

background_refresh_metadata_time_secs_since_last_access_secs

* Default:

86400

(24 hours) * Description: StarRocks will stop background refreshing a table if it hasn't been queried for this amount of time. * Version: Introduced in v2.5.5. 4. Planning Mode (Session Variable) From v3.3.3, StarRocks introduced a system variable that changes how the optimizer handles Iceberg metadata during the planning phase. * System Variable:

plan_mode

* Values:

default

(uses cache) or

collect

(forces more aggressive metadata retrieval). * Description: Setting this to

collect

can sometimes be used to bypass certain caching behaviors during heavy metadata updates, though it is primarily used for parsing statistics. * Version: Introduced in v3.3.3. 5. Memory Cache Expiration Settings If you want the cache to "die" naturally so that the next query is forced to fetch from the source, you can shorten the global expiration TTL for the metadata cache entries. * FE Configuration Keys: * `iceberg_metadata_memory_cache_expiration_seconds`: TTL for metadata in FE memory. (Default:

86500

) * `iceberg_metadata_disk_cache_expiration_seconds`: TTL for metadata in FE disk cache. (Default:

604800

) * Version: Generally available since v3.2.x (when the two-level metadata cache was formalized). Summary Table Mechanism | Config Key | Version Event-Driven |

enable_metastore_event_listener

(FE) | v2.4.0+ Table Refresh Interval |

iceberg_table_cache_refresh_interval_sec

(Catalog Property) | v3.5.7+ Polling Task Expiry |

background_refresh_metadata_time_secs_since_last_access_secs

(FE) | v2.5.5+ Parsing Control |

plan_mode

(Session Variable) | v3.3.3+ Cache TTL |

iceberg_metadata_memory_cache_expiration_seconds

(FE) | v3.2.4+

@Rocky I want to verify the first one. In 4.0.x the actual config key is enable_hms_events_incremental_sync (not enable_metastore_event_listener), and grepping the source shows that MetastoreEventsProcessor.registerCacheUpdateProcessor is only called from HiveConnector.java, not from IcebergConnector.java. So even if the listener processes HMS events, the Iceberg cache does not receive any invalidation callback. Can you confirm: does enabling enable_hms_events_incremental_sync actually invalidate the Iceberg metadata cache when an Iceberg table's metadata_location is updated in HMS? If yes, please point me to the exact code path in the Iceberg connector that subscribes to these events. If no, is there a planned PR to add this for the Iceberg connector?