Hey @Rocky, We are wondering about using StarRocks shared-data architecture. Nevertheless I haven't found too much info about refresh rate of Iceberg Metadata. How long does it take for the StarRocks engine to refresh iceberg metadata?

In StarRocks, the refresh of Iceberg metadata is primarily managed by the Frontend (FE) through an asynchronous background mechanism. This behavior is consistent across both shared-nothing and shared-data architectures. 1. Default Refresh Rates * Background Refresh Interval: By default, StarRocks polls the Iceberg metastore (Hive Metastore, Glue, etc.) every 10 minutes to detect changes. * Configuration:

background_refresh_metadata_interval_millis

(Default:

600000

ms). * Active Refresh Window: StarRocks only maintains this background refresh for catalogs that have been accessed within the last 24 hours. * Configuration:

background_refresh_metadata_time_secs_since_last_access_secs

(Default:

86400

s). 2. Metadata Freshness Logic StarRocks uses a tiered logic to balance query latency and data freshness: * 0 – 60 seconds: Metadata is considered "fresh." Queries use the cache immediately. * After 60 seconds: If a query is issued, StarRocks still uses the cached metadata for that specific query to ensure zero latency, but it simultaneously triggers an asynchronous refresh in the background so subsequent queries will see the new data. * Manual Refresh: You can bypass the background timer and force an immediate update of the metadata using the following command:

sql
    REFRESH EXTERNAL TABLE <table_name>;

3. Key FE Configurations If you need to tighten these intervals, you can modify these FE dynamic parameters: Parameter | Default | Description

enable_background_refresh_connector_metadata

|

true

| Enables/disables periodic background polling.

background_refresh_metadata_interval_millis

|

600000

| Frequency of background metadata checks.

iceberg_table_cache_refresh_interval_sec

|

60

| The "freshness" window before an async refresh is triggered by a query. Note on Shared-Data Architecture In the shared-data (compute-storage separation) architecture, these metadata operations still reside on the FE. While the Compute Nodes (CN) may cache actual data blocks on local disks to accelerate I/O, the Iceberg snapshot and manifest metadata lifecycle are controlled by the FE configurations listed above. References • data_source/catalog/hudi_catalog.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/maxcompute_catalog.md • data_source/catalog/hive_catalog.md • data_source/catalog/iceberg/procedures.md

So, it would take up to 60 seconds to get fresh iceberg data available in SR? @Rocky

The 60-second interval is an asynchronous trigger, not a hard delay. It’s important to distinguish between how StarRocks uses the cache versus how it updates it: 1. The "60-Second" Rule (v3.5.7+) By default, StarRocks uses the parameter

iceberg_table_cache_refresh_interval_sec

(default: 60s) to decide when to refresh: * 0 – 60 Seconds: If you query the table, StarRocks uses the cached metadata immediately. It assumes the data is fresh enough. * After 60 Seconds: When a query arrives, StarRocks still uses the cached (potentially old) metadata to answer that specific query instantly (zero latency). However, it simultaneously triggers an asynchronous background refresh. * The Next Query: Once that background refresh completes (usually within milliseconds to a few seconds depending on the Iceberg catalog's response time), all subsequent queries will see the new data. 2. The "10-Minute" Rule (Background Polling) Even if no queries are running, StarRocks has a passive background thread (

background_refresh_metadata_interval_millis

) that polls your metastore every 10 minutes (default) to keep the cache warm. 3. Summary of Freshness Expectations Action | Latency to see new data | Impact on Query Performance Normal Querying | Variable. If the cache is >60s old, the first query sees old data but triggers a refresh; the second query (seconds later) sees new data. | None (Asynchronous) Manual

REFRESH

| Instant. The next query is guaranteed to see the latest data. | Small delay for the

REFRESH

command itself. Idle Table | Up to 10 minutes. The background poll will eventually update the cache even without queries. | None Key Takeaway for Shared-Data In a shared-data architecture, this metadata logic lives on the Frontend (FE). If your pipeline requires that the very first query after a data load must see the new results, you should trigger a

REFRESH EXTERNAL TABLE <table_name>

at the end of your ingestion pipeline. If a ~60s "eventual consistency" is acceptable, the default settings are usually sufficient. Note: The 60-second async trigger (

iceberg_table_cache_refresh_interval_sec

) was officially introduced in v3.5.7. If you are on an earlier version, the system relies primarily on the 10-minute background poll or manual refreshes. References • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/System_variable.md • data_source/feature-support-data-lake-analytics.md