http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I am using an iceberg external cata...
# questions-and-troubleshooting
a
@Rocky I am using an iceberg external catalog. While that works fine, I have a problem with freshness, (unable to read fresh data since the metadata refresh interval is 10m), I am planning to add another external catalog with caching disabled
r
For Iceberg external catalogs, StarRocks uses a dedicated metadata caching mechanism. If you are experiencing freshness issues due to the 10-minute refresh interval (controlled by the global FE configuration 
background_refresh_metadata_interval_millis
), creating a second catalog with caching disabled is a valid strategy. 1. Disable Caching for a Specific Catalog Since v3.3.3, you can use the 
enable_iceberg_metadata_cache
property when creating or altering an Iceberg catalog. Setting this to 
false
will bypass the metadata cache and fetch the latest snapshot from the source for every query.
Copy code
sql
CREATE EXTERNAL CATALOG iceberg_fresh
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive",
    "hive.metastore.uris" = "thrift://...",
    "enable_iceberg_metadata_cache" = "false"
);
2. Lower the Refresh Interval (v3.5.7+) If you are on version v3.5.7 or later, you can adjust the refresh interval at the catalog level without disabling the cache entirely. This is often more performant than a full disable.
Copy code
sql
ALTER CATALOG your_iceberg_catalog SET (
    "iceberg_table_cache_refresh_interval_sec" = "60" -- Refresh every 1 minute
);
3. Global Configuration (Applies to all catalogs) The 10-minute interval you noticed is governed by a global FE dynamic configuration. You can reduce this globally if needed, though it may increase the load on your Metastore (Hive/Glue).
Copy code
sql
ADMIN SET FRONTEND CONFIG ("background_refresh_metadata_interval_millis" = "60000"); -- 1 minute
Key Differences for Iceberg: * 
REFRESH EXTERNAL TABLE
: Unlike Hive or Hudi, the 
REFRESH EXTERNAL TABLE
command is generally not used for Iceberg in StarRocks, as Iceberg's metadata is handled via file-based caching rather than the standard Metastore cache. * 
plan_mode
: Since v3.3.3, you can use the session variable 
SET plan_mode = 'distributed';
for tables with very large metadata to bypass local FE memory bottlenecks, though this is more about performance than freshness. Recommendation: Adding a second catalog with 
"enable_iceberg_metadata_cache" = "false"
is the most reliable way to ensure real-time freshness for specific queries without affecting the performance of your main cached catalog. References • sql-reference/sql-statements/table_bucket_part_index/REFRESH_EXTERNAL_TABLE.md • data_source/catalog/iceberg/iceberg_catalog.md • data_source/catalog/hive_catalog.md
a
@Rocky any other config needed to disable caching? Do i need to set / disable the below by any chance:
Copy code
enable_statistic_collect
iceberg_manifest_cache_with_column_statistics
iceberg_data_file_cache_memory_usage_ratio
iceberg_delete_file_cache_memory_usage_ratio
refresh_iceberg_manifest_min_length
iceberg_table_cache_refresh_interval_sec
r
To ensure a "fresh" Iceberg catalog that bypasses metadata caching, setting 
"enable_iceberg_metadata_cache" = "false"
 in the catalog properties is the primary and most important step. Regarding the specific configurations you listed, here is whether you need to adjust them: 1. Cache-related params (Redundant if cache is disabled) If you set 
"enable_iceberg_metadata_cache" = "false"
, the following parameters are ignored because the metadata cache is bypassed entirely for every query: * 
iceberg_manifest_cache_with_column_statistics
* 
iceberg_data_file_cache_memory_usage_ratio
* 
iceberg_delete_file_cache_memory_usage_ratio
* 
refresh_iceberg_manifest_min_length
* 
iceberg_table_cache_refresh_interval_sec
2. Statistics and Plan Freshness The parameter 
enable_statistic_collect
 (and the related session variable 
enable_stats
) controls whether StarRocks uses the Cost-Based Optimizer (CBO) statistics. Even if your metadata is fresh, the optimizer might use stale statistics cached in StarRocks' internal 
_statistics_
database. For the best "fresh" experience with Iceberg, do not disable statistics entirely. Instead, tell StarRocks to read statistics directly from the Iceberg metadata (which is updated with every write) rather than its internal cache: * In Catalog Properties:
Copy code
sql
    "enable_get_stats_from_external_metadata" = "true"
* As a Session Variable (recommended):
Copy code
sql
    SET enable_get_stats_from_external_metadata = true;
3. Metastore Freshness (New Tables/Databases) While 
enable_iceberg_metadata_cache
handles data freshness (new rows/snapshots), StarRocks also caches the list of tables and databases from the Hive Metastore/Glue. If you need to see newly created tables immediately: * Set 
"metastore_cache_refresh_interval_sec" = "0"
 in your catalog properties to disable the metastore cache. Recommended "Fresh" Catalog Definition:
Copy code
sql
CREATE EXTERNAL CATALOG iceberg_fresh
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive", -- or glue/rest
    "hive.metastore.uris" = "thrift://...",
    "enable_iceberg_metadata_cache" = "false", -- Bypasses snapshot/manifest cache
    "metastore_cache_refresh_interval_sec" = "0", -- Bypasses table/DB list cache
    "enable_get_stats_from_external_metadata" = "true" -- Uses fresh metadata for CBO
);
Note on Performance Disabling the cache will significantly increase the load on your Metastore (Hive/Glue) and object storage (S3/HDFS) because StarRocks must fetch the Iceberg 
version-hint
and metadata JSON for every query. This is usually acceptable for a specific "freshness" catalog but should be monitored for high-concurrency workloads should keep the cache enabled and use a shorter 
iceberg_table_cache_refresh_interval_sec
instead. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SET_CONFIG.md References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md • data_source/catalog/deltalake_catalog.md • data_source/catalog/hive_catalog.md
a
My another primary reason to to have a minimal fe memory footprint, will this take care of that
@Rocky My another primary reason to to have a minimal fe memory footprint, will this take care of that
r
Disabling the metadata cache will indeed help reduce the persistent memory footprint on the FE, but it won't prevent transient memory spikes during query planning. To achieve a truly minimal FE memory footprint for Iceberg, you should combine disabling the cache with Distributed Planning. 1. The Biggest Lever: 
plan_mode = distributed
 By default, StarRocks parses Iceberg manifest files (which can be very large) on the FE. If you have many large Iceberg tables, this is the most common cause of FE OOM or high memory usage. * Action: Set the session variable 
plan_mode
to 
distributed
. * Effect: This offloads the heavy lifting—reading, decompressing, and filtering Iceberg manifest files—to the BE/CN nodes. The FE only coordinates the plan, drastically reducing its memory consumption during the "Planning" phase.
Copy code
sql
SET GLOBAL plan_mode = 'distributed';
2. Disabling the Persistent Caches While your catalog setting 
enable_iceberg_metadata_cache = false
handles the snapshot/manifest pointers, there are other internal caches you can tune: * 
metastore_cache_refresh_interval_sec
: Set to 
0
in Catalog properties. This prevents the FE from caching the list of all databases and tables from Hive/Glue in memory. * 
iceberg_metadata_memory_cache_capacity
: (FE Config) This is a global limit for Iceberg metadata. You can lower this from the default 512MB to a smaller value (e.g., 64MB) in 
fe.conf
to cap how much memory the Iceberg connector can ever claim.
Copy code
properties
    iceberg_metadata_memory_cache_capacity = 67108864
3. Avoid "Internal Statistics" Bloat StarRocks often collects and stores statistics for external tables in its internal 
_statistics_
database (which lives on FE disk/memory). * Action: Ensure you are reading stats directly from Iceberg metadata instead of letting StarRocks generate its own. * Catalog Property: 
"enable_get_stats_from_external_metadata" = "true"
Summary Checklist for Minimal FE Memory: Feature | Setting | Why? Data Freshness | 
enable_iceberg_metadata_cache = false
| Bypasses snapshot cache. Planning Memory | 
plan_mode = 'distributed'
| Crucial. Offloads manifest parsing to BEs. Catalog Metadata | 
metastore_cache_refresh_interval_sec = 0
| Doesn't store table/DB lists in FE memory. Global Cap | 
iceberg_metadata_memory_cache_capacity
| Caps total Iceberg memory usage in FE. CBO Stats | 
enable_get_stats_from_external_metadata = true
| Avoids storing heavy stats objects in StarRocks. Result: By combining these, your FE will act more as a "stateless" router for Iceberg queries rather than a heavy metadata manager, which is exactly what you need for a minimal footprint. References • knowledge/trouble_shooting/fe_memory_problems.md • data_source/catalog/iceberg/iceberg_catalog.md • faq/Others.md • faq/fe_mem_faq.md
a
@Rocky which of these are catalog properties and which are others
r
To minimize the FE memory footprint and ensure data freshness, you should distinguish between Catalog Properties (set during catalog creation) and Session Variables/Global Configs (set via SQL or config files). 1. Catalog Properties These are set within the 
PROPERTIES
block of your 
CREATE EXTERNAL CATALOG
statement. They apply only to that specific catalog. * 
enable_iceberg_metadata_cache
: Yes. The most important for your goal. Setting this to 
false
prevents the FE from permanently caching Iceberg manifest data for this catalog. * 
metastore_cache_refresh_interval_sec
: Yes. Controls how often the FE refreshes the list of databases and tables from the metastore. Set to 
0
to disable caching of the table list. * 
enable_get_stats_from_external_metadata
: Yes. (Can also be a session variable). When 
true
, it tells StarRocks to read stats from Iceberg files instead of its own internal memory/storage. * 
iceberg_table_cache_refresh_interval_sec
: Yes. (v3.5.7+). If you decide to keep caching enabled but want more frequent updates. * 
iceberg_manifest_cache_with_column_statistics
: Yes. * 
iceberg_data_file_cache_memory_usage_ratio
: Yes. * 
iceberg_delete_file_cache_memory_usage_ratio
: Yes. * 
refresh_iceberg_manifest_min_length
: Yes. ────────── 2. Session Variables These are set using 
SET
(for your current connection) or 
SET GLOBAL
(for all future connections). They control the behavior of the query engine. * 
plan_mode
: Session Variable. Crucial for memory footprint. Set to 
'distributed'
to move the manifest parsing from FE to BE/CN nodes. * 
enable_get_stats_from_external_metadata
: Session Variable. You can toggle this on/off per session even if it's set in the catalog. ────────── 3. Global FE Configurations These are set in 
fe.conf
or via 
ADMIN SET FRONTEND CONFIG
. They apply to the entire cluster and all catalogs. * 
iceberg_metadata_memory_cache_capacity
: FE Config. This puts a hard cap on the total bytes the Iceberg connector can use for its internal metadata cache across all catalogs. * 
enable_statistic_collect
: FE Config. This is a global switch for the automatic statistics collection background task. * 
background_refresh_metadata_interval_millis
: FE Config. The global heartbeat interval for checking metastore changes. Summary Checklist for your "Minimal Memory" Catalog: Use this template for your new catalog:
Copy code
sql
CREATE EXTERNAL CATALOG iceberg_fresh
PROPERTIES (
    "type" = "iceberg",
    "iceberg.catalog.type" = "hive", -- or glue/rest
    "enable_iceberg_metadata_cache" = "false", -- Catalog property
    "metastore_cache_refresh_interval_sec" = "0", -- Catalog property
    "enable_get_stats_from_external_metadata" = "true" -- Catalog property
);

-- Then run this to offload planning memory to BEs:
SET GLOBAL plan_mode = 'distributed';
-- Then run this to ensure the optimizer uses fresh stats: SET GLOBAL enable_get_stats_from_external_metadata = true; ``` ## References - [administration/management/FE_configuration.md](https://docs.starrocks.io/docs/administration/management/FE_configuration)
2 Views
Open in Slack
PreviousNext
Powered by