http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> When using Starrocks with external ...
# questions-and-troubleshooting
s
@Rocky When using Starrocks with external Iceberg tables, my frontend nodes keeps running out of memory every few days due to metadata even though I have set relevant metadata cache env variables on fe nodes. We are using Starrocks 4.0
Copy code
enable_iceberg_metadata_disk_cache = true
iceberg_metadata_disk_cache_capacity = 4gb
iceberg_metadata_memory_cache_capacity = 512mb
iceberg_metadata_cache_max_entry_size = 32mb
r
It sounds like you are encountering a known issue where the StarRocks Frontend (FE) memory usage grows over time due to the accumulation of Iceberg metadata objects, eventually leading to an Out of Memory (OOM) error. Even with the metadata cache limits you have set, the FE can still OOM because those parameters primarily control the raw metadata file cache, but not necessarily the deserialized Java objects (like 
TableMetadata
, 
BaseSnapshot
, or 
ManifestFile
objects) generated during query planning or background refreshes. To resolve this, I recommend the following steps: 1. Enable Distributed Metadata Planning By default, StarRocks often uses a "local" planning mode for Iceberg where the FE node is responsible for parsing all manifest files. For tables with many snapshots or files, this is the primary cause of FE OOM. Set the 
plan_mode
to 
distributed
to offload the manifest parsing tasks to the Backend (BE) nodes:
Copy code
sql
-- Set globally for all sessions
SET GLOBAL plan_mode = 'distributed';
This reduces the FE's memory burden significantly as the BEs will handle the heavy lifting of metadata retrieval and filtering. 2. Configure Memory Usage Ratios (v3.5.6+) If you are on version 3.5.6 or later, StarRocks introduced specific ratio-based controls for Iceberg caches in the FE heap. You can add these to your `fe.conf`:
Copy code
properties
# Limits the memory ratio for data/delete file manifest caches in FE heap
# Default is 0.1 (10% of heap)
iceberg_data_file_cache_memory_usage_ratio = 0.05
iceberg_delete_file_cache_memory_usage_ratio = 0.05
3. Adjust Background Refresh & Expiration The background refresh mechanism (
enable_background_refresh_connector_metadata = true
) periodically polls for updates. If you have thousands of tables, this can keep a large number of objects in the heap. * Reduce refresh interval: If your data doesn't change every few minutes, increase the interval.
Copy code
properties
    # Default is 600,000 ms (10 mins). Try 30 mins or 1 hour.
    background_refresh_metadata_interval_millis = 1800000
* Shorten inactivity expiration: Stop refreshing metadata for tables that haven't been accessed recently.
Copy code
properties
    # Default is 86400s (24 hours). Consider reducing to 4-8 hours.
    background_refresh_metadata_time_secs_since_last_access_secs = 28800
4. Check for Known Object Leaks There is a documented issue (e.g., GitHub #68426) where 
IcebergTable
objects and their associated snapshots are not cleared properly from the FE's 
MetadataMgr
in some older 3.x versions. * Recommendation: Check if your StarRocks version is up to date (at least v3.3.6+, v3.4.1+, or v3.5.0+). * Heap Analysis: If the OOM persists, use 
jmap -histo:live <pid>
or a heap dump to see if 
org.apache.iceberg.BaseSnapshot
or 
java.lang.String
objects are consuming the majority of the memory. 5. Disable Memory-Intensive Profiles If you are querying very large catalogs, the session profile can consume significant memory. You can disable the following in 
fe.conf
to save space:
Copy code
properties
proc_profile_mem_enable = false
proc_profile_cpu_enable = false
Summary of recommended actions: 1. Switch to 
SET GLOBAL plan_mode = 'distributed';
. 2. Upgrade to the latest patch version of your current release branch. 3. If on 3.5.6+, apply the 
usage_ratio
parameters.
In StarRocks 4.0, while metadata caching is designed to optimize Iceberg queries, the symptoms you described (OOM every few days) are often related to how the cache handles deserialized objects or misconfiguration of units. 1. Correct the Configuration Units The FE configuration parameters you listed (like 
iceberg_metadata_memory_cache_capacity
) generally require raw bytes as values. Unlike the BE 
mem_limit
, the FE parser for these specific Iceberg variables typically does not support units like 
mb
or 
gb
. If you set them as 
4gb
or 
512mb
in 
fe.conf
, StarRocks may be ignoring them or using the default values (Default: 512MB for memory, 2GB for disk). * Correct values (in bytes): * 
iceberg_metadata_disk_cache_capacity = 4294967296
(4GB) * 
iceberg_metadata_memory_cache_capacity = 536870912
(512MB) * 
iceberg_metadata_cache_max_entry_size = 33554432
(32MB) Action: Run 
ADMIN SHOW FRONTEND CONFIG LIKE '%iceberg_metadata%';
to verify if your custom values are actually active. 2. Monitor "Background Refresh" Memory Pressure StarRocks 4.0 enables 
enable_background_refresh_connector_metadata
by default. This periodically polls the Metastore and refreshes metadata for frequently accessed tables. * If you have a large number of Iceberg tables or very large manifest files, this background process can accumulate a large volume of deserialized Java objects in the heap. * Recommendation: If you don't require near-real-time metadata updates for all tables, try disabling this or increasing the interval:
Copy code
sql
    -- fe.conf
    enable_background_refresh_connector_metadata = false
    -- OR increase interval (default is 600,000ms / 10 mins)
    background_refresh_metadata_interval_millis = 3600000
3. Check Metadata Expiration The memory cache entries are kept for 24 hours by default (
iceberg_metadata_memory_cache_expiration_seconds = 86400
). If your FE heap is tight and you access many different tables/partitions over a few days, the cache might not be clearing fast enough. * Action: Try reducing the expiration to a few hours (e.g., 
14400
for 4 hours) to force more frequent eviction. 4. Known Issues in 4.0.0 There is a documented issue in early 4.0 versions (e.g., #65381) where the metadata cache could point to stale files or cause instability every 2-3 days. * Verify: Check your 
fe.warn.log
for 
File does not exist
or 
java.lang.OutOfMemoryError
stack traces. * Mitigation: If you see specific tables causing issues, you can manually trigger a refresh:
Copy code
sql
    REFRESH EXTERNAL TABLE iceberg_catalog.db_name.table_name;
If OOM persists, you may need to upgrade to a later 4.0.x patch (e.g., 4.0.12+) where several memory-related stability fixes for wide-string columns and metadata isolation were introduced. 5. FE JVM Heap Size Metadata for Iceberg (especially manifests and schema objects) is stored in the JVM Heap. If your FE only has a small heap (e.g., 8GB), a 512MB cache plus the overhead of processing metadata during planning can easily lead to OOM. * Action: Ensure your FE 
-Xmx
is at least 16GB (or more if you have millions of tablets/metadata objects) in 
fe.conf
. References • data_source/catalog/iceberg/iceberg_catalog.md • administration/management/BE_configuration.md • faq/Others.md • administration/management/FE_configuration.md • administration/management/resource_management/Memory_management.md • using_starrocks/caching/block_cache.md
2 Views
Open in Slack
PreviousNext
Powered by