@Rocky we are currently running StarRocks 4.0.10 with Iceberg tables and seeing odd weird behavior where multiple SQLAlchemy clients are writing at the same time to the same table but the metadata after a couple hours get out of sync where it references a blob file in Azure that does not exist. I was looking through open issues but have not seen anything like this. Are there any known issues or limitations with concurrent StarRocks writes?

The behavior you are describing—Iceberg metadata referencing non-existent files on Azure—often occurs when concurrent writes are not properly serialized or when the underlying storage lacks the atomic "compare-and-swap" capabilities required by the Iceberg catalog. In StarRocks 4.0.10, there are several relevant factors and known improvements to consider: 1. Iceberg Commit Queue (Concurrency Management) StarRocks uses Optimistic Concurrency Control (OCC) for Iceberg. When multiple clients write to the same table, they may conflict at the commit stage. • The Feature: A new mechanism called the

enable_iceberg_commit_queue

was introduced (stabilized around the 4.0.x/4.1.x transition) to serialize commits to the same Iceberg table through a single-threaded executor on the FE. • Check Status: Verify if this is enabled in your environment. If disabled, high-concurrency writes might lead to commit conflicts where one writer succeeds but another's metadata becomes stale or inconsistent. • Related Fix in 4.0.10: The 4.0.10 release notes specifically mention fixes for race conditions in

ConnectorSinkPassthroughExchanger

and

LoadChannel

(#71848, #71843), as well as a fix for a race where legitimate transaction logs could be classified as "orphan" and dropped, leading to inconsistent states (#72237). 2. Azure Storage Consistency and Protocol The specific "blob file does not exist" error can be triggered by how Azure handles file visibility: • ADLS Gen2 vs. Blob Storage: If you are using the standard Azure Blob Storage (

wasb://

) instead of Azure Data Lake Storage Gen2 (

abfs://

), you may encounter consistency issues. Iceberg strongly recommends ADLS Gen2 because it supports atomic directory operations and stronger consistency required for reliable metadata updates. • Recommendation: Ensure your catalog is configured to use the

abfs

or

abfss

protocol. 3. Catalog Type Limitations If you are using a Hadoop Catalog (storing the metadata directly on the file system) on object storage, Iceberg cannot guarantee ACID compliance under concurrent writes because object stores lack atomic rename operations. • Recommendation: Use a "robust" catalog type like Hive Metastore (HMS), AWS Glue, or the REST Catalog. These catalogs provide the necessary locking mechanisms to prevent two writers from corrupting the table metadata. 4. Known Issues in 4.0.x • Metadata Cache: StarRocks 4.0 introduced changes to how metadata is cached. In some scenarios, the FE might be reading a cached version of the metadata that points to a file that was deleted or moved by a concurrent compaction or cleanup job. • Action: Try setting

enable_iceberg_metadata_cache = false

temporarily to see if the "missing file" errors persist. If the error disappears, it indicates a cache invalidation issue. Troubleshooting Steps 1. Identify the missing file type: Is the missing file a

.parquet

data file, or an Iceberg

.metadata.json

/

.avro

manifest? • If it's a data file, it likely means a transaction "committed" its metadata, but the file upload failed or was garbage-collected prematurely. • If it's a metadata file, it's almost certainly a catalog synchronization issue. 2. Check FE Logs: Look for

CommitFailedException

or "Base metadata location is not same as the current table metadata location". 3. Verify Configuration:

sql
   -- Check if commit queue is enabled
   ADMIN SHOW FRONTEND CONFIG LIKE 'enable_iceberg_commit_queue';

4. Update Protocol: If you are currently using

wasb

, migrate to

abfs

for ADLS Gen2. If the issue persists despite these settings, it is recommended to collect the FE

sys.log

from the time of the failed write and post it to the StarRocks GitHub issues or community Slack, as the race condition in #72237 fixed in 4.0.10 suggests there have been recent refinements in this specific area. References • release_notes/release-4.0.md

@Rocky does the commit queue work across FE nodes or just within a single FE node?

The behavior you're seeing—Iceberg metadata referencing non-existent files on Azure during concurrent writes—is likely due to a combination of Optimistic Concurrency Control (OCC) conflicts and known limitations in StarRocks v4.0.x metadata caching. 1. The Commit Queue (v4.1.0+ only) The

enable_iceberg_commit_queue

feature you may have seen in documentation was specifically introduced in StarRocks v4.1.0 to address this exact problem. * What it does: It serializes commits to the same Iceberg table through a single-threaded executor on the FE. * Version Limitation: Since you are on v4.0.10, this feature is not available. In your version, StarRocks relies entirely on the Iceberg library's standard OCC (retry-based) mechanism. When multiple SQLAlchemy clients write concurrently, they are all racing to update the table's metadata pointer. * Cross-FE Behavior: This queue is local to each FE node. Even in newer versions, if your SQLAlchemy clients are distributed across multiple FE nodes (e.g., via a load balancer), they will still conflict at the Iceberg Catalog level (HMS, Glue, or Azure). 2. Known Issues in StarRocks 4.0.x There is a documented issue (#65381) in the 4.0.x branch where StarRocks attempts to access stale metadata files that no longer exist. This happens because: * Metadata Cache: StarRocks 4.0 introduced a more aggressive metadata cache. If one writer updates the table, other FE nodes might still be using a cached version of the previous metadata file. When they attempt to read or commit, they point to a file that has been superseded or deleted by the "winning" writer. * Azure Consistency: If you are using the standard Azure Blob protocol (

wasb://

), you may encounter consistency issues. Iceberg requires atomic "rename" or "put-if-absent" operations which are only natively supported on Azure Data Lake Storage Gen2 (

abfs://

). Recommended Actions 1. Switch to ADLS Gen2 Protocol: Ensure your catalog and storage paths use

abfs://

or

abfss://

instead of

wasb://

. This is critical for Azure storage to support the atomic operations Iceberg needs for metadata consistency. 2. Disable/Clear Metadata Cache: To mitigate the "out of sync" behavior in v4.0.10, you can try disabling the FE metadata cache for Iceberg:

sql
    -- Check current settings
    ADMIN SHOW FRONTEND CONFIG LIKE '%iceberg_metadata%';

    -- Try disabling the cache (requires FE restart or dynamic set if applicable)
    -- Setting capacity to 0 effectively disables it
    update_iceberg_metadata_memory_cache_capacity = 0;

3. Upgrade to v4.1.x: The

enable_iceberg_commit_queue

and numerous fixes for Iceberg commit race conditions (such as #72237) were stabilized in the 4.1.x release. This version handles high-concurrency writes much more reliably. 4. Use a Robust Catalog: If you are using a Hadoop-style catalog (files on disk), concurrent writes are highly prone to corruption. Ensure you are using a proper catalog like Hive Metastore (HMS) or REST Catalog, which provides a locking mechanism to serialize the final metadata update. If you cannot upgrade immediately, the best workaround is to centralize writes to a single FE node (the Leader FE) to minimize cross-node cache synchronization issues, or reduce the concurrency of your SQLAlchemy clients. References • administration/management/FE_configuration.md • loading/azure.md • quick_start/iceberg.md • deployment/shared_data/azure.md