@Rocky I am currently using StarRocks with external tables built on Apache Iceberg and Minio I am encountering a strange issue where running the exact same query multiple times intermittently results in missing data or completely empty results. To investigate this further, I executed REFRESH EXTERNAL TABLE and immediately ran a SELECT COUNT(1) query. The row count, which is normally around 25,000, suddenly dropped to 6,000. However, after a few minutes, it went back up to 25,000 without any changes on my side. Have you experienced this kind of consistency issue or intermittent data loss with StarRocks + Iceberg? Any advice on how to troubleshoot or resolve this would be greatly appreciated. Thanks!

This behavior—intermittent missing rows or inconsistent counts when querying Apache Iceberg via StarRocks—usually points to a synchronization issue between the StarRocks Metadata Cache and the Iceberg Catalog (especially on object storage like MinIO). Based on the symptoms you described (the count dropping to 6,000 and then returning to 25,000 after several minutes), here are the most likely causes and troubleshooting steps: 1. Known Issue:

enable_connector_incremental_scan_ranges

There is a documented issue in some StarRocks versions (specifically 3.x) where an optimization for scan ranges can cause incorrect or empty results on Iceberg tables. * Fix: Try disabling this optimization in your session to see if the row count becomes consistent:

sql
    SET enable_connector_incremental_scan_ranges = false;

If this resolves the issue, you can set it globally or consider upgrading to a newer maintenance release where this bug is addressed. 2. Metadata Caching & Asynchronous Refresh StarRocks caches Iceberg metadata (like the latest snapshot ID and manifest list) to speed up query planning. * Asynchronous Behavior: Even if you run

REFRESH EXTERNAL TABLE

, StarRocks might still be in the middle of an asynchronous refresh cycle, or different Frontends (FEs) in your cluster might have different versions of the cache. * Refresh Interval: By default, StarRocks considers metadata "fresh" for 60 seconds (

iceberg_table_cache_refresh_interval_sec

). If a query is run after 60 seconds, StarRocks uses the existing (stale) cache to execute the query while triggering an asynchronous refresh in the background. This explains why your results "fixed themselves" after a few minutes. * Troubleshooting: _ Verify the current snapshot being read:

SELECT _ FROM table$snapshots;

* If you have multiple FEs, ensure they are synchronized. * Try lowering the refresh interval or disabling the cache temporarily to test:

sql
        -- FE dynamic configuration
        ADMIN SET FRONTEND CONFIG ("enable_background_refresh_connector_metadata" = "true");

3. StarRocks Data Cache (BE Side) If you have Data Cache enabled (which caches the actual Parquet/ORC data on the BE's local disk), there can be a mismatch if Iceberg snapshots are being expired or rewritten. * Check: See if

enable_scan_datacache

is set to

true

. * Verification: Run the query with the data cache disabled:

sql
    SELECT /_+ SET_VAR(enable_scan_datacache = false) _/ COUNT(1) FROM your_table;

If this returns the correct 25,000 rows while the cached query returns 6,000, then the issue lies in the BE-side data cache being stale or corrupted. 4. MinIO / Hadoop Catalog Consistency If your Iceberg catalog type is

hadoop

(storing metadata directly in a folder on MinIO without a separate Hive Metastore or REST catalog): * The Hadoop catalog is not recommended for S3/MinIO because it relies on atomic file renames/listings, which can be inconsistent on object storage. This often leads to "missing" snapshots or partial data visibility during write operations. * Advice: If possible, switch to the Iceberg REST Catalog or Hive Metastore to manage snapshots. Summary of Recommended Actions 1. Disable incremental scan ranges:

SET enable_connector_incremental_scan_ranges = false;

2. Bypass Data Cache: Test with

enable_scan_datacache = false

. 3. Check Metadata Sync: If you just updated the data from an external engine (like Spark), wait at least 1-2 minutes for StarRocks' background refresh to pick up the new snapshot, or ensure your

REFRESH

command completed successfully on all FEs. References • data_source/catalog/iceberg/iceberg_catalog.md • data_source/feature-support-data-lake-analytics.md

@Rocky i use starrocks 4.0.4

Since you are using StarRocks 4.0.4, you are on a version that introduced several significant optimizations for Apache Iceberg (like faster metadata parsing and backend metadata caching), but also some known bugs related to how these caches handle updates. The "drop to 6,000 rows" and then "recovery to 25,000" behavior strongly suggests a partial or stale metadata cache entry that gets corrected once the background refresh or TTL expires. 1. Primary Suspect:

enable_connector_incremental_scan_ranges

In StarRocks 4.x, there is a known issue where "stale scan ranges" can persist or be incorrectly calculated during query planning for Iceberg. This can cause the scanner to miss files or scan a partial list of manifest files. * Fix: Disable this optimization in your current session to see if the count stays consistent at 25,000:

sql
    SET enable_connector_incremental_scan_ranges = false;

2. Partial Manifest Cache Writes (Bug #70657) There is a documented bug in the 4.0.x line (fixed in later maintenance releases) where the Iceberg Manifest Cache could perform partial writes. If a query triggers a cache fill and only part of the manifest is written before being used by the scanner, you will see a subset of the data (like 6,000 instead of 25,000). * Workaround: You can temporarily bypass or clear the metadata cache by adjusting these FE/BE configurations:

sql
    -- In FE (to disable metadata caching temporarily for testing)
    ADMIN SET FRONTEND CONFIG ("enable_iceberg_metadata_cache" = "false");

* Permanent Fix: This issue is addressed in v4.0.6 and later. If disabling the cache fixes the issue, an upgrade is highly recommended. 3. Asynchronous Metadata Refresh In StarRocks 4.0, if

REFRESH EXTERNAL TABLE

is called, it triggers a refresh, but the FE might still serve queries using an older version of the metadata while the refresh is ongoing in the background across all FE nodes. * Check if you have multiple FEs. If so, verify if the "6,000 rows" result only happens when connected to a specific FE. * Check the snapshot ID StarRocks is actually reading during the "bad" query:

sql
    SELECT * FROM your_catalog.your_db.your_table$snapshots;

4. MinIO S3 Consistency & Credentials Since you are using MinIO, ensure your catalog properties include

enable_path_style_access

. In 4.0.4, metadata parsing was moved to the BE, and inconsistent access to the S3 endpoint can lead to "File not found" or partial listings if the BE nodes cannot reach MinIO consistently. * Check BE logs: Look for

[ExternalScan]

or

Iceberg

related errors in

be/log/be.INFO

or

be.WARNING

. Specifically, look for

manifest file not found

or

timeout

errors. Summary of Troubleshooting Steps for 4.0.4: 1. Test without incremental scan ranges:

SET enable_connector_incremental_scan_ranges = false;

2. Test without Data Cache:

SET enable_scan_datacache = false;

(to rule out local BE disk caching issues). 3. Inspect Snapshots: Run

SELECT count(*) FROM your_table$snapshots

to see if StarRocks sees the correct number of snapshots. 4. Upgrade: v4.0.4 has several known metadata consistency bugs that were resolved in v4.0.6 and v4.0.10. If the issue persists after disabling the scan range optimization, upgrading the FE/BE is the most reliable path.

@Rocky Also, was the issue related to the "enable_connector_incremental_scan_ranges" option not resolved in version 4.0.4? We are currently on this version and wondering if that could be the cause.

While StarRocks 4.0.4 includes many improvements for Apache Iceberg, the issue with

enable_connector_incremental_scan_ranges

is not fully resolved in that specific version. Furthermore, there are additional metadata-related bugs in the 4.0.x line that match your symptoms even more closely. Here is the specific breakdown for version 4.0.4: 1.

enable_connector_incremental_scan_ranges

Status In version 4.0.4, this optimization is enabled by default. While it improves performance for incremental updates, it is known to cause intermittent "missing data" or empty results when the underlying Iceberg metadata changes (e.g., after an

INSERT

or

VACUUM

in MinIO). * Recommendation: If you are experiencing inconsistent counts, you should still keep this set to

false

as a baseline:

sql
    SET enable_connector_incremental_scan_ranges = false;

2. The Manifest Cache Bug (The likely culprit) Your specific symptom—a count dropping from 25,000 to 6,000 and then "fixing itself" later—is characteristic of a Partial Manifest Cache Write. In StarRocks versions prior to 4.0.10, there was a bug (#70657 and #70675) where: * The FE/BE would attempt to cache Iceberg manifest files. * If the cache process was interrupted or only partially completed, StarRocks would sometimes read that partial cache entry as if it were the full manifest. * As a result, the query would only "see" a subset of the data files (e.g., only 6,000 rows). Once the cache expired or was refreshed fully, the count returned to normal. 3. Recommendations for 4.0.4 If you cannot upgrade immediately, use these workarounds to ensure data consistency: * Disable the Iceberg Metadata Cache: This forces StarRocks to read directly from MinIO manifests for every query, bypassing the buggy cache.

sql
    -- Check if this exists in your configuration
    ADMIN SET FRONTEND CONFIG ("enable_iceberg_metadata_cache" = "false");

* Disable the Data Cache: If you are using the BE-side data cache, disable it for the table to rule out stale block caching:

sql
    ALTER TABLE your_table SET ("enable_scan_datacache" = "false");

* Check for Multiple FEs: In 4.0.4, different FEs can have inconsistent views of the metadata cache. If you have a Load Balancer, one FE might have a "bad" cache entry while another has a "good" one, causing the results to flip-flop between 6k and 25k. Recommended Path The most stable version for the 4.0 branch currently is 4.0.10 (or later). It contains specific fixes for: 1. #70657: Partial manifest cache writes. 2. #70675: Lack of validation for manifest cache completeness. 3. #71913: Missing column statistics in manifest cache entries. Summary: The

enable_connector_incremental_scan_ranges

setting is a known factor, but the Manifest Cache bug is the most probable cause of your specific "dropping row count" issue in 4.0.4. Disabling the metadata cache or upgrading to 4.0.10 should resolve it.