@Rocky Subject: Inquiry regarding all potential scenarios for intermittent data visibility issues Hi StarRocks Team, I am experiencing an issue where data intermittently disappears and then reappears in our cluster. I would like to request a comprehensive explanation of all possible technical scenarios and root causes that could lead to this phenomenon. [Issue Description] Previously queryable data suddenly becomes invisible (returns 0 results) for several minutes, and then appears again without any manual intervention. This is often observed after a BE (Backend) restart or during background operations. [Request for All Possible Scenarios] Please provide a detailed breakdown of every potential cause for this "visibility gap," Could you please provide all possible technical scenarios that could cause this phenomenon in StarRocks?"

Hello, I'm Rocky! The phenomenon you are describing—where data intermittently disappears (returns 0 results) and then reappears after a few minutes—is typically referred to as a "visibility gap" or "consistency fluctuation." In StarRocks, this usually occurs due to a mismatch between the FE's metadata state (which thinks the data is visible) and the BE's local state (which may not yet have "applied" that data version). Below is a comprehensive breakdown of the potential technical scenarios and root causes for this behavior: 1. Replica Inconsistency & Version Lag (Shared-nothing) This is the most common cause in standard StarRocks clusters. * The Scenario: When data is loaded, StarRocks requires a "Write Quorum" (usually a majority of replicas) to succeed. If one replica is lagging or was offline during the load, it won't have the latest data version. * The Visibility Gap: If the FE's query scheduler routes a request to a lagging replica, the query will return 0 results (or old data). If the next query is routed to a healthy replica, the data "reappears." * Why it resolves: Background tasks (Clone or Peer Repair) eventually synchronize the lagging replica, making the data visible across all nodes. * How to verify: Run

ADMIN SHOW REPLICA STATUS FROM table_name;

and check for

Inconsistent

status or version mismatches. 2. Primary Key Index (PKI) Recovery & "Error" State If you are using Primary Key tables, visibility depends on the Primary Key Index. * The Scenario: After a BE restart, the BE must load the Primary Key Index into memory (or from the persistent index on disk). If the index loading fails or the tablet is marked as "in error state" during recovery, it cannot serve the latest data. * The Visibility Gap: A tablet in an "Error" state may temporarily return empty results or fall back to an older version until the index is successfully rebuilt or the tablet is repaired. * Why it resolves: The system automatically attempts to repair error-state tablets or rebuild indexes in the background. 3. Publish Version Latency * The Scenario: StarRocks uses a two-phase commit. A transaction moves to

VISIBLE

on the FE after the data is written, but the FE must then "Publish" the version to all BEs. * The Visibility Gap: If

enable_sync_publish

is disabled or there is high network/RPC congestion, there is a small window where the FE marks the transaction as successful, but some BEs haven't yet updated their local metadata to point to the new Rowset. * Why it resolves: The publish task eventually succeeds, and all BEs acknowledge the new version. 4. Shared-data Architecture: Metadata Sync & Vacuuming In shared-data (Cloud-native) mode, there are unique scenarios: * Metadata Stale in Follower FEs: If you have multiple FEs, a Follower FE might have a slightly stale view of the tablet's latest version compared to the Leader FE. * Vacuuming vs. Long-running Queries: If a query is using an old version that was just vacuumed (deleted) from object storage, it might fail or return inconsistent results. * Persistent Index Inconsistency: If the persistent index on object storage for a PK table becomes inconsistent during a BE restart or migration, it can lead to "missing" rows until the index is re-synchronized. 5. Known Bug: Version Recognition (v3.2.x specifically) * The Scenario: There was a known issue (#58404) in some v3.2 releases where a "defect in priority evaluation logic when loading tablets" caused incorrect version recognition after BE restarts. This could lead to temporary data loss where the BE failed to recognize the newest rowsets. * The Visibility Gap: Data would appear "gone" after a restart and only reappear after the BE correctly re-identified the versions through background metadata checks. 6. Tablet Migration & Load Balancing

@Rocky My environment is StarRocks Iceberg on S3. Is it possible for StarRocks to serve stale data by reading from an old disk cache snapshot immediately after a restart, and then start showing the latest data only after that cache expires?

In the context of StarRocks with Iceberg on S3, the phenomenon you are describing—where data "disappears" (returns 0 results) and then "reappears" a few minutes later—is almost certainly related to Metadata Cache Inconsistency, rather than just the BE's raw data cache. While a BE restart might seem like the trigger, the "visibility gap" is typically a result of how the FE (Frontend) manages Iceberg metadata snapshots. Here is a breakdown of the technical scenarios that cause this: 1. FE Metadata Cache (Most Likely Scenario) StarRocks caches Iceberg metadata (Snapshot IDs, manifest lists, and file locations) to avoid hitting S3/Metastore for every query. * The Scenario: If you recently loaded data into Iceberg, a new Snapshot ID is generated in the Metastore (Glue/HMS). However, StarRocks FEs cache the "Metadata Pointer" (the latest Snapshot ID). * The "0 Results" Gap: If the FE has a cached pointer to an older or empty snapshot, it will continue to query that snapshot until the cache expires. If the table was empty or had a different partition state in that old snapshot, you will see 0 results. * Why it resolves: By default, StarRocks considers metadata "fresh" for only 60 seconds (

iceberg_table_cache_refresh_interval_sec

). After this, it triggers an asynchronous refresh. Once the refresh completes, the latest data "reappears." * BE Restart Connection: A BE restart doesn't directly cause this, but it may lead to query retries or FE re-planning, which makes you more likely to notice the cache staleness as the system "settles" after the restart. 2. FE Metadata Disk Cache (Persistent "Stale" State) If you have enabled

enable_iceberg_metadata_disk_cache = true

in your FE configuration: * The Scenario: StarRocks saves Iceberg metadata to the FE's local disk. * The Visibility Gap: Upon an FE restart, StarRocks may load the metadata snapshot from its local disk cache instead of fetching the latest version from S3. If the disk-cached version is significantly older than the live version in S3, you will see stale data or 0 results. * Why it resolves: The background refresh mechanism (

background_refresh_metadata_interval_millis

, default 10 mins) eventually identifies the version mismatch and updates the metadata. 3. Consistency Between Multiple FEs If your cluster has multiple FEs (Leader and Followers): * The Scenario: A query might hit a Follower FE that has a stale metadata cache, while the Leader FE has already refreshed. * The Visibility Gap: If a BE restart causes the client to reconnect or the load balancer to shift queries to a different FE, you might intermittently hit a "stale" FE that returns 0 results, and then hit a "fresh" FE where the data appears. * Why it resolves: All FEs eventually sync their metadata with the Metastore through background polling. 4. BE Data Cache (Disk Cache on S3) The BEs maintain a "Data Cache" (blocks of data from S3 stored on local BE disks). * Is it possible to read a "stale snapshot" from here? * For Data: Generally, no. Iceberg is immutable (it writes new files for updates). If the FE gives the BE a list of new files, the BE won't find them in the old cache and will fetch from S3. * For Footers: BEs cache file footers (Parquet/ORC). If a file was somehow replaced with the exact same name/size/modification time (rare in Iceberg), the BE might read the old footer. * The "0 Results" link: Data Cache issues usually cause stale data (old rows), not 0 results. If you see 0 results, the problem is almost always at the FE Metadata level, where the FE tells the BE "there are no files to read." 5. AWS Glue / Metastore Sync Latency If you use AWS Glue as your Iceberg Metastore: * The Scenario: Sometimes there is a slight propagation delay in Glue itself.

@Rocky Earlier you explained this in terms of BE. Does the same also apply to CN?

Yes, the exact same logic and scenarios apply to CN (Compute Nodes). In StarRocks, for external catalog queries (like Iceberg on S3), a CN node functions identically to a BE node in terms of data processing and caching. Here is how the scenarios I mentioned earlier specifically manifest on CN nodes: 1. Data Cache (Block Cache) is Identical Both BE and CN nodes use the StarRocks Data Cache (formerly called Block Cache) to store data from S3 on local disks. * Behavior on Restart: If your CN nodes use persistent storage for their cache directory, the cache survives a restart. If they use ephemeral storage (common in Kubernetes), the cache is wiped. * Stale Data Risk: Just like a BE, a CN generates cache keys based on

filename + modification_time

. Since Iceberg data files are immutable, a "stale" file shouldn't technically exist—you either read the correct file or the file is gone. However, if the FE metadata is stale, it will tell the CN to read an old file list, and the CN will happily serve those files from its local disk cache if they are still there. 2. Distributed Metadata Planning For Iceberg specifically, StarRocks often uses a Distributed Plan for large tables. * The Scenario: The FE assigns the task of parsing Iceberg manifest files (which list the actual data files) to the BE/CN nodes. * The Visibility Gap: If a CN is struggling or has just restarted, there might be a slight delay in it becoming "ready" to participate in this distributed metadata parsing. * Impact: If the FE tries to distribute a metadata scan to a CN that is still initializing its cache or network stack, the query might fail or return incomplete results until the CN is fully healthy. 3. Consistency Across Nodes If you have a mix of BEs and CNs (or multiple CNs): * A query might be scheduled to CN-A for one execution and CN-B (or a BE) for the next. * If CN-A has a certain set of data blocks cached and CN-B does not, you might see a performance difference, but you shouldn't see a "visibility" difference unless the FE itself is providing different plans to different nodes (which brings us back to the FE Metadata Cache). 4. The "0 Results" Root Cause (Still FE-Driven) The "0 results" issue you described is almost always an FE-level event. _ The CN is just an "executor." It asks the FE: _"Which files should I read?"* _ If the FE says _"There are no files"* (because its Iceberg metadata cache hasn't refreshed yet), the CN returns 0 results. * Whether you use BEs or CNs for the computation doesn't change the fact that they are both waiting for the FE to "see" the new Iceberg snapshot. Summary for CN Environment: If you are running in a Cloud-Native/K8s environment with CNs: 1. Check Ephemeral vs. Persistent Cache: If your CN disks are wiped on every restart, they will always have a "cold start" period where queries to S3 are slower, but this shouldn't cause data to disappear (it just causes higher latency). 2. CN Readiness Probes: Ensure your Kubernetes readiness probes are correctly configured. If a CN is marked "Ready" by K8s but hasn't fully joined the StarRocks cluster metadata, the FE might send it queries that result in errors or empty sets. 3. Metadata Freshness: Focus on the FE settings I mentioned previously (

iceberg_table_cache_refresh_interval_sec

). This is the "brain" that controls what the CNs "see." In short: Switching from BE to CN (or vice versa) will not solve the visibility gap, as the root cause lives in the FE's metadata management logic. References • data_source/catalog/maxcompute_catalog.md • data_source/catalog/hive_catalog.md • administration/management/FE_configuration.md