@Rocky hey rockey good morning. if using data cache pvc for cn pod, there are serveral pod on k8s around 5~7. in some situation, query result sometimes different. can you exaplain why theses changes happened?

Good morning! It sounds like you are encountering a situation where your StarRocks Compute Nodes (CN) are returning inconsistent query results while using Data Cache (Block Cache) with PVCs. There are several reasons why query results might vary across different executions or nodes in a distributed environment like Kubernetes. Here are the most likely causes and how to troubleshoot them: 1. Metadata Staleness and Refresh Interval StarRocks caches metadata (file lists, partition info) for external catalogs (like Hive, Iceberg, or Hudi) to improve performance. * The Cause: By default, StarRocks refreshes external table metadata every 10 minutes. If your source data is updated, some CN pods might still be using the old metadata cached in the FE, while others (or subsequent queries) might trigger a refresh and see the new data. * The Symptom: Query results change suddenly after a few minutes or vary depending on whether the FE has refreshed its cache. * Solution: You can manually trigger a refresh using

REFRESH EXTERNAL TABLE <table_name>

or reduce the refresh interval by adjusting the

metacache_refresh_interval_sec

configuration in the FE. 2. Data Cache (Block Cache) Consistency Issues The Data Cache on your CN pods uses a key composed of

hash(filename) + fileModificationTime + blockId

. * The Cause: If your storage system (S3, HDFS, etc.) does not update the Modification Time (mtime) correctly when a file is overwritten, StarRocks will assume the cached block is still valid and read "old" data from the PVC. * The Symptom: Since you have 5–7 CN pods, some pods might have the old version of a file block in their PVC, while others (which haven't cached it yet) fetch the new version from remote storage. Depending on which CN the scheduler picks for a specific data split, the result changes. * Troubleshooting: Check if your source system reuses filenames without updating timestamps (common in some Iceberg or manual HDFS overwrite scenarios). * Solution: * Disable I/O adaptation by setting

enable_datacache_io_adapter = false

in CN configs to ensure it doesn't bypass cache inconsistently. * If the source data changed and you suspect the cache is stale, you can clear the cache by deleting the

datacache

directory in the PVC or by setting

datacache_disk_size = 0

and then back to your desired size via

admin set frontend config

. 3. Non-Deterministic Queries (Ordering) This is a common "trap" in distributed databases that is often mistaken for a cache issue. * The Cause: If your query uses

LIMIT

without an

ORDER BY

clause, or if the

ORDER BY

column has duplicate values (e.g., sorting by

date

where many rows have the same date), the result set can be different every time. * The Symptom: Data is technically correct, but the "top 10" rows keep changing. * Solution: Always use a unique column (like an ID) in your

ORDER BY

clause to ensure deterministic results. 4. Shared PVC Misconfiguration * The Cause: Block Cache is designed for local disk access. If your Kubernetes setup accidentally uses a shared PVC (ReadWriteMany) for multiple CN pods to store the Data Cache, it will lead to data corruption and highly unpredictable query results. * Solution: Ensure each CN pod has its own dedicated PVC (using

volumeClaimTemplates

in a StatefulSet). 5. Asynchronous Cache Population * The Cause: By default, Data Cache populates blocks asynchronously (

enable_datacache_async_populate_mode=true

). * The Symptom: The first query might read some data from remote storage and some from cache. While this usually only affects performance, if combined with the I/O Adaptation feature (which redirects requests to remote storage if the disk is busy), you might be getting a mix of cached and fresh data if the source changed. Recommended Troubleshooting Steps:

@Rocky what is exactrly enable_datacache_async_populate_mode?

enable_datacache_async_populate_mode

is a system variable in StarRocks that controls when and how the system writes data into the local cache (the PVC in your case) after reading it from remote storage (S3/HDFS/etc.). Here is the breakdown of how it works and why it relates to your "different results" issue: 1. What it does * Synchronous Mode (

false

): When you run a query, StarRocks reads the data from remote storage and waits to write it into the local cache before finishing the read operation. _ _Pros:* The cache is fully populated after the very first query. _ _Cons:* The first query might be slower because it has to wait for the disk write to finish. * Asynchronous Mode (

true

): When you run a query, StarRocks fetches the data for your query as fast as possible. In the background, a separate thread quietly copies that data into your local cache. _ _Pros:* The first query is faster because it doesn't wait for the cache to "warm up." _ _Cons:* If you run the same query twice in quick succession, the second query might still find the cache empty (or partially empty) if the background write hasn't finished yet. 2. Why it might cause "different" results Technically, this setting is for performance, not data correctness. However, in a Kubernetes environment with multiple CN pods and PVCs, it can contribute to inconsistent results in these specific scenarios: * Scenario A: Data has changed at the Source If the source file (e.g., in S3) was updated but the mtime (modification time) didn't change or StarRocks hasn't refreshed the metadata: * Pod A might have the "old" data already cached (Synchronous or earlier Async task). * Pod B might be currently reading "new" data from remote storage because its Async task hasn't filled the cache yet. * Because you have 5–7 pods, your query might be distributed across a mix of "stale-cached" pods and "fresh-remote" pods, leading to a mixed/incorrect result. * Scenario B: I/O Adaptor Interaction StarRocks has another feature called

enable_datacache_io_adaptor

(enabled by default). If a CN pod’s disk (PVC) is busy or the Async background write is heavy, the I/O adaptor might decide to skip the cache and read from remote storage instead. If the remote data has changed but the cache is stale, the results will fluctuate depending on whether the adaptor chose the cache or the remote source for that specific request. Summary of Defaults Version | Default Value Earlier than v3.3.0 |

false

(Synchronous) v3.3.0 and later |

true

(Asynchronous) How to test if this is the cause: Try running your query with the cache completely bypassed to see if the "different results" disappear:

sql
-- Disable cache for this session
SET enable_scan_datacache = false;
SELECT ... -- Run your query multiple times

If the results are now consistent, it means your local cache (PVC) contains stale data that doesn't match the remote storage. You may need to trigger a

REFRESH EXTERNAL TABLE

or clear the PVC data. References • sql-reference/System_variable.md • data_source/data_cache.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md • administration/management/BE_configuration.md