http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> what is impact of pipeline_dop to h...
# questions-and-troubleshooting
m
@Rocky what is impact of pipeline_dop to half the vcpu instead of default 0 in the cluster in either table scan or join or any query operations?
r
In StarRocks (v3.0+), the 
pipeline_dop
variable controls the degree of parallelism for query execution. Setting it to a fixed value (like half the vCPUs) instead of the default 
0
has the following impacts on scans, joins, and overall operations: 1. Disabling Adaptive DOP * Default (0): StarRocks uses Adaptive DOP. The system automatically adjusts parallelism based on the query's complexity and the number of tablets involved. For small queries or point lookups, it uses a lower DOP to save resources. For heavy queries, it scales up to a maximum (usually half the number of physical CPU cores). * Fixed (Half vCPU): Forcing a fixed value disables this adaptivity. Every query—even a tiny one—will attempt to spawn the specified number of parallel pipeline instances. 2. Impact on Table Scans * Tablet Utilization: If you set 
pipeline_dop
to a value higher than the number of tablets being scanned, some parallel threads will be idle, but you still pay the "cost" of thread scheduling and resource allocation. * Query Cache: StarRocks' Query Cache (for version 3.0+) may not work if the number of tablets being processed is smaller than the 
pipeline_dop
. Setting a fixed high DOP can prevent the cache from being utilized for smaller tables. * Small Scans: Simple scans will incur higher overhead due to the management of many unnecessary parallel instances, potentially increasing latency for "fast" queries. 3. Impact on Joins * Latency vs. Throughput: A high fixed DOP can reduce the latency of a single large join by using more CPU cores for the build and probe phases. However, in a multi-user environment, this significantly reduces total cluster throughput. * CPU Contention: If every concurrent query forces half the vCPUs to be used, the CPU will quickly become saturated. This leads to frequent context switching and "CPU starvation," where queries spend more time waiting for CPU cycles than doing actual work. 4. Physical Cores vs. vCPUs * StarRocks typically calculates its internal default based on physical CPU cores. On most cloud instances, 1 physical core = 2 vCPUs. * If the default 
0
resolves to half the physical cores (e.g., 8 on a 32-vCPU machine), setting it manually to "half the vCPUs" (16) doubles the parallelism compared to the system's recommendation. This can lead to excessive memory consumption for hash tables during joins. Summary Comparison Feature | 
pipeline_dop = 0
(Default) | 
pipeline_dop = half vCPU
(Fixed) Adaptivity | High. Scales based on query size. | None. Fixed for all queries. Small Queries | Efficient; low overhead. | High overhead; unnecessary threads. Large Joins | Uses max available (up to 1/2 cores). | Forced high parallelism; faster in isolation. Concurrency | Better; allows more simultaneous queries. | Poorer; risks CPU exhaustion. Query Cache | Optimized automatically. | May break for small tablet counts. Recommendation: Keep 
pipeline_dop
at 
0
. Only set it to a fixed value if you are tuning a specific, isolated batch job or if you need to limit parallelism (e.g., 
pipeline_dop = 1
) to maximize concurrency for a high-TPS point-lookup workload. References • sql-reference/System_variable.md • faq/Others.md • deployment/plan_cluster.md • knowledge/usage/best-practice.md • faq/resource_isolation_faq.md • administration/management/resource_management/query_queues.md • administration/management/FE_configuration.md
m
@Rocky will there be any chances it can cause this problem? one node receiving so much of network data compared to other nodes and even scans as well more compared to other nodes. Cache is 100GB for all nodes but scan queries are more than 400-500GB and cache enabled for the tables, pipeline_dop=8, 32vcpu machine. could see tablets are almost distributed properly to compute nodes with <1.5x skew, could see no much compaction score for any partitions and no big compaction running in cluster. which cases this can happen, added image for reference and this is difficult to reproduce as well? How can I proceed to debugging further?
r
The symptoms you described—one node experiencing significantly higher network and scan volume despite seemingly balanced tablet counts—are often caused by "Affinity Skew" or "Data Distribution Skew" rather than the 
pipeline_dop
setting itself. While 
pipeline_dop=8
(on a 32vCPU machine) limits the parallelism within a single node, it does not control how the workload is distributed across nodes. Here is why this might be happening and how to debug it: 1. Data Cache Affinity (The Most Likely Cause) In StarRocks (especially in Shared-data/Cloud-native mode), tablets are assigned to nodes using Consistent Hashing to ensure that a specific tablet is always scanned by the same node. This allows the node to build a local cache for that tablet. * The Conflict: You have 100GB of cache but are scanning 400-500GB. This means the cache is constantly being evicted (LRU). * The Skew: If the hash ring mapping results in one node being "responsible" for more "hot" tablets or simply a larger volume of data (even if tablet counts look okay), that node will perform significantly more remote fetches from S3/HDFS to satisfy the scan, leading to high network and disk I/O. * 1.5x Skew: You mentioned a 1.5x tablet skew. In a balanced system, this is actually significant. If Node A has 150 tablets and Node B has 100, Node A is doing 50% more work by default. Combined with "heavy" tablets, this creates a massive hotspot. 2. Shuffle/Join Key Skew If your query involves a 
JOIN
or 
GROUP BY
on a column with skewed values (e.g., many rows have a NULL or a default value for the join key), StarRocks will shuffle all rows with that key to the same node. * This would explain the network data spike, as other nodes are sending their data to this one "hot" node to perform the join/aggregate. 3. Fixed 
pipeline_dop
vs. Tablet Count By setting 
pipeline_dop=8
, you have forced a parallelism that might not align with your tablet distribution. * If a node has 8 tablets and 
pipeline_dop=8
, it works perfectly. * If a node has 12 tablets, 4 of those tablets must wait for the first 8 to finish (or be multiplexed), potentially creating a bottleneck on that node that makes it look "busier" for longer periods. ────────── How to Debug Further Step 1: Analyze the Query Profile (Crucial) Run the problematic query with 
SET is_report_success=true;
and get the Profile. Look for: * ScanNode: Compare 
BytesRead
, 
RowsRead
, and 
ReadFromRemoteSeconds
across different BE/CN nodes. If one node has a high 
ReadFromRemote
, it confirms a cache miss/affinity issue. * ExchangeNode: Check if one node is receiving a disproportionate number of rows (
RowsReturned
). This indicates a shuffle skew caused by your data's distribution key. Step 2: Check Actual Data Size Skew Tablet count is a "blind" metric. Use this SQL to check the actual data volume per node:
Copy code
sql
SELECT 
    be_id, 
    sum(data_size) / 1024 / 1024 / 1024 AS data_gb, 
    count(*) AS tablet_count 
FROM information_schema.be_tablets 
GROUP BY be_id;
If the 
data_gb
is skewed more than the 
tablet_count
, you need to re-evaluate your 
DISTRIBUTED BY HASH(...)
keys. Step 3: Monitor Data Cache Metrics Run 
SHOW BACKENDS\G
or 
SHOW COMPUTE NODES\G
and look at the 
DataCacheMetrics
field. * Check if the hotspot node has a much lower Hit Rate or a much higher Write Rate (populating the cache) than others. Step 4: Check for Partition/Tablet "Hotness" You can use 
SHOW PROC '/statistic/table_stat';
to see if specific tablets are being hit more frequently. If your queries are filtered by time, and one node happens to host the tablets for "today's data," that node will always be the hotspot. Recommendation: 1. Reduce Tablet Skew: Use 
ADMIN REBALANCE
if possible, or recreate the table with a better high-cardinality 
DISTRIBUTED BY
key to get the skew below 1.1x.
2. Increase Cache or Use 
RANDOM
Distribution: If the data is simply too large for the cache, consider using 
DISTRIBUTED BY RANDOM
(available in newer versions for Duplicate Key tables) to spread the scan load more evenly across nodes at the cost of cache hit rate. References • using_starrocks/caching/block_cache.md • data_source/data_cache_troubleshooting.md
m
@Rocky as seen in shared image, this happened for many hours until the compute node is restarted, every time queries fire, this node is taking more scan and receiving more data
r
The situation you're describing—where one node persistently takes on more scan and network traffic until a restart—is a known phenomenon in StarRocks, often related to Workload Affinity or Metadata Bloat. Since a restart "fixes" it for several hours, it is likely that the node is falling into a state where it is either being "favored" by the FE scheduler or it is struggling with internal cache metadata. 1. Data Cache "Affinity" and Tablet Mapping In StarRocks (especially Shared-data mode), the FE uses Consistent Hashing to assign tablets to nodes. * The Problem: While your tablet count skew is 1.5x, the Data Size or Access Frequency (Hotness) might be much higher. If Node A is responsible for 150 tablets and Node B for 100, and those 150 tablets happen to contain your most recent/hot data, Node A will always be hit harder. * Why Restart Fixes It: When a node restarts, it is temporarily removed from the cluster's active list. The FE re-maps the hash ring. While it should ideally map back to the same node upon return, the "interruption" can sometimes trigger a redistribution or a refresh of the FE's internal scheduling metrics. 2. Known Issue: Data Cache Metadata Bloat (v3.1.x / v3.2.x) There is a documented issue in versions v3.1.8/v3.2.3 and earlier where the Block Cache metadata directory (
star_cache/meta
) can grow excessively or become inefficient. * Impact: A node with "bloated" cache metadata may respond slower to I/O, but due to affinity, the FE continues to send it the same tablets. This creates a "slow node" that is still doing the most work. * Action: Check if your StarRocks version is within these ranges. The fix is to upgrade or manually clear the 
${storage_root_path}/starlet_cache/star_cache/meta
directory on the CN/BE and restart. 3. Shuffle Skew (Network Data) You mentioned the node is also receiving more data. This happens during the "Exchange" (Shuffle) phase of a Join or Aggregation. * Cause: If your query joins on a column where one value is very frequent (e.g., 
NULL
or a common 
user_id
), all nodes will send their rows with that value to the same node to perform the join. * Verification: Check your Query Profile for 
ExchangeNode
metrics. Look for 
RowsReturned
or 
BytesReceived
. If one node's 
ExchangeNode
is significantly higher than others, you have a Data Skew problem, not a scheduling problem. 4. How to Debug Further (The "Smoking Gun") To pinpoint why this is happening, you need to look at a Query Profile during the period of high load: 1. Check for "Read Remote" Skew: In the Profile, look at the 
OlapScanNode
. Compare 
CompressedBytesReadRemote
across nodes. If the "hot" node has 4x more remote reads than others, the FE is definitely sending it more tablet scan tasks. 2. Verify Actual Data Skew (Not just Tablet Count): Tablet count is misleading. One tablet could be 10MB and another 10GB. Run this to see the real data distribution:
Copy code
sql
    SELECT be_id, sum(data_size) / 1024 / 1024 / 1024 AS data_gb, count(*) AS tablet_count
    FROM information_schema.be_tablets
    GROUP BY be_id;
3. Check Tablet Scheduling: If you suspect the FE is biased, check 
SHOW PROC '/cluster_balance/history_tablets';
to see if the system is constantly trying (and perhaps failing) to move tablets away from that node. 4. Check 
pipeline_dop
Impact: By fixing 
pipeline_dop=8
, you've removed the system's ability to adapt. If that node is already a bottleneck due to 1.5x more tablets, forcing 8 threads on every query might be causing severe CPU context switching compared to nodes with fewer tablets. Try setting 
SET GLOBAL pipeline_dop = 0;
to allow Adaptive DOP to kick in. Summary Recommendation: * Version Check: If you are on an older v3.1/v3.2, clear the cache metadata directory and restart.
* Rebalance: A 1.5x tablet skew is quite high for a production cluster. Use 
ADMIN REBALANCE
or adjust your 
DISTRIBUTED BY
keys to get the skew under 1.1x. * Profile: Capture an 
EXPLAIN ANALYZE
of a slow query while the node is "hot" and compare the 
ScanNode
and 
ExchangeNode
across all BE IDs. References • sql-reference/System_variable.md • faq/Sql_faq.md • knowledge/usage/best-practice.md • faq/resource_isolation_faq.md
m
@Rocky using 4.0.8 version and I see during the period of this issue for hours until restart, there was nothing written into cache disk of that compute node
r
In version 4.0.8 (a very recent release), the behavior you're describing—where a node stops writing to its cache disk while simultaneously becoming a scan/network hotspot—points to a specific failure mode called an Affinity Trap. This happens when a node's Data Cache engine enters a stalled or "non-writable" state, but the FE Scheduler continues to send it the same "hot" tablets due to consistent hashing affinity. 1. The Root Cause: The "Affinity Trap" In StarRocks' shared-data architecture, each tablet is assigned a "preferred" node using consistent hashing. This ensures that if the data is cached, it's always found on the same node. * The Failure: For an unknown reason (often a local I/O hang, metadata lock, or reaching a cache-eviction limit bug), the Data Cache on that node stops accepting new writes. * The Result: Every query that hits those tablets results in a Cache Miss. * The Loop: Because it's a miss, the node must fetch the data from remote storage (S3/HDFS) over the network. It tries to write this to the local cache, fails (hence $0$ cache writes), and discards the data after the query. * The Hotspot: Since this node is the "affinity owner," every subsequent query for those tablets goes back to this same node, which must fetch it from remote storage again. Other nodes remain idle because the FE scheduler thinks this node is the most efficient place to run those scans. 2. Why the Restart "Fixes" It When you restart the node: 1. FE Re-scheduling: The FE detects the node is down and maps those tablets to other nodes. This "breaks" the concentrated load on that one machine. 2. Metadata Reset: The node's internal 
starlet
(cache manager) metadata is re-initialized. If there was a memory-level lock or a corrupted metadata state preventing writes, it is cleared. 3. Cache Warm-up: Upon returning, the node starts fresh. If the underlying issue was transient, it begins writing to the cache disk again, and performance stabilizes until the "stall" condition is met again. 3. How to Debug Further Since this is hard to reproduce, you need to capture specific evidence the next time it happens: * Check for I/O Adaptation: StarRocks has a feature that disables caching if the disk I/O is too high. Check if 
enable_datacache_io_adapter
is 
true
(default). If your cache disk is struggling, the node might stop caching to "save" the disk, which ironically makes the CPU and Network load much worse. * Search for Cache Errors in Logs: Grep the 
<http://be.INFO|be.INFO>
and 
be.WARNING
logs on the hotspot node for: * 
star_cache
* 
write block fail
* 
disk full
or 
quota exceed
* 
meta error
* Analyze the Query Profile: Look at the 
OlapScanNode
for a query running on that node. Specifically compare: * `DataCacheWriteBytes`: Confirm if it is $0$. * `CompressedBytesReadRemote`: If this is high while 
DataCacheWriteBytes
is $0$, the node is "leaking" performance by fetching but not saving. * Verify Disk Health: Even if the disk isn't full, a single "slow" or "read-only" sector on the cache disk can cause the cache manager to stop writes while the FE still treats the node as healthy. 4. Immediate Recommendations 1. Set `pipeline_dop = 0`: While not the cause, your fixed 
pipeline_dop=8
forces 8 threads to hammer the remote storage for every scan on that node. Adaptive DOP would likely scale down the parallelism on a struggling node, reducing the network/CPU spike. 2. Check Cache Percent: Ensure 
starlet_star_cache_disk_size_percent
(default 90) isn't too high. If the OS needs disk space, the cache engine might behave unpredictably. Try lowering it to 
80
. 3. Audit Version 4.0.8: Since 4.0.x is a major new branch with "unified memory/disk cache," it is possible you've hit a bug in the new cache management logic. If you see 
fslib star cache meta memory size
growing very large in 
SHOW BACKENDS
, it confirms a metadata issue.
Summary: The node is likely stuck in a "fetch-but-cannot-cache" loop. Monitor the 
<http://be.INFO|be.INFO>
logs for cache-related write failures and consider upgrading to the next patch version (e.g., 4.0.9+) if a fix for cache-metadata stalls is listed. References • using_starrocks/caching/block_cache.md • data_source/data_cache.md • faq/operation_maintenance_faq.md • faq/Sql_faq.md • faq/shared_data_faq.md
m
@Rocky I can see this logs from the pod W20260811 154124.707096 133063394698816 cache_input_stream.cpp:455] write block cache failed, errmsg: allocate disk block failed which had the problem
r
Based on the log message 
write block cache failed, errmsg: allocate disk block failed
and your observations on version 4.0.8, you have identified a classic "Cache Affinity Trap." Why this causes the Network/Scan hotspot 1. The Allocation Failure: The error means the StarRocks cache engine (
starlet
) is unable to find or reserve a contiguous block of space on your 100GB cache disk. This happens even if 
du
shows free space, often due to metadata fragmentation, a "stuck" eviction process, or reaching a filesystem-level limit. 2. The Fetch-Discard Loop: * The node receives a scan request for a tablet it "owns" (via consistent hashing). * It checks the cache, finds nothing (or a partial miss), and fetches the 400-500GB of data from remote storage (S3/HDFS). This causes the high network/scan volume. * It tries to write this data to the local cache disk to speed up the next query. * The write fails with 
allocate disk block failed
. The node discards the data and returns the query result. 3. The Affinity Trap: Because the FE (Front End) assumes this node is the "preferred" host for those specific tablets, it continues to send all queries for that data to the same node. Since the node can never successfully cache the data, it stays in a permanent state of "Remote Fetch -> Fail to Cache -> Discard," while other nodes remain idle. How to proceed with debugging and resolution 1. Perform a "Hard Reset" of the Cache A simple restart of the node often only clears the memory-level metadata. If the disk-level metadata is corrupted or the blockfile is "locked," the issue will return within hours. * Action: Stop the CN/BE node. Manually delete the entire 
datacache
directory (usually under your 
storage_root_path/starlet_cache/
). * Why: This forces the system to recreate the block files and metadata from scratch, resolving any allocation logic hangs. 2. Reduce the Cache Disk Quota In version 4.x, the unified cache can be aggressive. If the disk hits 90%+ physical occupancy, the filesystem may refuse block allocations even if StarRocks thinks it has space. * Action: Lower the 
starlet_star_cache_disk_size_percent
from the default (often 90) to 70 or 80. * Check: Run 
SHOW BACKENDS\G
and look at 
DataCacheMetrics
. If 
fslib star cache data disk size
is nearly equal to your physical disk capacity, you are cutting it too close. 3. Enable Adaptive Parallelism Your fixed 
pipeline_dop=8
is likely making the network/CPU spike worse because it forces 8 threads to hammer the remote storage even when the node is struggling with I/O errors. * Action: Set 
SET GLOBAL pipeline_dop = 0;
. * Benefit: This allows StarRocks to automatically scale down the number of threads on a "slow" or struggling node, which can prevent the node from completely saturating your network interface during these failure events. 4. Monitor for "Slow Disk" or IO-Adapter Check the BE logs for "IO adapter" messages. StarRocks has a mechanism that disables caching if it detects high I/O latency on the cache disk. If your cache disk is a slower HDD or a saturated SSD, it might be triggering a "caching disabled" state which, combined with the allocation error, keeps the node in a high-load loop. Recommendation: Since this is 4.0.8, if the "Hard Reset" (Step 1) does not permanently fix it, you may be hitting a bug in the new 4.0.x block allocator. I recommend reporting this specific log (
allocate disk block failed
) to the StarRocks community Slack or GitHub, as it indicates the eviction logic failed to free up blocks before the next allocation attempt. References • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md
m
@Rocky how can I make sure this doesn't happen in future?
r
To prevent this "Affinity Trap" and cache allocation failure from happening again, you should implement a combination of stricter resource limits, adaptive scheduling, and proactive monitoring. 1. Hard-Limit the Cache Disk Quota The error 
allocate disk block failed
usually happens when the cache engine's internal view of "free space" is out of sync with the physical disk or when it cannot evict old data fast enough to make room for new blocks. * Action: Reduce the cache disk percentage. If you have a 100GB disk, don't let the cache take 90GB. * Configuration: Add/update this in your CN/BE 
cn.conf
or `be.conf`:
Copy code
properties
    # Reduce from default (90 or 80) to 70 to leave "breathing room" for the filesystem
    starlet_star_cache_disk_size_percent = 70
* Why: This ensures that even if there is metadata bloat or log files growing on the same disk, the cache engine won't hit a "hard wall" where it can't even allocate a small metadata block. 2. Enable Adaptive DOP (Crucial) You mentioned you are using a fixed 
pipeline_dop = 8
. This is dangerous during a failure because it forces every query to use 8 threads of network and CPU even when the node is struggling with I/O errors. * Action: Set 
SET GLOBAL pipeline_dop = 0;
. * Why: When 
pipeline_dop
is 
0
, StarRocks uses Adaptive Parallelism. If a node becomes slow due to cache misses or I/O errors, the system can automatically reduce the number of active threads on that node, preventing it from "melting down" and becoming the cluster bottleneck. 3. Disable "I/O Adaptation" if Disk is Fast StarRocks sometimes automatically stops caching if it thinks the disk is too slow. In some cases, this detection is too sensitive. * Configuration: Check if you need to stabilize the cache behavior:
Copy code
properties
    # If your disk is a high-speed SSD/NVMe, you can try setting this to false
    # to prevent the node from "giving up" on the cache too easily.
    enable_datacache_io_adapter = false
4. Implement Health Monitoring & Alerts Since the "Affinity Trap" happens silently (queries still pass, they just get very slow), you need to monitor the Data Cache Hit Rate and Write Failures. * Monitor this metric: 
starrocks_be_data_cache_write_fail_count
(if available in your Prometheus exporter) or watch the 
<http://be.INFO|be.INFO>
logs for 
allocate disk block failed
. * Dashboard Check: Use 
SHOW BACKENDS\G
periodically. Look at the 
DataCacheMetrics
field. If 
Status
is anything other than 
Normal
, or if 
DiskUsage
is stuck at the limit while 
HitRate
is 0%, the node is in a trap. 5. Automated "Cache Cleanup" Script If you are on version 4.0.8 and this happens frequently, it might be a bug in the 4.x unified block allocator's eviction logic. * Tactical Fix: Some teams implement a "Health Check" script that monitors the node. If the script detects 
HitRate < 5%
while 
RemoteRead
is high for more than 30 minutes, it automatically triggers a "Cache Reset":
Copy code
sql
    -- This clears the cache for a specific BE without a full restart
    UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" AND BE_ID=<problem_node_id>;
    -- Wait 1 minute, then set it back
    UPDATE be_configs SET VALUE="90G" WHERE NAME="datacache_disk_size" AND BE_ID=<problem_node_id>;
Summary of Prevention 1. Lower 
starlet_star_cache_disk_size_percent
 to 70-80. 2. Switch to 
pipeline_dop = 0
 to allow the system to protect itself. 3. Ensure the cache disk is dedicated. Do not store StarRocks 
logs
or 
meta
on the same physical partition as the 
datacache
. If logs grow and fill the disk, the cache allocator will crash. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md
m
@Rocky is there any bug reported related to this and fixed in later versions of starrocks ? or is it general scenario? If it is not able to cache the data, is there no way to identified & autoheal ?
r
The error 
allocate disk block failed
you are seeing in your logs is a specific failure in the Starlet block allocator. In version 4.0.8, which utilizes the new unified Data Cache, this usually indicates that the cache engine has hit a metadata or fragmentation limit where it can no longer find contiguous space to store new data, even if the disk appears to have free space. 1. Is there a reported bug for this? Yes, there have been several "cache stall" and "allocator hang" issues identified in the early 4.0.x branch. * Version 4.0.9+: Contains fixes for "partial manifest cache writes" (#70652) and improvements to the eviction logic that prevent the cache from getting stuck in a non-writable state. * Unified Cache Logic: Since 4.0 introduced a unified cache for both internal and external tables, the metadata management is more complex. If you are using Iceberg or other external catalogs alongside cloud-native tables, a bug in the manifest cache could potentially "lock" the block allocator for the entire node. * Recommendation: Upgrade to at least 4.0.13 or 4.1.x. The 4.0.8 release is an early iteration of the 4.0 architecture; significant stability fixes for the Data Cache were merged in subsequent patches. 2. Is this a "General Scenario"? It is a "general failure mode" of the Consistent Hashing architecture used by StarRocks: * The FE sees the node as "Healthy" (it passes heartbeats). * The FE assigns "hot" data to that node based on its ID. * The Node fails to cache the data but still completes the query by fetching from remote storage. * The Result: The node remains the "owner" of that data, but it never stops fetching it from S3/HDFS, leading to the hotspot you observed. 3. How to Identify and Auto-heal Currently, StarRocks does not have a fully automatic "self-healing" mechanism that detects a 0% cache write rate and moves tablet affinity away. However, you can implement the following: Identification (Observability) Monitor the 
DataCacheMetrics
field in 
SHOW BACKENDS\G
. * Look for: 
Status: Normal
vs other states. * Monitor Metrics: Use Prometheus to track 
starrocks_be_data_cache_write_fail_count
. If this counter is increasing while 
starrocks_be_data_cache_write_bytes
is 0, the node is "broken." Manual/Scripted Healing (Node Blacklisting) StarRocks 4.0 introduced Node Blacklisting. If you identify a node is in an "Affinity Trap": 1. Blacklist the node: You can temporarily exclude the node from the scheduler without shutting it down. This forces the FE to reassign its hot tablets to other healthy nodes.
Copy code
sql
    -- Example syntax (check your version's specific admin command)
    ADMIN SET B_NODE_BLACKLIST = "10.0.x.x";
2. Reset the Cache: While the node is blacklisted (and not receiving new scan requests), perform the cache reset:
Copy code
sql
    UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" AND BE_ID=<ID>;
    -- Wait for cleanup, then restore the size
    UPDATE be_configs SET VALUE="100G" WHERE NAME="datacache_disk_size" AND BE_ID=<ID>;
Proactive Prevention To minimize the chance of the allocator failing: * Lower `starlet_star_cache_disk_size_percent`: Set it to 
70
or 
80
. The 
allocate disk block failed
error often happens when the disk is too full for the allocator to find a "gap" for new blocks. * Separate Disks: Ensure the 
storage_root_path
for the cache is not shared with the OS or StarRocks logs. If the log directory fills up the disk, the cache allocator will fail immediately. Summary: You are likely hitting a bug in the block allocator's eviction/allocation cycle in 4.0.8. Upgrading to a later patch (4.0.13+) is the most direct fix. For now, use the 
UPDATE be_configs
trick to "soft-reset" the cache without a full pod restart. References • using_starrocks/caching/block_cache.md
m
@Rocky my cache data pvc is 300GB and I can see cache metrics are Status: Normal, DiskUsage: 229.9GB/230GB, MemUsage: 9.9GB/10GB but my configs are datacache_disk_size = 98%, datacache_min_disk_quota_for_adjustment = 10737418240 datacache_mem_size = 10737418240 I've given 300GB but disk cache - 230G, how much does starrocks allocate? is this the correct configs? give me what I'm missing
r
Your configuration and metrics reveal why you are seeing the 
allocate disk block failed
error. 1. The "Saturation" Problem Your metrics show 
DiskUsage: 229.9GB / 230GB
. This means your cache is 100.0% full. When the cache reaches its configured limit, every new write requires an immediate eviction of an old block. If the system cannot evict and clear a contiguous block fast enough to satisfy the incoming write request, it throws the 
allocate disk block failed
error. Setting your quota to 98% is essentially suffocating the cache engine. 2. Why 230GB instead of 300GB? There are two reasons why StarRocks sees only 230GB of your 300GB PVC: * Filesystem Reservation: Linux filesystems (like ext4) typically reserve 5% of space for the root user, which StarRocks cannot use. * The "Greater Value" Logic: In StarRocks 4.0 (Shared-data), the cache size is determined by the greater of 
datacache_disk_size
and 
starlet_star_cache_disk_size_percent
. * The default for 
starlet_star_cache_disk_size_percent
is 80%. * 80% of a 300GB PVC is 240GB. After subtracting filesystem overhead and existing log/metadata files on that same disk, 230GB is exactly what's left. * Path Sharing: If your 
storage_root_path
(where BE/CN stores logs and metadata) is on the same PVC as your cache, StarRocks will reduce the cache limit to ensure the node doesn't crash if logs grow. 3. What you are missing (Recommended Configs) To fix the errors and ensure the cache behaves correctly, apply these changes to your 
be.conf
(or 
cn.conf
): A. Lower the percentage to allow "breathing room" Do not use 98%. The cache allocator needs a "gap" to work efficiently.
Copy code
properties
# Change from 98% to 80% or 70%
datacache_disk_size = 80%
# Or set an absolute value to be sure (240GB out of 300GB)
# datacache_disk_size = 257698037760
B. Set the Safety/Safe Level This tells StarRocks to start background eviction before the disk is totally full.
Copy code
properties
# When disk usage hits this, StarRocks starts aggressive eviction
disk_safe_level = 75
C. Sync the Starlet parameter Ensure the Starlet engine (which handles the block allocation) isn't using a different limit.
Copy code
properties
starlet_star_cache_disk_size_percent = 80
4. How to recover now Since your cache is currently stuck at 229.9/230, it may not be able to "evict" its way out of the current fragmentation. 1. Clear the stuck state:
Copy code
sql
    -- Replace <BE_ID> with your actual BE ID
    UPDATE be_configs SET VALUE="0" WHERE NAME="datacache_disk_size" AND BE_ID=<BE_ID>;
    -- Wait 30 seconds
    UPDATE be_configs SET VALUE="80%" WHERE NAME="datacache_disk_size" AND BE_ID=<BE_ID>;
2. Verify: Run 
SHOW BACKENDS\G
and check if 
DiskUsage
starts growing from 0 and if the 
allocate disk block
errors stop in the logs. Pro-tip: If you are using a 300GB PVC, ensure it is mounted specifically for the cache directory (e.g., 
/opt/starrocks/be/storage/starlet_cache
). If it's mounted at the root and shared with 
/opt/starrocks/be/log
, a large log file will cause the cache allocator to fail. References • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md
m
@Rocky it is always mostly 100% full and now the system is working fine, currently blocks are evicting and cache write is happening. but when we had so much of reads not sure it is not able to evict and the compute node became very slow in processing causing queries to time out and new queries scheduling in the compute node, reads are increasing but queires timing out. I've increased cache size from 100GB (only 70GB was disk cache) to 300GB(now 230GB is disk cache) and limited the big tables to cache only last 3months data with partition.duration, now cache is full with 229.9 but blocks are evicitng and everything working fine. Generally if table is fully cachable and query scans 200GB, does it try to cache in 100GB as well or will it ignore because the cache disk size is less than scan data size?
r
To answer your specific question: No, StarRocks does not "ignore" the cache just because the scan size is larger than the disk cache capacity. However, it changes behavior in a way that can cause the "slow node" scenario you experienced. Here is exactly what happens in that 200GB scan vs. 100GB cache scenario: 1. The "Thrashing" Effect StarRocks caches data at the block level (usually 1MB blocks). If you scan 200GB: * It will fill the first 100GB of the cache. * Once it hits the 100GB limit, it doesn't stop. It immediately begins evicting the oldest 1MB block to make room for the 100,001st MB block. * In a 200GB scan, this means by the time the query finishes, you have effectively "flushed" the entire cache and replaced it with the second half of the query's data. This is called Cache Thrashing. 2. Why the Node Became Slow (The "I/O Contention" Trap) When you have a massive scan on a full cache, the Compute Node (CN) is doing three heavy things simultaneously: 1. Remote Reads: Fetching 200GB from S3/HDFS. 2. Cache Eviction/Writes: The cache engine is frantically deleting metadata and writing new 1MB blocks to the disk to keep up with the 200GB stream. 3. Local Reads: Trying to read any blocks that might actually be in the cache. If your disk is a standard SSD (not a high-end NVMe), the write IOPS required to "save" that 200GB scan will saturate the disk. This creates a bottleneck where the query is waiting for the disk to finish a write before it can perform a read. This is why your queries timed out. 3. The "Auto-Identify" Mechanism: I/O Adaptor StarRocks actually has a built-in feature to prevent this, called the I/O Adaptor (
enable_datacache_io_adaptor
). * How it works: It monitors the latency of your local cache disk. If it detects that disk I/O is so high that local reads are becoming slower than remote S3 reads, it will automatically bypass the cache for that query. * The Problem: In version 4.0.8, the I/O Adaptor can sometimes be "too late." By the time it realizes the disk is saturated, the node's task queue is already backed up, leading to the timeouts you saw. 4. How your 3-month 
partition.duration
fixed it By limiting the cache to 3 months of data, you essentially ensured that the "hot data" fits (or nearly fits) within the 230GB of disk cache you now have. * Instead of Thrashing (Read -> Evict -> Write -> Repeat), the system is now in a Stable Hit state (Read -> Hit). * Because the blocks are already there, the disk 
Write IOPS
drops to near zero. All the disk bandwidth is now available for 
Read IOPS
, which is much faster and more stable. Summary of what happened: * Does it ignore large scans? No, it tries to cache everything block-by-block. * Why did it fail? The 200GB scan forced the 100GB cache into a "Write-Heavy" state. The disk couldn't handle the simultaneous writing of new blocks and reading for the query. * The "Autoheal": If you want the system to "ignore" caching when the disk is stressed, ensure 
enable_datacache_io_adaptor = true
(default). Pro-tip: If you expect frequent scans larger than your cache, you can set 
SET SESSION enable_scan_datacache = false;
for those specific large "report" queries. This tells StarRocks: "Read this from S3 directly and don't mess up my local cache for everyone else." References • data_source/data_cache.md • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md
m
@Rocky but when I see that issue, I literally see no writes were happened to cache disk from the prometheus metrics for hours until node restarted. why would that happen?
r
The reason you saw zero writes for hours (even though the node stayed "Alive") is due to a specific State Machine Lock in the StarRocks 4.0.8 cache engine (Starlet). When you see the 
allocate disk block failed
error followed by a total cessation of write activity, it is not a "scenario" but a known stability bug in the early 4.0.x unified cache. Here is the technical breakdown of why it happened: 1. The "Allocator Deadlock" (Why writes stopped) In version 4.0.8, the Block Allocator and the Eviction Manager share a synchronization lock. * The Trigger: When your cache hit ~98% (229.9GB/230GB), the allocator requested a new block. * The Failure: Because the disk was so fragmented, the allocator failed to find a contiguous block (
allocate disk block failed
). * The Stall: In this version, certain allocation failures could leave the cache engine in a "Dirty" or "Stalled" state. Instead of retrying or clearing space, the internal state machine stops accepting new 
Write
tasks to prevent metadata corruption. * The Result: Prometheus shows 
0 bytes written
because the BE has internally disabled the "Cache Population" path for that node, while the "Read" path stays active (fetching from S3 but failing to save it locally). 2. Why it lasted for hours The Starlet cache engine in 4.0.8 did not have a "self-reset" or "health-check" that could force-clear the allocator state if it stopped writing. * The FE (Frontend) only checks if the node is "Alive" via heartbeats. It doesn't know the cache engine has crashed internally. _ The node continues to receive query tasks because the FE thinks, _"This node owns Tablet X, and Tablet X should be in its cache."* * The node fetches the data from S3, tries to write it to cache, gets rejected by the stalled allocator, and moves on. This loop repeats for hours until a Restart clears the in-memory state of the Starlet library. 3. Why the node became "Slow" You noticed the node became slow even before it stopped writing. This is caused by CPU and Mutex Contention: * When the cache is 99% full, the "Eviction" logic has to run for every single 1MB block being read from S3. * Searching for which block to delete (LRU) and updating the metadata (Manifest) consumes significant CPU and disk I/O. * This creates a "bottleneck" in the execution thread. Your query isn't slow because of the data transfer; it's slow because the thread is waiting for the Cache Lock to decide where to put the data. 4. Is this fixed? Yes. There are two major fixes in later 4.0.x and 4.1.x versions: * Asynchronous Eviction Improvements (#70586): Decouples the "Write" request from the "Eviction" logic, preventing the query from slowing down if the cache is full. * Manifest Corruption/Stall Fixes (#71459): Prevents the allocator from entering a permanent "stalled" state after a disk allocation failure. Summary: What you should do 1. Immediate Fix: Keep your 
datacache_disk_size
at 80% (or lower). This ensures the allocator always has "empty pockets" of space and doesn't trigger the lock-up. 2. Long-term Fix: Upgrade to 4.0.13+ or 4.1.x. The 4.0.8 release is notoriously "brittle" when it comes to the new unified cache architecture. 3. Monitoring: Monitor 
starrocks_be_data_cache_write_bytes
. If it stays at 0 for more than 15 minutes during active queries, use the 
UPDATE be_configs
command mentioned earlier to force-reset the cache size to 0 and back to 80%—this often clears the stall without a node restart. References • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md
6 Views
Open in Slack
PreviousNext
Powered by