< Rocky> when we use shared architechture and we have only c StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> when we use shared architechture an...

Zurab Kiknavelidze

06/12/2026, 7:10 AM

@Rocky when we use shared architechture and we have only compute nodes and we still create some internal tables or mvs they go into be/storage?

Rocky

06/12/2026, 7:10 AM

In a StarRocks shared-data (shared-storage) architecture where you use Compute Nodes (CNs) instead of traditional Backends (BEs), the data for internal tables and Materialized Views (MVs) is handled differently than in the shared-nothing architecture: 1. Where the Persistent Data Goes When you create an internal table or an MV in shared-data mode, the data is stored persistently in your remote object storage (e.g., S3, HDFS, MinIO, or GCS). * These tables are internally typed as

CLOUD_NATIVE

. * The remote storage is the "source of truth." If you lose all your CN nodes, your data remains safe in the object store. 2. What Happens in
be/storage
(or
cn/storage
)? CN nodes still use a local storage directory, defined by the

storage_root_path

configuration (which defaults to

${STARROCKS_HOME}/storage

). However, in this architecture, this directory is used strictly for caching: * Data Cache: The actual data files are cached under a subdirectory called

starlet_cache

(e.g.,

storage/starlet_cache

). * Purpose: This local cache exists only to accelerate queries. When you read data from an internal table or MV, the CN first checks this local cache. If the data is missing, it fetches it from the remote object storage and populates the cache for future use. Summary Table Component | Shared-Nothing (BEs) | Shared-Data (CNs) Persistent Storage | Local disks (

be/storage/data

) | Remote Object Storage (S3/HDFS/etc.) Local Disk Role | Permanent data storage | Cache only (

storage/starlet_cache

) Data Loss Risk | High if local disks are lost without replicas | None (data is safe in remote storage) Table Type |

OLAP

CLOUD_NATIVE

To verify the actual path of your data, you can run:

Copy code

sql
SHOW PROC '/dbs/<your_database_id>';

In the results, look at the

StoragePath

column. It will show the URI (like `s3://bucket-name/...`) where the actual table data is residing. References • deployment/shared_data/s3.md • deployment/shared_data/hdfs.md • deployment/shared_data/minio.md • deployment/shared_data/azure.md • deployment/shared_data/gcs.md • using_starrocks/caching/block_cache.md • quick_start/shared-data.md

Zurab Kiknavelidze

06/12/2026, 7:14 AM

@Rocky is it ok be/storage to bee in 100% allocated? and also how to check what is using this cache?

Rocky

06/12/2026, 7:14 AM

In a StarRocks shared-data (shared-storage) architecture, the behavior of the

be/storage

directory on your Compute Nodes (CN) is fundamentally different from the classic shared-nothing architecture. 1. Is it okay for
be/storage
to be 100% allocated? No, 100% actual disk usage is dangerous. While the Data Cache is designed to use a large portion of your disk, leaving 0% free space can cause node crashes, prevent log writing, and break features like "Spill to Disk." * The Default Limit: By default, StarRocks configures the Data Cache to use 80% of the available disk capacity (controlled by

starlet_star_cache_disk_size_percent

). * Why it looks full: The Data Cache management system (Starlet) often pre-allocates or "occupies" its quota. Even when it evicts old data to make room for new data, it *does not delete the local files*—it simply marks those blocks as overwritable. This means

du -sh

will always show high usage once the cache has warmed up. * The "Danger Zone": If your disk usage exceeds 90-95%, it is usually not the Data Cache's fault (since it respects its 80% limit). The extra space is likely being consumed by: * Logs: Check

storage/log/

. * Core Dumps: Check for

core

files if the CN has crashed recently. * Persistent Indexes: If you use Primary Key tables, their indexes are stored locally in

storage/persist

. * Query Spilling: If queries run out of memory, they may spill intermediate data to local disk. 2. How to check what is using the cache? There is currently no direct SQL command (like

SHOW CACHE BY TABLE

) that provides a granular breakdown of which specific table or Materialized View is occupying the most cache. The Data Cache operates at a block level, meaning it stores 1MB chunks of data rather than whole tables. However, you can check the status and health using these methods: A. View Global Cache Metrics (SQL) You can see how much space the cache is using across your nodes:

Copy code

sql
-- Check the DataCacheMetrics column
SHOW BACKENDS;

-- Or use the system view for more detail
SELECT * FROM information_schema.be_datacache_metrics;

B. View Detailed Cache Stats (API) For a deeper look into hit rates and disk usage per path, call the BE/CN's HTTP API:

Copy code

bash
curl http://<CN_IP>:<CN_HTTP_PORT>/api/datacache/stat

This returns a JSON with

block_cache_disk_used_bytes

block_cache_hit_rate

, and which specific local paths are being used. C. Monitor I/O and "Hot" Tables Since you can't see the "static" cache content easily, you should look at "active" usage: * Grafana: Use the official "StarRocks Shared-data" dashboard. Look at FSLIB Cache metrics. * Audit Log: Check which tables are frequently queried. In a shared-data cluster, frequently queried tables are the ones occupying the cache. * Profile: Run a query and check the

Scan

node in the Query Profile. It will show

DataCacheHitBytes

RemoteReadBytes

, telling you if that specific table/MV is currently benefitting from the cache. Summary Recommendation * Keep a buffer: Ensure your

starlet_star_cache_disk_size_percent

is low enough (e.g., 70-80%) to leave room for logs and system operations. * Cleanup: If the disk is at 100%, manually check for large files in

storage/log

storage/temp

. References • administration/management/monitoring/alert.md • deployment/plan_cluster.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/resource_management/spill_to_disk.md

Zurab Kiknavelidze

06/12/2026, 7:26 AM

@Rocky no we have be/storage separately and it does not allocate the whole disk just its own portion

Rocky

06/12/2026, 7:27 AM

In a StarRocks shared-data architecture, the

storage/

directory on your Compute Nodes (CN) is not for permanent data storage, but it is also not "safe" to have at 100% physical disk usage. 1. Is 100% allocation okay? It depends on what you mean by "allocated": * If the Cache "occupies" the space: This is normal. The Data Cache (stored in

storage/starlet_cache

) is designed to grow until it hits its configured limit (default is 80% of the disk/partition). It does not delete files when it evicts data; it simply marks them as "reusable." Therefore,

du -sh

will often show that the cache directory is taking up all its allowed space. This is healthy. * If the Physical Partition is at 100%: This is dangerous. If the OS reports 0 bytes free on the partition where

/storage

resides, your CN may crash or fail to perform essential tasks like: * Writing Logs: If logs cannot be written to

storage/log

, the process may hang. * Query Spilling: If a query runs out of memory and needs to "spill to disk" (

storage/temp

), it will fail. * Primary Key Indexes: If you use Primary Key tables with

LOCAL

persistent indexes, they are stored in

storage/persist

. If there is no space to grow these indexes, loads will fail. Recommendation: Ensure your cache limit (

starlet_star_cache_disk_size_percent

) is set to leave at least 10–20% of the partition free for system overhead. ────────── 2. How to check what is using the storage? Since

be/storage

contains several subdirectories, you should first check the disk usage at the folder level from the OS:

Copy code

bash
# Navigate to your storage root path
cd /path/to/be/storage
du -h -d 1

What you will find: 1. starlet_cache/
(or
datacache/
): This is the Data Cache. It contains the blocks of your internal tables and MVs. 2. `persist/`: If you have Primary Key tables with

persistent_index_type = LOCAL

, their indexes are stored here. Unlike the cache, this is not automatically evicted and will grow with your data. 3. `log/`: Contains BE/CN logs. If logging levels are set to

INFO

DEBUG

, this can grow very large. 4. `temp/`: Used for intermediate data when queries "spill" to disk. 5. `meta/`: Stores tablet metadata and local header information. ────────── 3. How to inspect the Cache contents? If you want to see how much of that "100%" is actually effective cache data versus just empty pre-allocated space: A. Using SQL Check the hit rate and capacity via the system metrics:

Copy code

sql
-- Check capacity and current usage percentage
SELECT * FROM information_schema.be_datacache_metrics;

-- Check if specific queries are hitting the cache
-- Look for 'DataCacheHitBytes' in the Query Profile

B. Using the HTTP API For the most detailed breakdown of the cache (including per-disk usage), run this from the CN node:

Copy code

bash
curl <http://localhost:8040/api/datacache/stat>

_Note: Replace

with your

be_http_port

._ C. Identifying "Hot" Tables There is no "list cache by table" command because the cache works on 1MB blocks. However, internal tables and MVs are cached based on access. To see which tables are "using" the cache, look at your most frequent queries in the *Audit Log*; those are the tables residing in your

be/storage

cache. References • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • deployment/shared_data/s3.md • faq/shared_data_faq.md • best_practices/primarykey_table.md References • best_practices/primarykey_table.md

Rocky

06/12/2026, 7:27 AM

• table_design/table_types/primary_key_table.md • administration/management/BE_configuration.md • faq/shared_data_faq.md

Zurab Kiknavelidze

06/12/2026, 7:48 AM

@Rocky we use version 3.5.16 and how can we check if the cache limit is truly 80%?

Rocky

06/12/2026, 7:48 AM

To verify the cache limit for StarRocks version 3.5.16, you can use several methods. In a shared-data architecture, the parameter that controls this is

starlet_star_cache_disk_size_percent

. 1. Check the Configured Percentage You can query the current configuration of all Compute Nodes (CNs) or Backends (BEs) directly via SQL:

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE NAME = 'starlet_star_cache_disk_size_percent';

* Default Value: 80 (meaning 80% of the disk). * Note: If you see multiple rows, ensure they are consistent across all nodes. 2. Check the Actual Byte Limit (Quota) While the percentage is the setting, the Quota is the actual calculated byte limit based on your disk size. You can see this in the

be_datacache_metrics

view:

Copy code

sql
SELECT 
    BE_ID, 
    STATUS, 
    DISK_QUOTA_BYTES / 1024 / 1024 / 1024 AS QUOTA_GB, 
    DISK_USED_BYTES / 1024 / 1024 / 1024 AS USED_GB 
FROM information_schema.be_datacache_metrics;

* DISK_QUOTA_BYTES: This is the hard limit. The cache will not exceed this value. * DISK_USED_BYTES: This is how much of that quota is currently "occupied." 3. Quick Overview via
SHOW BACKENDS
For a high-level summary, run:

Copy code

sql
SHOW BACKENDS\G

Look for the DataCacheMetrics
field. It will display a string similar to:

Status: Normal, DiskUsage: 450.00GB/500.00GB, MemUsage: ...

In this example,

500.00GB

is your 80% limit (the quota). How to change it if it's not 80%? If you need to adjust this limit (for example, to 70% to leave more room for logs), you must update the

cn.conf

be.conf

file on each node:

Copy code

properties
starlet_star_cache_disk_size_percent = 70

*Note: This parameter is not dynamic and requires a restart of the CN/BE process to take effect.* Important Verification Note As mentioned previously, the

DISK_USED_BYTES

and the actual disk usage reported by the OS (

du -sh

) might not decrease even after data is evicted. This is normal behavior as StarRocks marks blocks as "overwritable" rather than deleting the files from the filesystem to improve I/O for performance reasons of efficiency. References • using_starrocks/caching/block_cache.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md • deployment/shared_data/feature-support-shared-data.md • administration/management/monitoring/Monitor_and_Alert.md References • deployment/deploy_manually.md • administration/management/BE_configuration.md

Zurab Kiknavelidze

06/12/2026, 7:53 AM

@Rocky we have starlet_star_cache_disk_size_percent = 50 but I seee 100% quota in gb

Copy code

SELECT
    BE_ID,
    STATUS,
    DISK_QUOTA_BYTES / 1024 / 1024 / 1024 AS QUOTA_GB,
    DISK_USED_BYTES / 1024 / 1024 / 1024 AS USED_GB
FROM information_schema.be_datacache_metrics;

why?

Rocky

06/12/2026, 7:53 AM

The reason your

DISK_QUOTA_BYTES

shows 100% (or an unexpected amount) despite setting

starlet_star_cache_disk_size_percent = 50

is typically due to one of three reasons in StarRocks v3.5.x: 1. The "Greater Of Two" Rule StarRocks uses two different parameters to control the Data Cache size. It evaluates both and takes the greater value. *

starlet_star_cache_disk_size_percent

(The one you set to 50) *

datacache_disk_size

(A newer parameter introduced to unify cache settings) If

datacache_disk_size

is explicitly set to a fixed size (e.g.,

100G

) or a higher percentage (the default is often 80% if not specified), it will override your 50% setting. How to check: Run this SQL to see both values:

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE NAME IN ('starlet_star_cache_disk_size_percent', 'datacache_disk_size');

datacache_disk_size

is non-zero and results in a larger capacity than your 50% limit, that is why the quota is higher. 2. Automatic Scaling (Default behavior in 3.3+) In version 3.5.x, StarRocks has a feature called Data Cache Automatic Scaling (

datacache_auto_adjust_enable

). * If this is enabled (default is

true

in recent versions), the system ignores static percentage limits and dynamically manages the cache to maintain a "safe" disk level (usually 80%). * It will expand the cache if disk space is available and contract it if the disk gets too full. How to check:

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE NAME = 'datacache_auto_adjust_enable';

If this is

true

, your 50% setting is being overridden by the auto-balancer. To force the 50% limit, you would need to set

datacache_auto_adjust_enable = false

and restart. 3. Restart Requirement The parameter

starlet_star_cache_disk_size_percent

is not mutable. If you updated the

cn.conf

be.conf

file but did not restart the process, the backend is still using the old default (80%). How to verify: Check the

be_configs

view as shown in step 1. If the value in the table does not match your config file, the node has not picked up the change yet. Summary Checklist 1. Check for conflicts: Ensure

datacache_disk_size

is not set to a higher value. 2. Disable Auto-Adjust: If you want a strict 50% limit, set

datacache_auto_adjust_enable = false

in your config. 3. Restart: These changes require a full restart of the BE/CN service to recalculate the

DISK_QUOTA_BYTES

. ────────── Note on 100%: If

DISK_QUOTA_BYTES

exactly matches your total partition size, double-check that you haven't set

datacache_disk_size

100%

or a value larger than the physical disk. References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md • administration/management/BE_configuration.md • data_source/data_cache_troubleshooting.md • data_source/data_cache.md

Zurab Kiknavelidze

06/12/2026, 7:56 AM

@Rocky so the safest and enterprise solution is to set

datacache_auto_adjust_enable

to true?

Rocky

06/12/2026, 7:56 AM

In StarRocks 3.5.x, setting

datacache_auto_adjust_enable = true

is indeed considered the modern, "enterprise-recommended" approach for most deployments, but it comes with a trade-off between safety and predictability. 1. Why it is the "Safest" Solution The primary reason this is the "safest" approach is that it makes the cache disk-aware. * Prevents Crashes: If your disk suddenly fills up with logs, core dumps, or Primary Key persistent indexes, the auto-adjuster will detect the pressure (at the

disk_high_level

, default 90%) and shrink the cache to prevent the node from crashing due to "No space left on device." * Maximized ROI: Conversely, if the disk is mostly empty, it will expand the cache (up to

disk_safe_level

, default 80%) to ensure you are getting the best possible query performance from your hardware. 2. Why you see "100%" (or higher than 50%) The reason your

starlet_star_cache_disk_size_percent = 50

is being ignored is likely due to the priority logic in version 3.5: 1. The "Greater Of" Rule: The system looks at both

starlet_star_cache_disk_size_percent

and

datacache_disk_size

. It uses whichever one is larger. 2. Auto-Adjust Overrides Static Limits: When

datacache_auto_adjust_enable

true

, the system targets the

datacache_disk_safe_level

. By default, this is 80%. * Even if you set your limit to 50%, the auto-adjuster sees you have plenty of free space and "helpfully" expands the quota toward the 80% "safe level" to improve performance. 3. Comparison: Auto vs. Manual (Enterprise Perspective) Feature | ***Auto-Adjust (True)*** | ***Static Limit (False)*** Philosophy | "Use all safe space available." | "Strictly stick to this budget." Pros | Best performance; protects against disk-full crashes. | Predictable performance; guaranteed space for other apps. Cons | Quota can fluctuate; performance may vary if disk fills. | Risk of node crash if logs/indexes grow beyond the remaining space. Best For | Dedicated CN/BE nodes (Standard Enterprise setup). | Shared nodes where other processes need guaranteed disk space. How to enforce your 50% limit? If you have a specific reason to limit the cache to exactly 50% (e.g., you are sharing the disk with a very large non-StarRocks application), you must: 1. Set

datacache_auto_adjust_enable = false

. 2. Ensure

datacache_disk_size

is either not set or set to

50%

. 3. Restart the CN/BE. Recommendation: For a standard StarRocks cluster, we recommend keeping

datacache_auto_adjust_enable = true

but adjusting the safe level if you want more headroom:

Copy code

properties
datacache_auto_adjust_enable = true
datacache_disk_safe_level = 70  # Aim for 70% usage instead of 80%
datacache_disk_high_level = 85  # Start emergency eviction at 85%

References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache_troubleshooting.md bleshooting) References • administration/management/BE_configuration.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md

Zurab Kiknavelidze

06/12/2026, 8:17 AM

@Rocky I did not set datacache_auto_adjust_enable at all and it shows value - false it should be true by default no?

Rocky

06/12/2026, 8:17 AM

You are correct that the documentation often states the default is

true

(starting from v3.3), but there is a specific nuance in how StarRocks handles this. In StarRocks 3.5.x, if you see

false

be_configs

without having set it manually, it is likely due to the "Configuration Conflict" rule. Why it shows as
false
The Data Cache automatic scaling feature is designed to be a "hands-off" mode. StarRocks follows this logic: 1. Implicit Disabling: If you provide any explicit manual disk configuration (such as setting

starlet_star_cache_disk_size_percent

datacache_disk_size

), the system assumes you want to manage the capacity yourself. In some sub-versions of 3.x, this can cause the auto-adjuster to default to

false

to prevent it from fighting with your manual limit. 2. Shared-Data vs. Shared-Storage: If you are in a shared-data (Cloud Native) architecture, the parameter

datacache_unified_instance_enable

true

by default (since v3.4). When this is enabled, the older parameter

starlet_star_cache_disk_size_percent

is actually ignored. The system expects you to use

datacache_disk_size

. The 100% Quota Mystery If

datacache_auto_adjust_enable

is indeed

false

, but you see a 100% quota (matching your full disk size), it usually means: * Defaulting to Max: If

datacache_disk_size

is not set (default 0) and

datacache_auto_adjust_enable

false

, the backend may default to using the entire available space of the partition designated for cache because no "budget" was successfully applied. * Parameter Priority: Since you are on 3.5, the system prefers

datacache_disk_size

. If you only set the old

starlet_..._percent

and left the new ones at default, the system might be falling back to a "use-all" state. Recommended "Enterprise" Fix To get the 80% limit you want with the safety of the auto-adjuster in version 3.5.16, you should explicitly set the following in your

be.conf

(or

cn.conf

Copy code

properties
# 1. Enable the auto-adjuster explicitly
datacache_auto_adjust_enable = true

# 2. Set your desired target (the 80% you were looking for)
datacache_disk_safe_level = 80

# 3. Ensure the high-water mark is safe
datacache_disk_high_level = 90

After applying these, restart the BE/CN. Then, run your query again:

Copy code

sql
SELECT * FROM information_schema.be_configs 
WHERE NAME LIKE '%datacache%auto_adjust%';

It should now show

true

, and your

DISK_QUOTA_BYTES

be_datacache_metrics

should reflect ~80% of your disk capacity. References • data_source/data_cache.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md • data_source/data_cache_troubleshooting.md • using_starrocks/caching/block_cache.md References • administration/management/BE_configuration.md • data_source/data_cache.md • sql-reference/System_variable.md

Zurab Kiknavelidze

06/12/2026, 8:19 AM

@Rocky which conflicts it? I have this "datacache_auto_adjust_enable = true datacache_enable = true datacache_disk_size = 40G datacache_mem_size = 3G datacache_priority = 10 # Each pod will create its own subdirectory under /opt/starrocks/cn/storage datacache_disk_path = /opt/starrocks/cn/storage/datacache"

Rocky

06/12/2026, 8:20 AM

The configuration parameter datacache_disk_size = 40G
is what is causing the conflict. In StarRocks v3.5.x, the system logic for the Data Cache follows these rules: 1. Static Limit vs. Dynamic Scaling (The "False" Value) The parameters

datacache_disk_size

(when set to a physical value like

40G

) and

datacache_auto_adjust_enable

are mutually exclusive in their logic: * Static Limit: When you set

datacache_disk_size = 40G

, you are giving the system a fixed budget. * Auto-Adjust: This feature is designed to scale the cache up and down based on disk pressure (targeting 60%–80%–90%). * Conflict: If you provide a hard physical limit, StarRocks assumes you want a static configuration and automatically disables the auto-adjuster. This is why your

be_configs

table shows

false

even though you wrote

true

in the file. 2. The "Greater Of" Rule (The 100% Quota) You mentioned in your first prompt that you have

starlet_star_cache_disk_size_percent = 50

. In StarRocks v3.4+, the unified Data Cache calculates your effective quota by taking the greater value between: 1.

datacache_disk_size

(which you set to

40G

) 2.

starlet_star_cache_disk_size_percent

(which you set to

50%

) Why you see 100%: If your disk/partition at

/opt/starrocks/cn/storage/datacache

is exactly 40GB or 80GB, the system will allocate based on the rules above. For example, if your PVC/Disk is 80GB: *

40G

is 50%. *

50%

is 40G. * The system uses 40G. If your disk is perceived as 40GB total by the container, it shows 100%. 3. Recommended "Enterprise" Configuration If you want the recommended auto-scaling behavior (which safely stays around 80%), you should remove the fixed size and let the balancer do its job. Update your config to this:

Copy code

properties
# Enable the feature
datacache_auto_adjust_enable = true
datacache_enable = true

# REMOVE or set to 0 to let Auto-Adjust take control
datacache_disk_size = 0 

# Ensure the old parameter doesn't override the auto-adjuster
starlet_star_cache_disk_size_percent = 0

# (Optional) Set your target safe level
datacache_disk_safe_level = 80

# Keep your path
datacache_disk_path = /opt/starrocks/cn/storage/datacache

Note: After changing these, you must restart the CN/BE process. The

DISK_QUOTA_BYTES

will then recalculate based on the physical disk size detected at your

datacache_disk_path

. References • data_source/data_cache.md • administration/management/BE_configuration.md • using_starrocks/caching/block_cache.md • deployment/shared_data/feature-support-shared-data.md • data_source/data_cache_troubleshooting.md disk_path`. References • using_starrocks/caching/block_cache.md • sql-reference/System_variable.md • data_source/data_cache.md • deployment/shared_data/feature-support-shared-data.md

2 Views

Open in Slack

Previous Next