http://starrocks.io logo
Join Slack
Powered by
Hi team, I have a problem with StarRocks Shared No...
# questions-and-troubleshooting
n
Hi team, I have a problem with StarRocks Shared Nothing 3.5.6. Data Cache Size limits are totally ignored. I have 
datacache_disk_size=300G
on all my BEs but currently my 
/datacache
directory is at 2TB on all nodes. If I got to the HTTP API 
/api/datacache/stat
, I can see that 
disk_quota_bytes
is around 3TB (about 80% of the total disk size). Automatic Scaling (
datacache_auto_adjust_enable
) is disabled. Just in case I also tried setting 
datacache_disk_high_level
, 
datacache_disk_safe_level
and 
datacache_disk_low_level
to 40, 30 and 20 respectively, but it made no difference. Is there anything else I should configure or does this seem like a bug?
k
can you get a list of the files under the 
/datacache
directory, 
ls -l
with files size info.
n
Untitled
Untitled
And the 
meta
dir just in case
Copy code
root@iad-prod-starrocks-be-8:/disk1/data/datacache# ls meta/ -lh
total 296M
-rw-r--r-- 1 starrocks starrocks  65M Oct 22 22:16 000077.sst
-rw-r--r-- 1 starrocks starrocks  65M Oct 22 22:16 000078.sst
-rw-r--r-- 1 starrocks starrocks  65M Oct 22 22:16 000079.sst
-rw-r--r-- 1 starrocks starrocks  33M Oct 22 22:16 000080.sst
-rw-r--r-- 1 starrocks starrocks  27M Oct 23 14:41 000081.log
-rw-r--r-- 1 starrocks starrocks  14M Oct 23 13:00 000082.sst
-rw-r--r-- 1 starrocks starrocks   16 Sep 24 09:01 CURRENT
-rw-r--r-- 1 starrocks starrocks   37 Sep 18 09:58 IDENTITY
-rw-r--r-- 1 starrocks starrocks    0 Sep 18 09:58 LOCK
-rw-r--r-- 1 starrocks starrocks  25M Oct 23 14:41 LOG
-rw-r--r-- 1 starrocks starrocks 3.7M Sep 23 11:28 LOG.old.1758627166031107
-rw-r--r-- 1 starrocks starrocks  69K Sep 23 12:32 LOG.old.1758630928173427
-rw-r--r-- 1 starrocks starrocks  36K Sep 23 12:35 LOG.old.1758631168546783
-rw-r--r-- 1 starrocks starrocks 704K Sep 24 08:59 LOG.old.1758704486625821
-rw-r--r-- 1 starrocks starrocks 3.5K Oct 23 14:41 MANIFEST-000033
-rw-r--r-- 1 starrocks starrocks 9.6K Sep 23 12:39 OPTIONS-000026
-rw-r--r-- 1 starrocks starrocks 9.6K Sep 24 09:01 OPTIONS-000036
k
datacache_disk_size was initially set to 300G or it was adjusted sometime later after initial setup?
n
300G from the first start
k
is a fresh install cluster from v3.5.6 or a cluster upgraded from an old version?
n
Fresh install
👌 1
Actually I started with 3.5.5 and almost immediately upgraded to 3.5.6
All the datacache files with non-zero size were created from 3.5.6 (I upgraded on Sep 19)
k
how's the configvars started with 
block_cache_*
?
n
Untitled.csv
Untitled.csv
I don't see the 
block_cache
directory though, and 
block_cache_enable
is 
false
k
how about the configvars starts with 
datacache_*
?
n
Just one BE this time to make it shorter but they're all identical
Untitled.csv
k
have you tried to enable the 
datacache_auto_adjust_enable
and set the 3 *_levels to 40/30/20 respectively, will the disk usage decrease?
n
I have not. At the moment I deleted the data and restarted BEs (the recommendation on the official docs), so I'll keep that in mind next time we're running out of space 🙂
k
yeah, the deletion should work.
I think when the auto adjust enabled, the datacache doesn't respect the disk_size configuration. need double check on this.
n
I think auto adjust is disabled because I have a hard limit with 
datacache_disk_size
. I never changed 
datacache_auto_adjust_enable
. I think I'll have to set size to 
0
and then enable auto adjust (or maybe it'll get enabled automatically after restart)
k
yeah, just my guess, need double confirm, and fix if necessary.
n
Alright thanks for looking into this! I'll post back a status update once we're running out of storage again 🙂
👌 1
Open in Slack
PreviousNext
Powered by