Hi everyone, We’re currently using GreptimeDB 1.0 in distributed mode and have been running into some issues with the write cache. We’ve noticed that disk usage keeps growing until the disks become completely full (100% usage) on both our datanode and frontend nodes. Our current configuration looks like this:

[[region_engine]]
[region_engine.mito]
write_cache_size = "10G"
write_cache_ttl = "1h"
sst_meta_cache_size = "256MB"
vector_cache_size = "512MB"
page_cache_size = "512MB"
selector_result_cache_size = "512MB"
enable_write_cache = true
write_cache_path = "/data/greptimedb_write_cache"

[region_engine.mito.index]
staging_size = "2GB"

Even though

write_cache_size

is set to

10G

, the

write_cache_path

directory continues to grow and eventually reaches around 200 GB. I was hoping someone could help me understand: • Is it expected for the write cache to grow beyond the configured

write_cache_size

? • Are there additional settings we should tune to better control disk usage? • Is there any configuration related to disk usage for frontend nodes? I wasn’t able to find anything in the documentation. Any suggestions or insights would be greatly appreciated. Thanks in advance!

> Is it expected for the write cache to grow beyond the configured

write_cache_size

? write_cache_size is a soft limit for an LRU cache. It may be larger than the configured size but should not it’s unexpected to grow beyond so much. Have you collected the metrics of the server? There is a

greptime_mito_cache_bytes

metrics for the cache size. You can check the

type = "file"

for write cache size. It’d be helpful if you can

du -sh

the directories under the

write_cache_path

to find out which one consumes the most of the space. > Are there additional settings we should tune to better control disk usage? No. The ttl and size limit usually work well. Did you check the datanode’s and frontend’s log? It looks like there are some issues with removing files. > Is there any configuration related to disk usage for frontend nodes? Currently, frontend nodes store logs on disk. You can reduce the max log files. https://github.com/GreptimeTeam/greptimedb/blob/d4cfbd3400b3ad5b6dcd0a94ce7e01f3d8ac01a9/config/frontend.example.toml#L296-L297

Thank you for your answer. We capture metrics: All of them indeed in correct range. Under 10G