Hi, I have a couple of questions about BE disk and memory metrics. I’m running StarRocks 3.3 on AWS EKS, deployed via Helm chart, and monitoring with Prometheus/Grafana. 1) I’m using the following BE disk metrics in Prometheus: •

starrocks_be_disks_total_capacity

•

starrocks_be_disks_avail_capacity

•

starrocks_be_disks_data_used_capacity

From the documentation, my understanding is: • `disks_total_capacity`: total capacity of the disk • `disks_avail_capacity`: available capacity of the disk • `disks_data_used_capacity`: used capacity of each disk (storage path) Intuitively I expected:

disks_total_capacity - disks_avail_capacity ≈ disks_data_used_capacity

But in my cluster,

disks_total_capacity - disks_avail_capacity

is consistently 5–10% larger than

disks_data_used_capacity

for the same backend. Is this difference expected? What exactly is counted in

disks_data_used_capacity

vs the “used” part implied by

total - avail

? 2) For all BE pods (for example

kube-starrocks-be-0

,

kube-starrocks-be-1

,

kube-starrocks-be-2

), I see a consistent gap between: •

starrocks_be_process_mem_bytes

(from StarRocks) • the actual Kubernetes pod memory usage reported in Prometheus/Grafana for those pods For every BE pod, the Kubernetes pod memory usage is roughly 10% higher than

starrocks_be_process_mem_bytes

. Is this difference expected?

reason for disk gap usage: 1. tablet/rowset deleted on BE side, has a background task sweeping the trashes, that will give some delay. 2. other unrelated sr usage, e.g. rocksdb metadata.

@Kevin Cai Thanks a lot for the explanation on the disk usage gap. If you have time, could you also share your thoughts on my second question about memory metrics?

2)

For all BE pods (for example

kube-starrocks-be-0

,

kube-starrocks-be-1

,

kube-starrocks-be-2

), I see a consistent gap between:

•

starrocks_be_process_mem_bytes

(from StarRocks)

• the actual Kubernetes pod memory usage reported in Prometheus/Grafana for those pods

For every BE pod, the Kubernetes pod memory usage is roughly 10% higher than

starrocks_be_process_mem_bytes

.

Is this difference expected?

starrocks_be_process_mem_bytes

is from the memtracker BE itself measuring the memory usage, it may not be absolutely accurate because of variant conditions: 1. memory allocation that didn't go with mem_tracker usage, mostly from thirdparty libraries that out of control from BE codebase 2. if JVM is started, JVM mem usage is also out of the tracker usage 3. memory hold by jemalloc, usually the memory released by application to jemalloc, won't be returned to OS immediately, it will be a gap between application usage and jemalloc usage, this can be confirmed by

starrocks_be_jemalloc_resident_bytes

and

starrocks_be_jemalloc_allocated_bytes

4. other processes running in the same pod, causes additional overhead, usually this part is trivial, but in case a customerized image is used, worth to check that this part can be taken out of consideration. so in k8s env, monitoring the actual pod usage is more practical than just be_process_mem_bytes.

Thank you very much for the detailed explanation — it was really helpful!