StarRocks, a Linux Foundation project, is a next-generation sub-second MPP OLAP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics, and ad-hoc queries. StarRocks received InfoWorld’s 2023 BOSSIE Award for best open source software.

StarRocks

Снимок экрана 2025-11-21 в 23.31.09.png

```weekly chart, sharp drops are a restart```

check jemalloc_allocated_bytes and jemalloc_resident_bytes, whethere these two numbers are close to the memory usage monitored from the pod.

`starrocks_be_jemalloc_allocated_bytes` and `starrocks_be_jemalloc_resident_bytes`

```So these metrics are not variables for configuration?```

these are prometheus metrics which can be pulled from CN via `http://&lt;CN&gt;:8040/metrics`

```starrocks_be_jemalloc_allocated_bytes 1541884120
starrocks_be_jemalloc_resident_bytes 1647411200```


```I scaled the FE down to 3 pods and noticed that this only happens on the leader.```


```Why CN? If the memory is leaking on FE.```


if FE nodes, check the jvm related configuration and JVM stack metrics.

Does jvm also have an endpoint for metrics? They're just not activated in the chart, so I didn't check.

"-Dlog4j2.formatMsgNoLookups=true -Xmx14G -Xms2G -XX:+UseShenandoahGC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"
16 GB ram он node,

GC changed in an attempt to solve the problem

usually xmx can be set to 70% ~ 80% of the node available Mem.

```Okay, then I experimented, scaled FE to 3 pods to get a quorum and reduced the memory to 4 GB per pod. I actually changed JAVA_OPT to "-Dlog4j2.formatMsgNoLookups=true -Xmx3G -Xms1G -XX:+UseShenandoahGC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time" The result is the same.```


```There is even a file with profiling and a graph)```


```t was solved in the following way: 2 Fepods and 4 CN. After that, the memory leak stopped. We need to somehow better understand how to build a quorum in Starrocks.```

Снимок экрана 2025-11-23 в 16.20.30.png

Снимок экрана 2025-11-23 в 16.20.37.png

Снимок экрана 2025-11-23 в 16.20.37.png,Снимок экрана 2025-11-23 в 16.20.30.png