Hello Team, Is there a memory leak in compactor? I...
# openobserve
p
Hello Team, Is there a memory leak in compactor? I see compactor pod slowly accumulates memory and increase continuously, We have to restart the pods to get rid of this and this happens with only when pod when 4-5 replicas of compactor are running in parallel. We are running OSS helm chart version 0.92.0
s
Does all 4–5 replicas show this same climb, or just some of them ? Also, what are your
ZO_COMPACT_MAX_FILE_SIZE
and
ZO_MEMORY_CACHE_*
settings, if they're not the defaults .In the meantime, capping the memory cache on the compactor (
ZO_MEMORY_CACHE_MAX_SIZE
) or turning it off (
ZO_MEMORY_CACHE_ENABLED=false
) is a quick way to rule out the cache as the cause. cc: @Hengfei
h
we have got anyone report this before and ourself didn't find that. slowly increase, wasn't it OOM?
p
Does all 4–5 replicas show this same climb, or just some of them ?
just 1 pod the oldest of them
Also, what are your
ZO_COMPACT_MAX_FILE_SIZE
and
ZO_MEMORY_CACHE_*
settings, if they're not the defaults
They are the defaults Alert triggered when it crossed the requests and we recycled it before it hits the limits so no OOM.
ping on this
h
which metrics are you looking for memory usage?
p
memory utilisation of compactor pod
h
what is the exactly metrics name