Hello Folks, I am trying to debug some memory issues in flink jobs. I am seeing in metrics that managed memory total is always equal to managed memory used. Does that mean managed memory portion is really completely being used? This is with setting

state.backend.rocksdb.memory.managed: "true"

If i set above config to false, managed memory used is zero. Please help me to understand this behaviour better. I am using flink operator, deployed our job in application mode. Thanks

This blog should help https://flink.apache.org/2021/01/18/using-rocksdb-state-backend-in-apache-flink-when-and-how/#tuning-rocksdb

Thanks. I did have a look at this. But some points are not clear to me. If i am setting

state.backend.rocksdb.memory.managed: "false"

Does that mean managed memory portion of TM is not used at all and may be other task off-heap memory is used to store rocks db related stuffs? In that case doesnt make sense to reduce managed memory fraction to less than 0.4 ?

I am trying to debug some memory issues in flink jobs. I am seeing in metrics that managed memory total is always equal to managed memory used. Does that mean managed memory portion is really completely being used? This is with setting

As far as I understand. When

memory.managed: "true"

, Flink will pre-allocate the managed memory buffers and the metric will reflect this as 100%. However, Flink does not have visibility into the exact utilisation and therefore it is 100%/does not change with actual usage. I am not too familiar with the behaviour of the metric when you set it to

memory.managed: "false"

, but I would assume that 0% does not reflect the actual usage (since Flink might not have visibility).

In that case doesnt make sense to reduce managed memory fraction to less than 0.4

This depends on your app. Generally speaking, do not set

memory.managed: "false"

unless you are an advanced RocksDB user and need fine grain control. If you are not using RocksDB, then it is advised to set the managed memory fraction to 0 (unless you are using Python). Reducing the fraction will result in additional memory allocated to the Java heap, but you will be taking away from RocksDB.

Thanks for the detailed answer. We want to use rockdb. We were seeing constant increase in memory in our jobs. Heap memory increase was not so much. But overall TM pod memory is always increasing. So i was trying to understand which memory i should increase. We started with managed.memory : false But that does not help much. So I will change this value to true first and then i might need to increase managed.memory. fraction.

Which Flink version? Does the memory eventually stabilise or are you getting container OOM kill? If you are getting OOM kill the general recommendation is to increase taskmanager-memory-jvm-overhead-fraction . Additionally ensure you are using Jemalloc and not Glibc in your Docker image (this reduces memory usage due to fragmentation).

Using flink 1.17 and flink operator 1.5. Yes memory increase continues for 2 or 3 days and one of the TM pods gets OOM Killed eventually. Yes it does stabilized for sometime though we are using flink:1.17.1 as base image and packaging our jars in application mode. Doest flink image use jemalloc by default?

Doest flink image use jemalloc by default?

The community Flink image does yes (AFAIK), but if you build your own then you may need to configure it

I can try increasing this too. Thanks.

They should have an equivalent effect, besides one being absolute and the other relative. If this does not help there may be a genuine memory leak somewhere

Sure . Thanks