Sharath Gururaj
06/27/2024, 8:05 PMtaskmanager.memory.task.heap.size: 1000m
taskmanager.memory.managed.size: 100m
taskmanager.numberOfTaskSlots: 2
state.backend.rocksdb.memory.fixed-per-tm: 5000m
state.backend.rocksdb.memory.managed: false
taskmanager.memory.jvm-overhead.max: 100m
taskmanager.memory.jvm-overhead.min: 100m
from reading the code, I see that flink costs the write buffers to the block cache . That means even write buffers come under the cache quota. from reading the logs, I see the cache seetings as ~4.3GB
block_cache_name: LRUCache
block_cache_options:
capacity : 4369066666
I also verified that all rocksdb instances are sharing the same block cache pointer, and hence would come under the same limit
block_cache: 0x7fd9e9351980
so the total resident memory of taskExecutor should be at most 4.3GB+1GB=5.3GB. I am running a job, But when I check the resident memory via htop, it is 9.5GB. Can someone explain how the memory calculation works?