Rehan Sayed
08/19/2025, 9:28 AMtaskManager:
managedMemoryFraction: "0.01"
nodeSelector:
flinknode: taskmanager
type: highmem32
tolerations:
- key: "flinknode"
operator: "Equal"
value: "taskmanager"
effect: "NoSchedule"
taskSlots: 32
resource:
memoryFactor: 110
processMemory: 220000 # 220GB. 245 GB is allocatable.Total memory assigned to pod is 220*1.1=242 GB
jvmMetaspace: 256
offHeap: 16000 # 16 Gb
cpu: 30
These are the current rocksDB configs :
rocksDbSettings: |
state.backend.type: "rocksdb"
state.backend.incremental: "true"
state.backend.rocksdb.checkpoint.transfer.thread.num: "8"
state.backend.rocksdb.thread.num: "8"
state.backend.rocksdb.memory.managed: "false"
state.backend.rocksdb.writebuffer.size: "256mb"
state.backend.rocksdb.block.cache-size: "128mb"
state.backend.rocksdb.block.blocksize: "64kb"
state.backend.rocksdb.writebuffer.count: "4"
state.backend.rocksdb.writebuffer.number-to-merge: "2"
state.backend.rocksdb.files.open: "400"
state.backend.rocksdb.predefined-options: "FLASH_SSD_OPTIMIZED"
Attached are diagrams of the block cache usage alongside the task manager memory graph (where increases in block cache size coincide with pod OOMs) and the heap usage graph
I would appreciate any advice on how to investigate or resolve this issue.Sumit Khaitan
08/20/2025, 4:33 PMDavid Anderson
08/21/2025, 8:26 PMstate.backend.rocksdb.memory.managed to false and configure RocksDB via ColumnFamilyOptions. Alternatively, you can use the above mentioned cache/buffer-manager mechanism, but set the memory size to a fixed amount independent of Flink’s managed memory size (state.backend.rocksdb.memory.fixed-per-slot or state.backend.rocksdb.memory.fixed-per-tm options). Note that in both cases, users need to ensure on their own that enough memory is available outside the JVM for RocksDB.Rehan Sayed
08/22/2025, 10:20 AM