Hey folks, The TaskManager pods in my flink pipel...
# troubleshooting
r
Hey folks, The TaskManager pods in my flink pipeline that use the RocksDB state backend have been consistently experiencing very high memory utilization, with memory usage remaining steadily around 95% over an extended period. Despite normal fluctuations observed in JVM heap memory, the overall memory consumption within the pods stays persistently high, which has been leading to resource pressure and intermittent pod failures due to OOM Using flink 1.18.1, flink-kubernetes-operator version 1.11.0, jemalloc as memory allocator and n2d-highmem-32 VMs Following are the task manager configs:
Copy code
taskManager:
  managedMemoryFraction: "0.01"
  nodeSelector:
    flinknode: taskmanager
    type: highmem32
  tolerations:
    - key: "flinknode"
      operator: "Equal"
      value: "taskmanager"
      effect: "NoSchedule"
  taskSlots: 32
  resource:
    memoryFactor: 110
    processMemory: 220000 # 220GB. 245 GB is allocatable.Total memory assigned to pod is 220*1.1=242 GB
    jvmMetaspace: 256
    offHeap: 16000 # 16 Gb
    cpu: 30
These are the current rocksDB configs :
Copy code
rocksDbSettings: |
  state.backend.type: "rocksdb"
  state.backend.incremental: "true"
  state.backend.rocksdb.checkpoint.transfer.thread.num: "8"
  state.backend.rocksdb.thread.num: "8"
  state.backend.rocksdb.memory.managed: "false"
  state.backend.rocksdb.writebuffer.size: "256mb"
  state.backend.rocksdb.block.cache-size: "128mb"
  state.backend.rocksdb.block.blocksize: "64kb"
  state.backend.rocksdb.writebuffer.count: "4"
  state.backend.rocksdb.writebuffer.number-to-merge: "2"
  state.backend.rocksdb.files.open: "400"
  state.backend.rocksdb.predefined-options: "FLASH_SSD_OPTIMIZED"
Attached are diagrams of the block cache usage alongside the task manager memory graph (where increases in block cache size coincide with pod OOMs) and the heap usage graph I would appreciate any advice on how to investigate or resolve this issue.
s
@David Anderson can you please help here ?
d
You're pretty far down the rabbit hole of customizing your rocksdb setup. I'm not sure what to expect from this combination of settings. I will just point out what it says in the docs under "Expert Mode": > To control memory manually, you can set
state.backend.rocksdb.memory.managed
to
false
and configure RocksDB via
ColumnFamilyOptions
. Alternatively, you can use the above mentioned cache/buffer-manager mechanism, but set the memory size to a fixed amount independent of Flink’s managed memory size (
state.backend.rocksdb.memory.fixed-per-slot
or
state.backend.rocksdb.memory.fixed-per-tm
options). Note that in both cases, users need to ensure on their own that enough memory is available outside the JVM for RocksDB.
👍 1
r
Thanks a lot @David Anderson