Hi team! I am using flink-operater with application mode (flinkdeployment CRD) for a metrics convert application.

taskmanager.numberOfTaskSlots: "1"

taskmanager.memory.managed.fraction: "0.0"

taskmanager.memory.managed.size: 100m

taskmanager.network.memory.buffer-debloat.enabled: "true"

execution.checkpointing.interval: 1m

execution.checkpointing.tolerable-failed-checkpoints: "3"

execution.checkpointing.unaligned: "true"

state.checkpoints.dir: s3://<>

state.savepoints.dir: s3://<>

high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory

high-availability.storageDir: s3://<>

s3.endpoint: http://<>

s3.access-key: <>

s3.secret-key: <>

s3.path.style.access: "true"

kubernetes.operator.reconcile.interval: 1m

metrics.reporters: prom

metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory

metrics.reporter.prom.port: "9999"

env.java.opts.taskmanager: "-XX:+UseParallelGC"

I change the taskmanager.memory.managed.fraction config to 0.0 (it means managed memory size = 0) because I am not use managed memory (not using rockDBstate) But, taskmanager always be deleted and has the error code 137 (OOM killed)

2023-02-06 07:54:33,801 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Worker log2metric-prod-example-taskmanager-1-1 is terminated. Diagnostics: Pod terminated, container termination statuses: [flink-main-container(exitCode=137, reason=OOMKilled, message=null)], pod status: Failed(reason=null, message=null)

2023-02-06 07:54:33,802 INFO  org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] - Closing TaskExecutor connection <> because: Pod terminated, container termination statuses: [flink-main-container(exitCode=137, reason=OOMKilled, message=null)], pod status: Failed(reason=null, message=null)

I increase taskmanager.memory.managed.fraction config to 0.4 and the taskmanager pod is running normally So, is flink still using managed memory even though we not using rockDBstate which causes pod to use more memory than requested?

You can try setting it to

0.1

I think the network layer also uses managed memory.

I also have some confusion around this. We use rocks db as state backend. 1. What is the implication of setting

state.backend.rocksdb.memory.managed: "false"

2. And the metric

flink_taskmanager_Status_Flink_Memory_Managed_Used

always shows 0. I am just wondering if managed memory being used in our case.

I will try it out, but I would like to ensure taskmanager not be killed by k8s because it reaches the resource limit in the pod. Is there some document that instructs us hard limit the taskmanager memory?

Thanks . Yes I did go through the document you shared. I was not sure about the behaviour after setting

state.backend.rocksdb.memory.managed: "false" as flink_taskmanager_Status_Flink_Memory_Managed_Used

showed zero.