My team and I are using v1.16 and have been trying to make use of Flink's asynchronous IO functionality and we've ran into an interesting problem: My main question is "_*What could be causing high memory usage on the Flink JobManager*_?" The JobManager memory usage seems to be filling up quite dramatically up until it gets OOMKilled by Kubernetes. Regardless of how much memory I seem to give the Flink Process i.e 8GB, 16GB, 32GB the issue persists. I did have some luck artificially constraining the heap size for the Job along with some other memory settings which stopped it from being OOMKilled by k8s at least, but once the heap has been filled, our Source step, slows to a crawl and the whole job eventually exits with no record of it appearing in the Flink History Server. I am still quite new to Flink and JVM memory management/tuning as a whole, so forgive me if these are really naive JobManager memory settings:

-Djobmanager.memory.process.size=32768mb
-Djobmanager.memory.heap.size=24576mb
-Djobmanager.memory.off-heap.size=7168mb
-Djobmanager.memory.jvm-overhead.min=192mb
-Djobmanager.memory.jvm-overhead.max=1024mb