I'm using a Flink 1.17 and a Flink Kubernetes operator 1.5.0. After each scheduled savepoint I see a memory increase. In my case it can be for up to 200Mib. It's pretty intensive job and has a lot of states. Savepoint location: s3, using s3-fs-presto-1.17.1. If one taskmanager works for quite a lot of time without restarts I will eventually reach a limit.

how to configure monitoring like this

@kingsathurthi we are using Prometheus Reporter: https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/metric_reporters/, and then connect Prometheus to Grafana.

What is the metric being displayed? Flink memory or pod memory? If it is the latter, then I would guess it is the OS Page cache filling up. RocksDB uses Page cache as a second cache when reading data from SSD. If you are reading the full state for a checkpoint, you most likely gonna hit the SSD and the data will be cached in the page cache.

Flink Main Container. Memory of the Taskmanager. We are usign EKS and storage is SSD

Can you paste the metric name ?

sum(container_memory_working_set_bytes{namespace="$namespace",container!="POD",container!="",pod!="", pod="$pod",container="$container"}) by (container) container is flink-mmain-container and pod is taskmanager

I think it is indeed page cache building up. If the memory reaches the limit, I would say it it will be reclaimed by the OS.

Well, initially limit was 14G and once it hit the limit backpressure started and app was 90-95% slower. Right now I gave 20G and waiting for it to build up.

Could you please share your question in the Speedb hive? Our team and community would be happy to assist and support