I am using Flink 2.1 but had the same problem with...
# troubleshooting
g
I am using Flink 2.1 but had the same problem with 1.20. We have fairly large snapshots 40-50 GB. Is it possible that savepoint at some point need to fit in TM memory? During savepoint pod memory usage jumps by the around size of the snapshot (half of the size for each of 2 TMs). No Flink memory metric notices the memory usage, only Kubernetes pod memory usage metrics.
It seems there is some memory issue/leak in jemalloc. Above problem is resolved when setting:
Copy code
MALLOC_ARENA_MAX=1
DISABLE_JEMALLOC=true
Reported issue: https://issues.apache.org/jira/browse/FLINK-38212